It's actually best you waited so long. The sandstorms tend to almost stop entirely after you defeat him.