Yeah, it's annoying, but try to think of it as which direction you want the *camera* to move in, instead of which direction you want to look in.

That helps me remember what to do!