There's a BIG difference between travelling across the globe and travelling across the vastness of space.
Plus, chances are that any alien life is either far less advanced than us - in which case they won't be space travelling - or far more advanced than us, in which case it's hard to see how they could possibly gain by taking over our planet, or whatever. They might not even take much interest in us, necessarily.
On a related note, I've got another fear that could go on the list. If advanced aliens turn up and their purposes are benevolent towards us, rather than malevolent, they might well make us realise what horrible, horrible people we all are. See Childhood's End by Arthur C. Clarke, where a bunch of aliens who look like demons show up and point out all the stuff we're doing wrong.