They really are a problem in the beginning of the game, when usually the thing that causes them to attack you is usually when your party does:

1. Using magick around a group of enemies when an elemental *conveniently* shows up.

2. A gambit using something such as Cure when a party member is at low health when one happens to be around.

3. Gambits such as Ally:Any > Protect that your party accidentally uses when you are walking by an elemental.

These were usually what happened to me when I got them angry at me.
Elementals are somewhat of a nuisance and potentially dangerous to me, but I could usually just use the magick element that they are weak against or string some Quickenings. Now Entites are the problem for me, since their attacks are MUCH more powerful than that of elementals; for example, while a Storm Elemental can cast Silence and Thundara, a Mardu Entite can use Silencega and Thundaga. They also have much more HP as well, so much that I can't kill them even with good strings of Quickenings from both my parties.

A word of advice to those who are not easily able to beat them:

Use R2, go into "fleeing" mode and RUN when you see them, just to be 100% sure you don't end up accidentally using magick against them due to either other enemies or gambits.