Between the beginning and middle of the game, I would have to agree with you, but once you get towards the end, or do any of the harder hunts, that becomes far from the case.
Also that is only one aspect of the whole battle system. Most people do not bother with quickenings since they drain your MP and somewhat based on agility and luck.