yeah in the NES version it worked by number of times used, regardless of the enemy.

in the GBA version it depends on the strength of the enemy, so yeah it is possible that if you have say level 15 ultima, and you are casting it on a rank 1 monster, it may take you an age to level up your spell. compared to the NES version levelling up quicker against the same monster.