Yeah, it frustrated me so much that I gave up. The only advice I can give you is to have patience, and just repeatedly kill the enemies on the lower Pharos areas. It's very time consuming, and requires a little luck due to the random appearence of the orbs.