You do only get up to three, but there's 4 Rare Truffles there, thus a potential total of 12 per repop. The reason I prefer to go to Neverland is mainly because of the repop rate. I've only seen the Rare Truffles in the Final Rest like 3 times out of my whole entire time playing (And I've fought there for hours to get Orichalcums from those Invisibles.). While, on the other hand with Neverland, I get them to show up after 3 times entering that area.

That, and I always forget to unequip that Aerial Sweep.