You have to capture certain numbers of each fiend type. I have no reccollect whatsoever as to what numbers they are, but you'd be better off just to go and get at least five of every monster. That way you'll [probably] end up getting [at least] most of the species creations, and then the originals should follow from there.

Of course, to get nemesis, you need all ten, which by the time you're catching those tonberries and stuff inside sin, gets to being a real pain. Start with five each, then you can get ultima buster and the clear spheres. Then you can build your stats (slowly, but it's so worth it) and eventually get enough dark matters to add break dmg limit on one of your capture weapons (which is also worth it. saves from haveing to have one of the celestial weapons and accidentally killing the stupid turtle or behemoth or whatever before you can get the last hit in with one of the dinky capture weapons)

there's a ton of good guides for this on gamefaqs, specifically the stat maxing guide. *loves on*