Yeah, don't recruit them at all.

I did it, and it's quite fun. Changes the ending a bit, not significantly or anything, but it does.