You're only moderately screwed. It will be a hard fight without Fenix Downs, but it is possible, just a matter of good preparation, improvision, and trial & error. Don't give up, you can do it.

Don't bother attacking the SoulSavers at all, focus everything on Wrexsoul when you get him to emerge.