One of the most powerful sci fi I ever read that could fit those requirements is Hyperion. The aren't other races, but some humans have evolved while others get extreme plastic surgery, both adopting animal-esque features. The plot is incredibly unpredictable with several ongoing mysteries and wonderfully imaginative with all the crazy Ish going on. The first book has been compared to the Canterbury Tales, with the backgrounds of a group of pilgrims told in drastically different genres, from horror and satire to military fantasy and cyber noir. Don't sleep on it.

Right now I'm reading The Dagger and the Coin series and it actually does fit all those requirements perfectly. I only recommended Hyperion first because it's insane and D&C is only good-great, but it does keep getting even better and better. The taxonomy of races is really interesting and builds into the lore really well. They were (supposedly) created by now-extinct Dragon-rulers for various purposes, and the mystery behind them only gets revealed in titillating bits and pieces over time. The plot is fairly unpredictable with something coming completely out of left field a ways into the first novel. The author is a protege of George RR Martin and the books overall read very well. It's also a series in progress, so you can get in on it now and be a hipster about it later.