I don't remember having any issues with this at all. Have you finished the game? It might be in the end-game that you can do it.

EDIT: As I typed that, the more confident I became that it is in the end-game.