if you equip ultima as an extra item at the bottom of the equip screen, I think you could leave them with it. Don't quote me on this though