I the Porre Army is still there, you have to bring Norris along with you! Later in the game, the Porre Army has withdrawn so you can enter as you wish