I never took city/town/castle sizes to be literal as to what you saw in game, If you did you'd have to assume that most people go and squat outside behind a tree to go to the bathroom, and that food was either uncooked or was magically cooked without kitchens.

Same goes for number of npcs you can interact with, and the actual number of people living on a planet.