exactly...if you played morrowind you would notice the same thing...if you focue on speechcraft and mercantile to level up, you should stay out of the dungeons, because your physical skills and attributes suffer, wheras those of the monsters you're facing increase.