It's only five points... That's not much to speak of, and in the end, there are far greater bonuses to be had from accessories than small Defense bonuses.