I think it's just semantics. There's a lot of ways to answer that.

It's not as if you have to give a super short answer and leave a lot up to imagination. You can just say "I'm naked except for the shoes I'm wearing" and they'll get the point.

Let's just say someone asks you if you're naked and you have to give a quick yes or no answer. If you're wearing a hat only, just go for the yes. What the person's really asking is "Is your junk showing?".