Then why didn't you say as such? On it's own it wasn't a big deal, but considering the large amount of suspicious behavior up to this point, it is seen as a negative. The other players hopped on the vote, sure, but there were what, like three votes on Yoruichi at he time? When you voted for Yoruichi, it was after a plea from her and after an hour of waiting.

Keep in mind your behavior beforehand, you're the only remotely active person with any bit of suspicion on you.

Keeping that in mind