Of course, if you are just voting them out because the game is more fun with active players then I'm down with that.
This isn't a game; it's life.

Which means they are no more likely of being mafia than town. In fact, they are more likely of being town since the majority of players are generally town-aligned. It's pretty much a crap shoot with low odds.
As is every vote. However, I'd rather play with people whose actions I can appropriately judge. With all due respect, I have faith in my abilities to call out bulltrout.