It's relevant because not even the mafia would know it.

How can you say this. Everyone does something suspicious, town or not.

Because townies accusing other townies doesn't happen.
You're missing the point completely. It doesn't matter if whatever he was doing looked suspicious or not, it wasnt because he was not mafia. In fact, he was one of the Townies most important players. Whether the mafia knew he was a doctor or not makes absolutely no difference, they would have known he was a Townie and he was leaving himself open to be gotten rid of easily, it's a simple choice to make.