The doctors aren't there to get their rape on. They're there to make sure that you're okay. They swear the Hippocratic oath and don't tell anyone about your medical conditio without your consent.

And while you may remember the doctor, the doctor will probably forget you, because you're just another patient. It's not like if they see you in public, they instantly remember "ZOMG that guy has cancer!!"