I don't see why it wouldn't be possible from my entirely unscientific point of view. Naturally having only one set of genes would cause the baby to be more susceptible to disease and the whatnot and have only the traits from the man and his ancestors. Returning to my entirely unscientific point of view, it would be heresy, so we'd have to burn the man as a witch.