Huh, the HP Wiki says that's true. I don't remember that being mentioned ever, though (especially since they make a pretty big deal in the 5th book about him never being a prefect).