I don't see anything wrong with saying someone is more attractive because they are a bit lighter either.

I mean, I'm pretty comfortable with saying I think women who have a light brown skin color are more attractive to me than women who are really pale. Why would it be bad for other people to be attracted to the opposite of this?