Swimming eats at colour pretty harshly (as Jiro found out when he went to the beach with coloured hair and walked out of the water basically colourless again). It's obviously fine if you don't put your hair under (this is my strategy) but if you want to actually swim ​then yeah, I'd wait.