I hold hands with my friend sometimes and were not involved in anyway other than friendship. Just tell him you don't think it should happen again if you are concerned.