I haven't had this happen often, but I wouldn't be too keen on someone I barely know or am not particularly close to touching me during conversation.