Do not attack him if he appears. Just wait it out. Sometimes it takes him a minute to decide to return. Other times he jumps right on.