Just try to go all over the cave before meeting him up lastly at the area near the exit. Look for an area in the map which is a dead end and is shaped like a small square. I'm sure that’s where you’re missing a fight with Yojimbo!