Yes, you can fight him anytime you can dig, i.e. , since chapter 2, particularly on the Western area. Before chapter 5 all you have to do is last for a couple of turns, since it just throws you out after it gets X amount of danger. And you can pilfer a Megaelixir every time you fight him (pretty handy for Bevelle's cloister later)