Yeah, you can go back to it and fight it later. In fact, most people end up leaving it till later because the time limit imposed on that fight is much shorter than the time limit in the mandatory second fight.