I do the same. I like to keep them leveling all together.
I had to fight those darn Wizards twice to get my Crown. After the first time I totally forgot all about the chest and I didn't realize that until I was out of the cave. So, I had to go back and do it all over again.
I am currently Level 11... almost Level 12 and I have two Silver Swords. They sure are nice. I'm retrieving all I can with the Mystic Key before going on.