It shouldn't be doing that. Heres how you should use Swap Magic:
<ul><li>Put Swap Magic in and turn PS2 on</li><li>Wait for Swap Magic to bring up the "Load program" screen</li><li>Open lid (or slide tray out without pressing eject) and remove Swap Magic</li><li>Put in the game you want to play</li><li>Press X</li></ul>
Then the game should boot. If you're doing exactly that then the disc or your PS2 is faulty or incompatible. It's probably the disc.