Whoops, no. Keeping artifacts doesn't affect your Seal Rating, instead it's your Evaluation Level that drops. You get a -5 on your Evaluation.

If your Evaluation Level drops to 0, and you go into a town or dungeon after that, you get the C ending, which is really much more of a Game Over.

Your Seal Rating, instead, is only to be worried about when you're shooting for the A ending.

If you keep the Artifacts, you still get the Event EXP, so don't worry about that. Most of the time, keeping the artifacts is a good idea. There are plenty of rare items, weapons, and armor that really help out.