Could there be dust or something between the connections of the Memory Card...?

Personally I never remove my memory card from my PS2 once I put it in, I don't see any need to.