If I remember correctly there's a game called 'Eternal Darkness: Sanity's Requiem' which, you guessed it, has a sanity mechanic. Basically, the more stressed your character gets, they start to hallucinate, which can include the character, but also the screen as well, like it going black.

I don't know that much about it, but it has a weird-ass plot apparently. And monsters. Have a look I guess.