In general it's pretty hard to think of any RPGs that were scary or unnerving because that's just not what they're going for most of the time, even the usual haunted house/ship/whatever tends to be really tame in RPGs.

That said, The Secret World had a lot of it (makes sense, it's all about horror, lovecraftian stuff and other creepy things).
One place in TSW that stands out in particular is the Amity House facility, a lab where they test/experiment on kids for supernatural powers and things went wrong (of course).