Depends what you take really. I think marijuana is the biggest long term memory effecter, with other stuff you just don't remember the night if you take enough of it.
Apparently LSD improves your memory but I've never been rich enough to buy it because it contains alot of expensive hormones.