"Yeah I went there" is sometimes a comedic way of acknowledging that you've said something some might consider risky, risqué or politically incorrect; more more often, people just use it as an excuse to say something disgusting, profane, or blatantly offensive.

Example:

"Hi guys, could I get your opinion on something? I'm having real difficulty getting a job, no-one wants to hire me even though I'm good. Why do you think this is?"

"Well, obviously it's because you're black/a woman/totally gay - YEAH I WENT THERE, HURR HURR HURR"