You need to get to the end of some of those confidant stories!
I can't remember what my "it's definitely this" guess was, but they ended up guessing it and it was wrong
Yes, that might have tipped you off.
This is why I'm glad I hated him for every single moment I knew of his existence in the game. I never invested any gear in him at all.