Well my smiter just got to NM so I'm going to have to investigate this for him.

Looks like you need to talk to Warriv after you do your Andy quest before you exit the game: What causes Andariel bug drop. Test results. - Diablo 3 & Diablo 2 Forums

I typically speak with Warriv to go to Act 2 after I defeat Andy, so my chars should have the bug activated.