I'd be curious to see the answer to this. For me it felt like Sega didn't have many games that grasped your attention after you left the Genesis for the Saturn and Dreamcast. With the Dreamcast I can only remember Sonic Adventure, Crazy Taxi, Space Channel 5 Part 2, Sega Bass Fishing, WWF Attitude (which was multiplatform but did have slightly better graphics than the PS1 and N64 ports) Legacy of Kain (which was also multiplatform) and Resident Evil Code Veronica before Code Veronica X came out for the PS2.
As for the system itself I think one of the major things that hurt it was the hype for the PS2 overshadowing its existence as well as the planned introduction for the Nintendo Gamecube and Microsoft making a bid to enter the console race with the Xbox.