For me, Ma'am is never used. Miss is used for some of my female friends, usually when I want to give them a title for some reason. I have never given a male a title, and never will.

IN that vein, some suppliers at my work call me Sir. It freaks me out to be called Sir when I am only 22.