If it makes you feel any better, I tended to vote out friends before non-friends (once you take away the alliance) because more often than not, they were the ones with enough other friends in the game to make another alliance. It was all very strategic.