I'd keep it exactly how you have it in your example diagram. Or I'd simply not look too much into it. As long as you're sitting by your date, it doesn't matter much where the other people sit.