Vaan and Penelo aren't annoying to me as characters, but their presence in my party annoyed me. They could and should have been replaced by people more relevant and interesting. Vaan at one point even says he's "just along for the ride." He serves no purpose. I can't think of a time Ashe acknowledged Penelo even exists.