Originally, it obviously stood for "you all" since English doesn't really have something like "Vosotros", etc.
That's why y'all itself doesn't bother me. It's a contraction to create the same conjugate (or whatever it would be called) as some other languages already have.
But yes, "All of you all" is a bit redundant and referring to one person as "you all" is a bit silly. I always chuckle. However, if y'all is used "correctly" then it doesn't really bother me.