Daryl was never acclimated with societal norms even before the apocalypse so I don't see that happening now especially after all he has been through. I don't see the people inside being much of a problem because a lot of them will probably die anyway because they're playing it too safe and they're naive.
Congress lady and L.L. Bean made the right choice bringing in what are basically their bodyguards. Eventually they'll have to get that they need to let Rick's group make the choices or they will end up dead. Glenn knows what he's talking about. They're not equals and are definitely more fit for survival.
Also Carol for president. Wolf in sheep's clothing. Daryl's "you look ridiculous." XD