In truth I think (theory here not fact) that they blamed Barrett and Dyne because they were the unofficial leaders of the community (despite there being a Village Elder) and that Barrett went around convincing everyone to join with the Shin Ra.

There is also the fact that Barrett was not there when the community got destroyed, sure he lost his wife, but he wasn't there to lead them. They in turn felt that they lost because Barrett and Dyne were not there, and resent that and soon enough that resentment turned to anger and then hate *tips hat to Master Yoda*.

Lastly, Barrett was a willing scapegoat. In the face of all that suffering, someone walks in and blames himself, people in their warped mind (any mind would be warped if their entire village and family had been wiped out) are going to latch onto to that no matter what, he really only has himself to blame for blaming himself.