FH's residents are all a bunch of "outcasts" from Esthar. If you did the Master Fisherman sidequest, he explains all about Mayor Dobe and FH's history.