Lowtown is where the original poorer inhabitants of Rabanastre were forced to move when Archadia invaded. This is explicitly stated in the game. And since several of Lowtown's residents know who your proposed ghosts are, including the main characters, and they talk about growing up with them, around them, and watching them grow up in Rabanastre, it proves that they aren't ghosts.