What exactly were The Silence up to? I remember that they said they needed weapons and something about stopping anyone who got in there way, but were the really evil enough to wipe out like that.
They had been on Earth longer than humans, so The Dr didnt really have a right to relocate or kill them, IMO.