Perhaps this is missed in the TV show, but not so much in the Books. Even common folk in the middle of nowhere who don't want the bloody throne are being massacred. The River Lands were wrecked by the war of the seven kings and even after that you had The Mountain's men, Roose Bolton's men and the bloody Mummers going round raping and killing.

Chances are, unless you lucked out and was in the Vale... you'd probably die.