There is an action you can take at this point-- I want to say asking the guard for help with the Brothers quest, or maybe exiting the dungeon -- that lowers your SeeD ranking.