Seymour is vulnerable to poison. If he's too powerful just do that, then you can concentrate on keeping your party healthy and his death will only be a matter of time.