I don't know if it would be that much difficult than it is with your actual party. If you really want a challange with a less fighter party, try the one you already got, but kill the fighter at the very begining of the game. That would make the game a little bit harder.