Your position kind of sucks, because you didn't date a co-worker, your dated someone who became your co-worker later. But, I echo the sentiments that you should probably return the favor, and you don't know if you might need his help again in the future. Saying no may take a rocky situation and make it outright hostile. But, if you really don't want to, you can always say you have plans or have to go to the doctor's or something.

I dated a co-worker once. He was actually a manager of mine. Never, ever again. He was professional enough that after we broke up he didn't give me any trout, but I was lucky with that.