I'd say it like this: " I hope you don't mind me saying so...". I don't know why, but that's what my so-called English abilities tell me.

I don't know the reason why, but maybe it has something to do with the differences between English and American..? - I'm clearly just guessing here.