I have no problem saying, 'I don't feel like talking to you right now.' And blocking momentarily seems to work just as well.