The person calling should be understanding that you are busy - and because they've reached out, there's a pretty good chance that you can raincheck, right? Hang out with the person you are already with.
This, and also the fact that I hate talking on the phone unless I have to.