The reason you say "Who cares" shows not only that you show no interest, but also makes the person who brought it up small and significant because you're saying that the vote of that person doesn't count, thus making him/her amount to nothing.

If they reply with "I care", then that's the arguement; and is why language invented that reply.