How do you know it's their shadowball? Also, if they put it there - an object of darkness and indescribable eldritch qualities, isn't that suspicious? By taking that, you might as well have prevented some sort of unholy ritual!