Why Caring About Someone Is Never an Excuse to Tear Them Down

You are sitting in a group of people you like, and someone starts talking about a person who is not in the room. Their voice softens, they say they are only saying it because they are worried, they say it comes from a good place. And something small turns over inside you, because the more they talk, the clearer it becomes that this is not worry at all, it is a quiet list of everything that person is getting wrong.
Maybe the person being talked about could not find the right words in a language that was never theirs to begin with. Maybe they said something the wrong way, or stepped somewhere no one expected them to step. And you sit there already knowing that if the room ever turned toward you, that same soft and caring voice would find plenty to be worried about in you too.
So here is what I keep coming back to. When we say we care about someone, and then we use that care to quietly take them apart, what exactly are we caring about?