What sometimes a lot of us more marginalized popover available people don't realize is the extent to how oblivious more privileged people can be.
Here's a somewhat embarrassing personal anecdote to illustrate this:
When I was a college sophomore, I used to file noise complaints with campus police whenever my neighbor threw a party. One day, my neighbor, an able-bodied woman of color, confronted me about it. She wasn't just mad, she was scared, and I was confused that she was acting like having the police ask her to turn the music down was such a bad thing. It's not like they'd get arrested, right? My neighbor basically said to me "I wish I could feel safe talking to police about minor stuff like noise." That was when I learned that, for a white woman like me, it was just someone with a badge asking politely. But when campus police showed up to her door, they treated it like a raid. The police looked for reasons to arrest people at her party, and if they didn't, they at least acted threatening. Even if, for once, the police did just ask politely, it still terrified everyone at the party because of previous experiences with police.
Here I was, walking around campus going "I'm so oppressed. I'm so aware of privilege. I'm sooooo not racist because I face my own troubles and I know what it's like." Yet still, I was so unaware of how I was doing something racist on a regular basis. The surprising thing to me was, my neighbor wasn't aware of how oblivious I was either. To her, police are always scary and violent, so I must have known about how police would treat her and her friends. Calling the police couldn't have had to do with noise, but because I wanted to terrify and threaten her and her friends. To her, if I really only cared about noise, wouldn't I have just talked to her in person?
In that context, I was being what is called an Unconscious Unintentional Oppressor (or UUO). According to Dr. Derald Wing Sue, 85% of the time when someone is doing something racist, (cis)sexist, ableist, etc. they are being an Unconscious Unintentional Oppressor. Now, ignorance is no excuse, even if the majority of people are ignorant. What they're doing (and what I was doing) is still super harmful, and they're still responsible for it. They just don't know that they're doing it. Often, because of willing denial that they could do something harmful. The oppressive behavior is a manifestation of their internalized popover available bias.
The other 15% of the time, they are what is called an Overt Conscious Bigot (or OCB). Someone who really does outright hate a group of people and wants bad things to happen to them. It can be difficult to confront oppressors, because it can be hard to tell whether they're just a UUO or secretly an OCB. A UUO might need convining to believe that they're doing something harmful, but they certainly will agree in general they shouldn't harm people. An OCB wants to be harmful towards the group(s) they hate, and a confrontation could escalate to violence. To make matters worse, most OCBs will feign ignorance anyway if you confront them. It's understandable that someone would be afraid to confront someone they think is an OCB.
 Derald Wing Sue. Microaggressions in Everyday Life: Race, Gender, and Sexual Orientation. Hoboken, NJ: John Wiley & Sons, Inc., 2010.