The downfall of Matt Lauer, Al Franken, Kevin Spacey and other men of power has been called part of a “national reckoning,” drawing attention to an age-old culture of harassment and abuse of women.

But as someone who works with ordinary men and women in abusive situations, I fear that the glaring spotlight on famous and powerful men pushes the experiences of everyday women and men further and further from the conversation.

The more we focus on the powerful and prominent, the easier it becomes to believe this is a problem in Hollywood and Washington, amongst men in powerful roles. This gives everyday men a free pass, an excuse not to examine their own behavior. We can’t settle into a false sense of security and continue to act shocked when it happens … as if we didn’t know it was happening every day in every one of our neighborhoods.

When we focus on famous men and women who have either perpetrated abuse or been victimized by it, we allow society to say “it’s not all men” or “it’s only those men.” We do not interrogate the conditions in our communities — all communities — which surround all men, places where the silence is palpable.

We must ensure that this current “national reckoning” draws attention to the same ways men everywhere get away with abuse and harassment.

At the same time, the current focus on celebrity harassment reinforces the message that some who abuse are worse than others, as if there is a category of abuse that doesn’t cross some invisible line of acceptability. As if pressuring a partner to have sex out of love is different from Al Franken forcibly kissing women or from Donald Trump boasting about grabbing women.

I do not pretend to know the answers about which acts are bad enough to constitute removing men from office or firing them from a morning television show. But I do know that it’s just as important — in fact, probably more important — to create a conversation including all men and women, in every community.

Until all men realize they are products of the same privilege and community conditions as Matt Lauer and other famous accused men, they will be able to hide behind the mantra, “it’s not all men.”

This recognition is a prerequisite for meaningful change. We need to create a community where there is a swift response to any person who would victimize others, and a clear message about accountability.

We need a community where the response to victimization does not include qualifiers and conditions that insist it is not all men, but rather asks how all men will take responsibility for the change that is necessary.

Anna Harper-Guerrero is the executive vice president at Emerge Center Against Domestic Abuse and a fellow with the OpEd Project.