Not so. I'm part of a religious community for whom FICA is something we can opt out of via IRS Form 4029, and I can avoid contributions to FICA more broadly (not just SS taxes!) by handing HR the approved 4029 that I receive in response to filing.
I don't know how available this is to folks who aren't Mennonite or Mennonite-adjacent, but it's there.
About 0.14% of the population is Mennonite. By filing 4029, you assert that your community is going to provide the equivalent of SS and Medicare, so it amounts to about the same thing.