Not so. I'm part of a religious community for whom FICA is something we can opt out of via IRS Form 4029, and I can avoid contributions to FICA more broadly (not just SS taxes!) by handing HR the approved 4029 that I receive in response to filing.

I don't know how available this is to folks who aren't Mennonite or Mennonite-adjacent, but it's there.