It’s an incentive problem. If even one party defects in a society of pacifists, the pacifists have no real method of recourse besides refusing to interact with the defector, and how many people are going to do that if the defector starts killing people to enforce compliance?

Some subscribe to a soft pacifism where non-destructive violent resistance like disarming the defector or disabling the defector using less-lethal technologies like a tazer would be fine. Pure pacifists who don’t believe in any kind of physical resistance whatsoever are almost exclusively religious practitioners who don’t ascribe a high degree of value to life in this world because they believe non-resistance will bear spiritual fruit in the next world.