> I think for most people the intuition is something like: the state is there to protect people (in principle at least); when someone is sufficiently dangerous to others they forfeit this right and it becomes just about protecting others from them; whereupon 'vengeance' is basically permitted against them

There's no connection between "protect[ing] people" and "vengeance", The latter is not necessary for the former.

The retributive instinct should not be "fed" by the State. It is perfectly normal to have that feeling, but we should recognize it for what it is and let it stay as such. You might think this isn't possible or stable but the Nordics would prove you wrong.