You’re right. But consider that in order to be useful when not fused off, the design would need to have a bunch of additional logic (interconnect ports, power control machinery etc) at the periphery of the to-eventually-be-fused-off area that would likely remain even when things were fused off. That may impact power.
Apart from that there’s the other usual angles: The very fact that there’s additional logic in the compute path (eventually fused off) means additional design and verification complexity. The additional area, although dark, eats into the silicon yield at the fab.
Not saying it’s not possible.