One issue with Voxel-based physics destruction games is that the physics happens in continuous space (as opposed to voxel space). This means that the moment you break off a chunk of geometry, it has to be converted into a mesh and simulated like any other mesh-based model will. This makes voxels seem like more complicated Voronoi-noise based fractures. If you want the modelling workflow or the looks of voxels, it's fine. But assuming that voxels will somehow help with the destruction physics seems not to be a valid assumption.

Ideally, we would be able to do physics in voxel space itself (sort of like a cellular automata based classical mechanics), but that doesn't seem to be possible.