> It’s being able to over provision, dynamically reallocate hardware resources, or do things like live migration and entire system snapshots. That hardware/system management aspect is what VM’s give you and containers don’t.
None of those matter in the slightest with containers. Why would you need to reallocate hardware resources when the containers can run on another piece of hardware? You would snapshot the relevant storage, not the whole OS and kitchen sink.
VMs as an intermediary between hardware and containers is just a waste of resources - both directly (RAM, CPU, storage to run a useless OS with no benefit) and indirectly (all of those VM's OS needs maintaining and patching).
It's basically a hold over from the olden days of "everything is a VM".
We run everything on VM just for flexibility. Want to stand up a new machine for testing? Boom, run the script, new Ubuntu server. We need to decommission a machine. Shutdown all the guests, move them over to new machine and start them.
Sure, most of what we do is very cattle and we could run on bare hardware but why not take advantage of easy to add flexibility.