Well, you can avoid the Docker Desktop tax by not running Docker Desktop. colima is a perfectly usable implementation of Docker for macOS, without the bloat of Docker Desktop.

That said, colima still has the expensive VM that upthread is mentioning.