This is the way.
I’m doing that. Mac Mini M4 Pro with 48G RAM as a headless llama.cpp server.
I much prefer using " thin clients " as the interface to the big VMs running in my homelab