> The issue I have run into is that for running llama.cpp, which is a very actively developed bleeding-edge software, it seems like experimenting with different versions/configs/patches etc. was fighting with the Nix philosophy of immutable software

May be. I'd guess it probably does it in a high-friction way compared to what you expect.

e.g. in a typical linux distro, you can just replace the (globally installed) version of the package and use that.

With Nix, you've got a declaration of how to build a package, including its dependencies; and each of those dependencies is also a declaration of how to build that package, and so on. -- And so, "just change this version" results in rebuilding the full chain of dependencies.

If you want multiple readily-available versions of llama.cpp each with different configs/patches, then using Nix would make this easier I'd think. But for "I just wanna try this, then try that", it's going to add overhead.