alt Hacker Newsfor 32gb, this is my model of reference now, you have to run it with a patched version of llama and is still not available in lmstudio or omlx, waiting for that to streamline the experience a bit. But so far, the best i had till now.
for 32gb, this is my model of reference now, you have to run it with a patched version of llama and is still not available in lmstudio or omlx, waiting for that to streamline the experience a bit. But so far, the best i had till now.
It looks like upstream work is underway: https://github.com/ggml-org/llama.cpp/issues/29058
The sheer number of geniuses working so fast on this repo never ceases to amaze me.