alt Hacker NewsIf you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...
This should work:
cd /tmp
# Get the Prism macOS runtime
curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
tar -xzf bonsai-runtime.tar.gz
# Get the ~5.95 GB GGUF model:
curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf
# Run the server, I used port 8331
./llama-prism-b10685-7dffb15/llama-server \
-m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
--port 8331 -ngl 99 -fa on -c 32768
uvx llm openai endpoint http://127.0.0.1:8331/v1 \
--model bonsai-2-27b --responses hi
Replies
If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:
export HF_TOKEN=xxx # optional, speeds up the download
./llama-prism-b10685-7dffb15/llama serve \
-hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
--port 8331 -ngl 99 -fa on -c 32768Thanks for all of your exploration in public Simon.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
Where did you get these instructions?
They have a demo repo with a setup.sh script:
https://github.com/PrismML-Eng/Bonsai-demo
The release tag and weight file you suggest doesn’t match what they wrote.
It would be great to have upstream llama.cpp support for this!
I used that to Generate an SVG of a pelican riding a bicycle:
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.