alt Hacker NewsHow do you use this model locally, similarly to using `llama-server -m <model>`?
(I mean: outside direct or substantial use of Python, and running the Neural Network in the most efficient way.)
Replies
I use opencode + <a decent saas llm> to set up all this new ai generation stuff. GLM-5.3 is my current gun. Safely inside podman containers too because I dont trust this fast moving python eco system at all. Never do I want this running on my main OS.
I have FLUX.2 klein and dev, Ideogram, LaDA-Image and SenseNova locally. Works great. Ive never touched a file.
The days of making container yamls myself is over. I read them but I dont edit anymore.
I've set it up on my local machine just now, as my first local image diffuser. I can confirm it's very easy.
I tried stable-diffusion.cpp, following its compile guide here[0], and its Qwen Image-2.1 specific instructions here[1]. It works out of the box. I made a test pelican[2]. It took 3 minutes on a CPU.
[0] https://github.com/leejet/stable-diffusion.cpp/blob/master/d...
[1] https://github.com/leejet/stable-diffusion.cpp/blob/master/d...
There is difussion.cpp which is intended for those types of models. I set up krea-2-turbo with the help of ChatGPT 2 months ago, if you have a capable computer that's what I would suggest once it becomes supported.
I am using sd.cpp, which is the cousin of llama.cpp: https://github.com/leejet/stable-diffusion.cpp
it already has day-0 qwen image 2.1 support!
on the linked GitHub page they list support Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V with links to each
Unsloth Desktop is the easiest way imo. There are already gguf quants of this model, or simply wait until the official one comes out.
Probably ComfyUI is one of the easiest way to get started with local image/video models. Or perhaps vLLM, if they have support for it already, would be something like `vllm serve <model> --omni --port 9080`
why not just as you suggested i.e. https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.... then get the result either via a UI or wget/curl it back?
Additional question is what kind of local hardware would be required for this? 7B parameters sounds very light weight, but I'm not sure. (Edit: The download is 33 GB).
Edit x2: As usual I'm in a twisty maze of pip packages that don't work together, with obscure errors about missing modules, even though I followed the instructions on the page to the letter. I really wish people didn't use Python for this stuff. A simple C/C++ program would be so much better.