logoalt Hacker News

msdz • today at 8:22 PM • 0 replies • view on HN

Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.