alt Hacker News> This is an excellent use case for completely local, small model inference
Is it really ? LLM take lot of ram and drain battery. People run Firefox on low end computer.
> This is an excellent use case for completely local, small model inference
Is it really ? LLM take lot of ram and drain battery. People run Firefox on low end computer.
Indeed. Realistically a 'capable' small local LLM, even one that's definitely not as good as externally hosted ones will require a single 16GB GPU and access to basically all of the RAM on the GPU. That's not something people running Firefox on a $500 laptop with 8 or 16GB of total system RAM and a CPU-integrated basic graphics system have to spare.