alt Hacker Newsthis looks very interesting, it can't run a real (read : working on some real tasks) LLM, yet, but it's actually very interesting device.
this looks very interesting, it can't run a real (read : working on some real tasks) LLM, yet, but it's actually very interesting device.
Here’s another repo that might be interesting
https://github.com/slvDev/esp32-ai
It shows using Per Layer Embeddings to run models that normally take 100s of MB of ram run efficiently in just a few KB of ram and keeps most of the model on disk. It’s somewhat similar to MoE so it doesn’t stream in the whole model from flash for every token, but takes it even farther. If some of these devices have 1GB of ram and a few GB of disk there might be some interesting options here.
I’ve been investigating getting something like that running on the Wii too, though I haven’t made much progress in a while. Some basic calculations says a 1B model might be possible, at least a few hundred M