alt Hacker NewsHere’s another repo that might be interesting
https://github.com/slvDev/esp32-ai
It shows using Per Layer Embeddings to run models that normally take 100s of MB of ram run efficiently in just a few KB of ram and keeps most of the model on disk. It’s somewhat similar to MoE so it doesn’t stream in the whole model from flash for every token, but takes it even farther. If some of these devices have 1GB of ram and a few GB of disk there might be some interesting options here.
I’ve been investigating getting something like that running on the Wii too, though I haven’t made much progress in a while. Some basic calculations says a 1B model might be possible, at least a few hundred M
I would actually like to see the Wii one, I don't have one unfortunately so I can't directly try on it but this esp is interesting I would like to replicate and/or improve the results on it, this technique is similar to the streaming i did from flash but probably more efficient, I need to look it up and understand it more. Thanks for sharing it