alt Hacker NewsFor the last five months or so, opus and fable have both been able to do this task as a soneshot prompt. An inference engine is almost pure compute and to a first approximation the only operation your hardware needs to support to do llm inference is multiplication [the rest of an inference engine is "some trig"].
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
Replies
So true, people think they can prompt AI, tell it to create something amazing and "Novel" so they can feel cool and bragg about it. Shame on such people.
lol sure
"claudia slop me up some 600m, commit, push, make 500 pages of slopped README.md, github stars+++"
great, you should definitely do this, there will be absolutely no problems whatsoever and it'll also run at peak speeds while of course being completely unusable because who wants to run a slopped 600m
t. at 1200+ commits on my "oneshot"
I agree with you, and I'm using claude and other models heavily, the idea is not in the soneshot - because actually as you see in the logs I posted on github, it took around 107 + 50-ish experiments to figure out which models to run which and push the speed.
I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.
so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.
But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.