alt Hacker NewsShow HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
Hey HN,
Henry from Cactus here!
We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2.
The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges 300-700 on sub-$200 phones such as the Samsung A-Series.
On the tool call and mobile device use benchmarks, Needle 2 trades wins with closest small models like LFM2.5 230M and Apple Foundation Model, at 5x to 70x smaller, both at f16 vs Needle 2 at 2bit. Needle is based on Simple Attention Networks from our paper (https://arxiv.org/abs/2607.18363).
Edge AI has lately meant Macs and PCs, but that is just 1.5 billion of over 21 billion connected IoT devices in the world today, and in emerging markets most phones ship under $200, no NPU, cheap GPUs. These include budget phones, Raspberry Pis, microcontrollers, wearables, small robots like Reachy Mini, and connected home devices.
A conventional transformer of Needle's width and depth spends 164 MFLOPs per token, and even one squeezed down to Needle's parameter count spends 87, Needle spends 70. Even on a high-end phone, an always-on assistant lives inside a power budget; every MFLOP is milliwatt-hours, and Needle spends 7x to 85x fewer of them per token than the smallest performant LLMs. More about the architecture in the link.
When we structure intelligence for consumer devices as functions with typed parameters, the only hard part is mapping a messy sentence onto them; which function, with which values. Our research found that when framed that way, the problem needs no world knowledge and no open-ended prose, which is why 45M parameters suffice.
Needle 2 expands to structured extraction where the schema can be passed in-place of tools and the model returns structured output. You can use Needle as a text-classification model with an enum field, as a summarization model by providing a schema that extracts key fields, everything but free-range decode.
Every product has its own tool vocabulary and fine-tuning needle helps it achieve frontier-level performance on custom tasks, so using the python package (https://github.com/cactus-compute/needle), Needle can be fine-tuned Needle on a Mac/PC in minutes to a few hours, with automated data-generation pipeline, just pass a couple samples.
Nonetheless, every response carries a learned confidence score based our Cactus Hybrid technique. If above your threshold, act, below it, escalate to the cloud or bigger model. Combining Needle 2 with a private DeepSeek-v4-Flash deployment works particularly well for enterprise-level tasks at barely any cost, we can help with this setup.
We have put a lot of thoughts into Needle 2 but might still be missing quite a lot, please use the playground in the provided link to test Needle and share your thoughts, always appreciated!
Comments
My first query:
> Make it a little warmer in here.
The reply:
> "name": "set_thermostat", > "arguments": { > "temperature": 65, > "mode": "cool", > ... > "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.",
Maybe I'm doing it wrong?
What about use case for replacing regex? I.e "random formatted title.extension" - extract the title or some tag or something for more dynamic string manipulation for pulling structured data out of strings efficiently and more simply than regex provides?
Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:
Query: HN
Result:
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
Could someone please share how such open source micro-LLMs might have been created?
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
That's really cool - I was already thinking of compressing `functiongemma-270m-it` down to 1-2 bits so it would work flawlessly in the browser. Your `Fine-tuning` feature is even much more convenient.
I tested with
import needle
@needle.tool
def add(a: int, b: int):
"Add two numbers."
return a + b
agent = needle.Needle(tools=[add])
print(agent.run("calculate 1 + 1?")["reasoning"])
conclusion: completly useless
This is cool!
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
> turn on the tv
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "tv" } } ], "confidence": 0.0158 }
Very interesting, seems confidence is 0 when tool calls are right?
Congrats on this release. The WASM implementation is really cool. This is a surprisingly good fit for a lot of cases, and I totally want to try turning this into a helper assistant for an application.
Please, though, take a pass at humanizing the text on the page. It's Clauded up all over and makes it hard to read.
This is really cool, I'm curious how much knowledge can their be in smaller models? It seems the current trade off is you need sizeably larger models for more performance but I'm curious if in your work how far this is true, as edge ai is really what needs to get better before physical ai can take off (my two cents).
Makes me think of the demo from some time ago where someone got a ~29M parameter model running on an esp32. I wonder what kind of throughput this could get if a handful of esp32s were strung together...
Edit: I have a pile of d1 minis, but not much time.
I wonder if there's any way to get this to plan out a dag of tool calls? i.e. use the results from earlier calls as the parameters to later ones? I tried introducing a stack based system, but gave up pretty quickly.
Was really cool to see yous use Engrams to cut down compute!
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
Naïve and clumsy question: how would you pair this with speech-text-speech stuff, wake words etc.? Are there good examples of this for a Pi 5?
The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device.
ETA: pun not intended
Looking forward to npm version of needle-rs supporting v2. I added needle support for tool use in my side project.
Any plan to release on ollama?
Any instructions available for running this on an ESP32-S3 or P4 like the site says?
I’d be interested in attempting to run this in a network- enabled 32MiB RAM microVM.
"make it as dark as possible"
{
"function_calls": [
{
"name": "set_thermostat",
"arguments": {
"temperature": 72,
"mode": "cool",
"room": "living room"
}
}
],
"reasoning": "'as dark as possible' -> set_thermostat to warm; 'dark' implies higher temperature; 'cool' mode for darkness.",
"confidence": 0
}
Since it seems limited to matching a few templates and otherwise falling flat on its face, I wonder how 14MB of regexes would fare in its stead. Normally you wouldn't want to parse arbitrary natural language input with regex because of how tedious and brittle it would be, but for the tedium we have LLMs and this alternative isn't exactly robust either.
This is very interesting! I'm going to spend some time with this. This is really the only class of LLM I'm interested in at all. I sincerely hope on-device takes over and everyone looses their asses on these data centers.
Great work! Keep it up
What is the difference between this and random sentence generator?
[dead]
The learned confidence gate is the crucial piece for a 14MB action model. On ambiguous requests such as the HN example, what calibration target decides between abstaining locally and escalating to the cloud?
This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).