logoalt Hacker News

Scaevolus • today at 9:30 PM • 1 reply • view on HN

This is a solid improvement over Needle 2, which I tried using for a tool-calling interface to a Runescape database site. Unfortunately it's still not quite capable enough for my target compared to FunctionGemma.

   Model                            Correct tool shape    Exact arguments
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━
   FunctionGemma fine-tune, BF16       209/230 (90.9%)    196/230 (85.2%)
  ───────────────────────────────  ────────────────────  ─────────────────
   Needle 3 fine-tuned W4A8             74/230 (32.2%)     47/230 (20.4%)
  ───────────────────────────────  ────────────────────  ─────────────────
   Needle 2 fine-tuned W4               59/230 (25.7%)     43/230 (18.7%)

Replies

HenryNdubuaku • today at 9:35 PM

Thanks for testing Needle out! I'd be very interested in hearing more about the finetuning setup to see how we can make both the library's finetuning setup and the model better.