logoalt Hacker News

nltoday at 3:25 AM1 replyview on HN

Trinity Large Preview managed 17/25 on my agentic SQL benchmark: https://sql-benchmark.nicklothian.com/?#all-data which is a fairly mediocre score for a large model (Qwen 27B managed 23/25)

This non-Preview release scored 16/25. Probably the same model as the preview, or at least not particularly improved if you want agentic performance.

Good to see more options for large open models though!

It's hard to point definitively to a reason it underperforms but generally models that perform well at agentic tasks were trained on very large numbers of tokens (Qwen, frontier models) or were heavily post trained for reasoning (see eg Nemotron-Cascade-2-30B-A3B at 21/25 vs the base model Nemotron-3-Nano-30B-A3B-Base at 12/25 )


Replies

anon373839today at 5:30 AM

Bit of a tangent, but I'm pleased to see that Qwen 3.5 35B is tied with GPT-5.4 and just 2 points behind 4.6 Opus. That little model is so impressively capable and fast! I'm frequently still surprised that I have that level of capability and speed running locally on my laptop.

show 1 reply