logoalt Hacker News

gps372today at 1:52 PM2 repliesview on HN

Haven't gone through full PDF as its very detailed, few things have resonated with me so far.

Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.

You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).

Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.


Replies

BIGFOOT_EXISTStoday at 6:21 PM

Love your analogy.

Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc).

Maps well onto the LLM surface.

show 1 reply
yieldcrvtoday at 6:01 PM

so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness