logoalt Hacker News

YuechenLi • today at 12:34 AM • 1 reply • view on HN

Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.

But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.


Replies

khimaros • today at 12:49 AM

this seems to be a composite index based on TerminalBench, CursorBench, vals.ai, etc: https://github.com/git-aarvin/mei/tree/main/source/jsonc

looking at the graphic below the table, it does seem like data is missing for Neon which is present for Fable, and isn't penalized for the missing benchmarks.

➕ show 1 reply