alt Hacker NewsI'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such.
Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell.
And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them.
People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.
It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.
Replies
> People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.
At least 3 times last month I was asked to review a change in a .md file used by agents. And I am like: "yeah I guess it makes sense?"
It feels we need to write unit tests for this stuff, but even how to do so in reasonable time and complexity seems difficult.
I think that generally speaking, the tighter the feedback loop, the better AI performs. Ask it to do things that are fast and easy to check and I suspect not much else matters. That's why AI's eating programming but has mitigated success elsewhere, feedback loops in the digital world are orders of magnitude faster.