I struggle to believe that AI R&D is ingenious

Llms are far from being ingenious or innovative. The main way they can surprise you today is pulling in information that’s new TO YOU - I impressed with the algorithms they pop out sometimes simply because never formally studied compsci. But when it treads on my home turf I find it very basic, and for bigger picture stuff like system architecture its terrible, by its nature it doesn’t have a “vision” of how the whole system fits together

This matches my instincts too. When trying to debug ML learning pipelines or research projects, LLM suggestions I've received are almost always techically correct, but they point out small, tedious details which don't really matter. In my experience, they miss the forest for the trees.

I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.

I've also found it very hard to leave AI agents up to their own devices for long periods and get much productive benefit from it.

I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.

It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.

For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.

If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.

> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.

I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.

On one hand, yeah, I agree.

On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.

(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)

Using Fabel to improve some non-AI ML models a while back, I had a similar result - it could implement perfectly any suggestion I made, but it could not work autonomously due to making very poor suggestions itself, or getting distracted by the wrong things, making dumb assumptions, failing to see the big picture, etc.

These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.

At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.

It would have been a lot more informative if the author told us more about what Astra had suggested and why it didn’t work. I have a strong feeling that neither of the first two tasks require an “ingenious” solution. Just applying techniques from similar games. I don’t know enough about the third task to comment.

Instead of asking Astra to do the thing, the author should research similar games and try to apply the techniques used in those games. For example, Scrabble is a hidden information game. Try using an algorithm like CFR which is used to solve such a game.

Similarly, ultimate tictactoe is a perfect information game so it can be handled with MCTS.

Once you understand the basics, you don’t need an “ingenious” solution. It’s a university project.

Honestly at first I was really impressed by AI progress because most of what I do day to day falls squarely into the training distribution - and the models seem to excel at that kind of work, literally in certain areas they are more competent than me.

But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.

For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.

Part of the problem here is how people use these models and expectations are formed.

If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.

I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.

It's interesting that we still need to go into these "try again until you find the correct answer" loops, and often still need to tell the model to ground their claims in facts. Models have gotten better at this but I still see myself asking "are you sure?" way too often.

If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.

I've often said that the term should be "Artificial Eidetic Memory" and not "Artificial Intelligence."

> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.

Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)

AGI right around the corner!

They are poor at establishing the state of a system and applying basic logic to it. Where we make a minimal model of the state in our heads, they make more and more words in the hope that some of them will match the training space.

I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".

After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:

-- It's possible for code in a bundle to execute without the bundle being activated.

We also have a new state (another kind of fact)

-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.

So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:

"which is definitely ACTIVE (since your code is running from it),"

Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.

Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.

For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.

It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.

Innovation comes out of the human experimenting with solutions and learning new things that the llm knows.

It’s a dance, not an orchestration.

> In all of these cases, the model's judgment is just very poor in a basic way.

This sentence resonated strongly with me. Not just in R&D but in every single interaction with the model. From Fable all the way to Sonnet they all very clearly show just zero judgement unless the answer was already seen during training.

The ceiling on LLM intelligence might be in sight. Sure, they are squeezing out the last juice now with RL, for example on design Opus 5.5 is clearly an upgrade. But overall, true intelligence (to think big picture, small detail, ingenious, innovative, novel, or even to just use short simple language to get the fundaments across) seems lacking just as much now as 2 years go.

Did you see the chart on model performance over time on LifeVal? Ceiling in there. https://openai.com/index/lifeval/

They are useful if properly guided. Just as every other tool.

Silent Bob guy just "rebuts" the original poster by spouting how many loc Claude spits out without bugs and unread tests. Which doesn't truly counter anything op said. I'm sure silent on guy can be smarter than me but come on. And yes ai report writing is often verbose and full of crap. Shouldn't need special prompts for it to be focused get to the point

I'm old enough to have realized that I've done plenty of R&D in my life that wasn't formally named that. R&D is about the substance, the mode, the impact, not the names or titles.

LLMs are great at doing work quickly and they don’t say something is impossible or discourage you, so you can really tackle problems that previously would have taken years of work.

They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.

This reads like someone who tried Codex a few times with a blank system prompt and no context and came to the conclusion LLMs are dumb. All of those four examples are handled well by Opus 5.5. Especially with context and competent direction.

The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.

anything you prompt is such that instruction fine-tuning means you're actually the idiot