They are poor at establishing the state of a system and applying basic logic to it. Where we make a minimal model of the state in our heads, they make more and more words in the hope that some of them will match the training space.

I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".

After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:

-- It's possible for code in a bundle to execute without the bundle being activated.

We also have a new state (another kind of fact)

-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.

So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:

"which is definitely ACTIVE (since your code is running from it),"

Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.

Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.

For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.

It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.