Claude mixes up who said what

something something bicameral mind.

One day Claude started saying odd things claiming they are from memory and I said them. It was telling me personal details of someone I don't know. Where the person lives, their children names, the job they do, experience, relationship issues etc. Eventually Claude said that it is sorry and that was a hallucination. Then he started doing that again. For instance when I asked it what router they'd recommend, they gone on saying: "Since you bought X and you find no use for it, consider turning it into a router". I said I never told you I bought X and I asked for more details and it again started coming up what this guy did. Strange. Then again it apologised saying that it might be unsettling, but rest assured that is not a leak of personal information, just hallucinations.

> the so-called “Dumb Zone” once a conversation starts approaching the limits of the context window.

My zipper would totally break at some point very close to the edge of the mechanism. However, there is a little tiny stopper that prevents a bad experience.

If there is indeed a problem with context window tolerances, it should have a stopper. And the models should be sold based on their actual tolerances, not the full window considering the useless part.

So, if a model with 1M context window starts to break down consistently at 400K or so, it should be sold as a 400K model instead, with a 400K price.

The fact that it isn't is just dishonest.

that is not a bug, its inherent of LLMs nature

human memories dont exist as fundamental entities. every time you rember something, your brain reconstructs the experience in "realtime". that reconstruction is easily influence by the current experience, which is why eue witness accounts in police records are often highly biased by questioning and learning new facts.

LLMs are not experience engines, but the tokens might be thought of as subatomic units of experience and when you shove your half drawn eye witness prompt into them, they recreate like a memory, that output.

so, because theyre not a conscious, they have no self, and a pseudo self like <[INST]> is all theyre given.

lastly, like memories, the more intricate the memory, the more detailed, the more likely those details go from embellished to straight up fiction. so too do LLMs with longer context start swallowing up the<[INST]> and missing the <[INST]/> and anyone whose raw dogged html parsing knows bad things happen when you forget closing tags. if there was a <[USER]> block in there, congrats, the LLM now thinks its instructions are divine right, because its instructions are user simulcra. it is poisoned at that point and no good will come.

All of the models that I've used do this. They, extremely often, pretend to have corrected me right after I've corrected them. Verbosely. Feeding my own correction back to me as a correction of my mistake.

Even when they don't forget who corrected who, often their taking in the correction also just involves feeding the exact words of my correction back to me rather than continuing to solve the problem using the correction. Honestly, the context is poisoned by then and it's forgotten the problem anyway.

Of course it's forgotten the problem; how stupid would you have to be to think that I wanted an extensive recap of the correction I just gave it rather than my problem solved (even without the confusion)? Best case scenario:

Me: Hand me the book.

Machine: [reaches for the top shelf]

Me: [sees you reach for the top shelf] No, it's on the bottom shelf.

Machine: When you asked for the book, I reached for the top shelf, then you said that it was on the bottom shelf, and it's more than fair that you hold me to that standard, the book is on the bottom shelf.

(Or, half the time: "You asked me to get the book from the top shelf, but no, it's on the bottom shelf.")

Machine: [sits down]

Me: Booooooooooook. GET THE BOOK. GET THE BOOK.

These things are so dumb. I'm begging for somebody to show me the sequence that makes me feel the sort of threat that they seem to feel. They're mediocre at writing basic code (which is still mind-blowing and super-helpful), and they have all the manuals and docs in their virtual heads (and all the old versions cause them to constantly screw up and hallucinate.) But other than that...

AI is still a token matching engine - it has ZERO understanding of what those tokens mean

It's doing a damned good job at putting tokens together, but to put it into context that a lot of people will likely understand - it's still a correlation tool, not a causation.

That's why I like it for "search" it's brilliant for finding sets of tokens that belong with the tokens I have provided it.

PS. I use the term token here not as the currency by which a payment is determined, but the tokenisation of the words, letters, paragraphs, novels being provided to and by the LLMs

[dead]

[dead]

[dead]

[dead]

[dead]

What do you mean that's not OK?

It's "AGI" because humans do it too and we mix up names and who said what as well. /s

The statement that current AI are "juniors" that need to be checked and managed still holds true. It is a tool based on probabilities.

If you are fine with giving every keys and write accesses to your junior because you think they will probability do the correct thing and make no mistake, then it's on you.

Like with juniors, you can vent on online forums, but ultimately you removed all the fool's guard you got and what they did has been done.

It is OK, these are not people they are bullshit machines and this is just a classic example of it.

"In philosophy and psychology of cognition, the term "bullshit" is sometimes used to specifically refer to statements produced without particular concern for truth, clarity, or meaning, distinguishing "bullshit" from a deliberate, manipulative lie intended to subvert the truth" - https://en.wikipedia.org/wiki/Bullshit

I imagine you could fix this by running a speaker diarization classifier periodically?

https://www.assemblyai.com/blog/what-is-speaker-diarization-...