I gave chatgpt 3.5 the type signature for a co-algebraic encoding of a mealy machine:

    newtype Mealy s i o = Mealy { runMealy :: (s, i) -> (s, o) }

And it gave a really impressive analysis.

Then I scrambled all the names and asked with a fresh context like:

    newtype Foo z e g = Bar { blob :: (z, e) -> (z, g) }

It got completely confused and generated a bunch of non-sense. It was at that moment I realized that LLMs don't really understand anything.

And yes I understand that a newer model would not get confused by this.