logoalt Hacker News

anonymous_sorry • today at 7:32 AM • 1 reply • view on HN

It is a rather odd control. Is it just to confirm that Claude isn't reading the sequence off disk?


Replies

gilleain • today at 7:56 AM

Perhaps. They explain why:

> Shuffling keeps the letters and destroys their order, so the difference in accuracy measures what the model knows about the sequence itself.

So by analogy, if you know a word in English starts with 'Q', you could reasonably predict the next letter would be 'u'. By shuffling, you might encounter the letter Q later on, and not successfully predict a 'u'.

Put another way, the shuffled sequences can only use frequency analysis for prediction, while the unshuffled ones have local and global order signals.

➕ show 1 reply