where is the error? 9 years ago (2017) 'attention is all you need' hasn't been written yet, no llms exist. his take is literally 'it would be cool if something like llms existed', seems fine (if anything, vaguely predictive) to me.

As for Kokotajlo, he's a good SF author, but I don't think it's nearly as good a prediction as it appears at first glance.

Like any SF, he's correct about some things, and way, way off base about others. Specifically, he was mostly correct about short-term (1-2 year horizon) technical aspects, but anyone familiar with how transformers work would have guessed those. I'm not even in AI/rationalia land, but believe I was having these exact conversations by 2021. Specifically about multimodality and steam-of-consciousness ['chain of thought' reasoning]. He is spot-on about chip shortages (to my eyes, his most well-done prediction).

Other thoughts are a storytelling device, like a well-told horoscope. He makes non-categorical statements that were already happening as if they categorically will become a new thing. But really he's just going off the current trend that everyone knows, and thus will be nearly-impossible to falsify. For example, 'People who still remember 2021 think of it as the golden days, when conformism and censorship and polarization were noticeably less than they are now. Just as it is normal for newspapers to have a bias/slant, it is normal for internet spaces of all kinds—forums, social networks, streams, podcasts, news aggregators, email clients—to have some degree of censorship (some set of ideas that are prohibited or at least down-weighted in the recommendation algorithms) and some degree of propaganda.' Sounds scary! But it is literally just saying 'polarization and marketing specialization will continue to deteriorate'.... which, duh. In essence, you could make this statement for 2018-> 2021, or for 2008->2018 and 'feel' just as correct!

Other key thoughts were somewhat off. Specifically, 'It now costs an order of magnitude less compute (compared to 2020) to pre-train a giant model' is kind-of-true, in that it costs less now, but the models have gotten so much bigger that the prices haven't actually gone down much. Might be changing soon, but definitely several years off.

The scariest central prediction that the political stuff rests on is 'After years of tinkering and incremental progress, AIs can now play Diplomacy as well as human experts'. This part, as far as I can tell, is just not true and hasn't changed.

And, well, 2026 predictions are just not correct at all. Some examples: 'Mormons create a “Christian Coalition” internet stack, complete with its own email client, social network, payment processor, news aggregator, etc.' -- basically untrue. I know a few ex-mormons, such a thing is not prolific.

'The US and many other Western governments are gears-locked' The US is far from gears-locked, in fact, it's full-steam ahead to er, well, something.

'“Should I wear shorts today?” “Order some more toothpaste, please. Oh, and also an air purifier.” “Is this cover letter professional-sounding?”' -- kind of correct, but in practice most people just do the last one. I'm not aware of many that give chatgpt agents credit card access (refer to the famous vending machine experiment). But predicting this with GPT3 is not that crazy, cause people were already doing that.

Personally, I am much more biased towards the METR timelines/way of thinking: https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measu...

I found Kokotajilo's response comment to this kind of, idk, immature, and not especially mathematical.