alt Hacker NewsI looked up the original paper. It's an interesting read and foreshadows a lot of the current hot arguments around LLMs, but I'm not sure it's aged especially well:
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
However, from the perspective of work on language technology, it is far from clear that all of the effort being put into using large LMs to ‘beat’ tasks designed to test natural language understanding, and all of the effort to create new such tasks, once the existing ones have been bulldozed by the LMs, brings us any closer to long-term goals of general language understanding systems. If a large LM, endowed with hundreds of billions of parameters and trained on a very large dataset, can manipulate linguistic form well enough to cheat its way through tests meant to require language understanding, have we learned anything of value about how to build machine language understanding or have we been led down the garden path?
...
Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
...
Finally, we would like to consider use cases of large LMs that have specifically served marginalized populations. If, as we advocate, the field backs off from the path of ever larger LMs, are we thus sacrificing benefits that would accrue to these populations?
Especially in a world where a there's myriad open Chinese LLMs, it's not clear what policy changes are being recommended today. Gebru's paper explicitly advocates backing off from developing larger LMs than existed at the time, 6 years ago. Do those celebrating the paper continue to advocate that LLMs be scaled back to GPT2 level, for safety?