logoalt Hacker News

TimCTRLyesterday at 9:44 PM5 repliesview on HN

Yet none of them would have been here if Google hadn't published "Attention is all you need", the irony.


Replies

dumpsterdiveryesterday at 11:13 PM

The irony may only just be unfolding - check out this paper from 2015:

https://research.google/pubs/distilling-the-knowledge-in-a-n...

# LLM generated summary of the implied irony

Google may consider the standalone frontier-model arms race economically irrational, while still considering frontier-model capability strategically indispensable. Its longer game is probably not to avoid building the biggest models, but to build only enough of them to serve as capability factories—then turn that intelligence into a much larger population of cheap, purpose-built models.

(Human again) If Google knew what they were on to, why wouldn’t they make it their secret weapon from the start? I suspect it’s because they predicted there would be an arms race, and knew how to profit from it. They had a distillation paper published before “Attention is all you need”. In hindsight, is it ironic at all? Or is it obvious?

show 1 reply
pixl97yesterday at 11:14 PM

Even Google wouldn't be here.

It was kind of like radiation science before WWII, it would be freely published because it wasn't potentially world changing yet. After it became a government interest, even people doing things unrelated to weapons would be much more apt to hold their work close.

arjieyesterday at 10:02 PM

The authors of that paper are all at AI companies not named Google.

show 2 replies
mountainriveryesterday at 11:14 PM

Oh that built on a bunch of ideas that weren’t far off. It was a great discovery but like many it’s standing in tall shoulders