alt Hacker News> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
Replies
Even if he was wrong about those statements, exploring other types of models isn't necessarily a bad thing. Especially if we can find ones that perform well with less demanding hardware requirements.
Absolutely agree fwiw. Scale out over LeCunns desired exploratory research was absolutely the right call. Called out because it seems like MS is making a similar call.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
All statements he made about world models did not already turn out to be wrong. He may be right in the long term.
LLMs work well for search problems with the help of a strong verifier.
As it turns out, every problem is ultimately a search problem. But not every domain has an easily available verifier.
It's Meta, lol. They wasted 80 billion $ on Zuck's fever dream called Metaverse.
In the short term. In the long term, his research on learning will be the next revolution.
[dead]
Umm no. There’s the old factory way to build things like Ford model T; and then there’s the f*king Toyota method!
Which is superior, the big f*king monstrosity or the elegant on-demand, small batches, reduce waste?
Meta will go in the shitter and Deepseek will make the small batches.
Historically, that happens now and then. Examples include color television cameras and fluorescent lamp ballasts.
RCA developed the first good color TV cameras. They were monsters. Originally, electronic TV (as opposed to mechanical scanning) used Zworykin's iconoscope. This needed huge amounts of light. It didn't have a photo-multiplier stage to boost sensitivity, but it did accumulate charge from light over the whole frame time. There was an alternative approach, Farnsworth's image dissector, which also needed huge amounts of light. It had a photo-multiplier stage but only the light from the dot currently being scanned by the scanning beam mattered.
So RCA threw money at the problem. They developed the image orthicon camera tube. It combined the advantages of both approaches, integration over a full frame and a photo-multiplier stage. It was insanely expensive, costing thousands of dollars in the 1950s. It needed about twenty different voltages and multiple independent power supplies. It worked reasonably well.
Then came color. That took three image orthicons (RGB) inside a box the size of a small refrigerator, along with optics to split the incoming image into three colors. Plus a rack of analog electronics per camera, connected through a large cable. Plus an alignment problem from hell, with a huge number of adjustment knobs.[1]
Scaling worked! From the 1950s to the 1970s, color TV broadcasting was powered by those monsters.
Now, independent of this, other people were working on other approaches. Vidicons, silicon detectors inside camera tubes, and finally fully solid state image sensors. Eventually, in 1983, after thirty years of image orthicon monsters, RCA produced a usable solid state camera. Today, a cell phone camera costs about US$10 in bulk, and it's far better than the monsters.
It took thirty years of low-priority development to get past the giant image orthicon cameras.
Electronic fluorescent lamp ballasts had a different problem. That was a suppressed invention. The magnetic ballast industry was dominated by Magnetek. Someone invented a solid state replacement, which was smaller and cheaper, and licensed their patent to Magnetek for royalties. Magnetek just sat on it, not wanting to gut their core market. The inventor hired Townsend, Townsend and Crew in Palo Alto and sued. (Read this in the law firm's brag book in their office.) Eventually, the inventor got a hundred million or so, solid state ballasts went into production, and Magnetek seems to be gone. (The company called Magnetek today, which makes crane controls, is an offshoot of a Litton unit, and seems to be unrelated.)
There are also parallels from the steam locomotive industry, the Baldwin Locomotive Works being the best known. Baldwin actually built the first Diesel locomotive. They made a few half-hearted tries at Diesels over several decades, then gave up and went back to steam. Baldwin ceased all locomotive production in 1956. The corporation lingered on for a while, not doing much.
[1] https://www.tvcameramuseum.org/rca/tk40a/pix/rcatk40aart.pdf