> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.

Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.

What if scaling fails to hold at a certain point? Whether that is a steep or gradual dropoff in returns. What happens to the empires of accumulated compute? How much value do they immediately lose once the returns to scaling ceiling is reached?

This is a dumb question since I know very little about LLMs and transformers but I keep hearing similar things such as "more GPUs/parameters/compute" leads to a predictable scaling in intelligence.

Is that observed scaling part of some equation or law? Is there a limit to how far that scaling goes before a ceiling or diminishing returns hit? Or is it simply based on more GPUs/parameters/compute plus how much human knowledge is in existence and will scale predictably forever?

What makes sense is to do both.

Race down the transformer highway as fast as possible and explore other AI methods.

One need not preclude the other.

I'm seeing the same pattern at other multinationals with huge R&D departments. Mostly colleagues with academic pedigree in traditional ai research who have been for years in denial that transformers are actually - in combination with recent advanced patterns in software development - the key to ai. it's really simple as that! except they weren't part of the revolution and still don't want to accept the reality that their research has become obsolete.

is scaling working? It is also my understanding that Opus 5.5 is probably a well under 10T parameter model. Probably a similar size class to Fable 5 earlier. And supposedly even GPT-4 was a 1T model. Clearly most of the gains are coming from something other than scaling up the model size. Yes, scaling up RL counts as scaling too, but the fact that labs have pivoted to a different axis of scaling is evidence that it's not just a "straight highway" to improve LLMs.

It's worth it to keep exploiting transformers while doing just a bit of exploration of alternatives because they work so well. Heck, just exploiting transformers more will accelerate exploration of alternatives...

This doesn't engage with the article in the precise way that transformer-based improvements and fundamental research that way are not being incorporated into models, at Microsoft AI, which is concerning.

> Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters

Moore's law: Computing power is multiplied by 10 every 6 months? Can't wait for the optimization and rationalization phase, where we'll recover a trillionth of that power to run powerful databases.

When a company pays billions to OpenAI, any activity that does not do a glorious dance in praise and support of that investment is taken to be an act of disloyalty.

100 trillion parameters doesn't make any sense if your dataset is ~30 trillion tokens. My uneducated guess is 1/10 compression ratio is good and that point to ~300 to ~1000 trillion tokens, which again, doesn't make any practical sense.

> If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.

Already happening: https://news.ycombinator.com/item?id=50397994

Obviously we'll see how it pans out, but other approaches feel a bit like trying to ignore the bitter lesson.

> Current top models are mere (mere in big datacenter terms) trillions of parameters right now. ... We'll soon start hitting 10's of trillions and 100's of trillions of parameter models.

Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.

We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).

FTA "The core philosophy there is scaling, and aggressively not research."

This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).

> Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.

Where's the money coming for those 10X and 100X ventures?

It seems extremely likely to me that scaling produces benefits, it will get better at particular things things continue but scaling by itself isn't heading to general intelligence.

I should also say that I use ChatGpt daily and enjoy it for researching algorithms. I tried to set up Claude last week and it/he was so overwhelming as to be unusable. So I think frontier models can be "unwieldy" or something for average use.

> Hey this Opus 5.5 with the increased parameters seems much better than the last one"

Is this true?

I barely notice any improvements ever since... 4.6? At least for software development.

Opus makes the same sort of mistakes and has the same sort of strengths ever since that point in my perception. My impressions is that increasing the number of parameters clearly has diminishing returns.

In fact, the best improvement I got from Claude was Haiku 5.5. Good cheap workhorse to implement plans.