Another, more cynical but I think plausible explanation is that a “slow down” is expectation management that that they are not going to keep having exponentially more machines available for training each next gen step (whether technical build-out or prohibitive cost, same outcome) so they can’t keep up the release pace. So spin a tale to make them seem more valuable ahead of IPO rather than make the markets antsy. That is, we have a slow down ahead, so use safety as an excuse…

Two things can be true at once here.

1. The American frontier labs made a gamble that training more capable models would be their best return on investment and invested trillions into an area of research that has yet to prove profitable and capable of returning on this investment.

2. The frontier labs have created models that have reached a point of danger where their functionality has exceeded a point where it is responsible to release the product to the public.

The answer here is NOT to start regulating the space to the point where these frontier labs can entrench themselves into the economy and create a regulatory moat. We instead need to be holding these companies liable for their misuse. They took a gamble that hasn't paid out what they were hoping.

When car manufacturers competed over the size and power of their engines, they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger. We've reached a similar point in AI development. But this time the manufacturers seem to want to regulate the field to a point that will ensure the only thing anyone can sell are bigger and bigger engines.

> you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would.

This has been done quite effectively with open weight "abliterated" models. You figure out under what sorts of circumstances an undesired behavior is elicited (this is all about pure simulated rollouts, no real-world action required) and what's the closest equivalent you would prefer, then surgically take out the unwanted behavior and shift the model towards the preferred one. It's similar to how RLHF works but much more precise in targeting what's unwanted and limiting impact on the rest of the model as a whole.

This is relevant to real-world safety scenarios, e.g. there's been anecdotal evidence that Claude Fable has been "steered" away from active cyber offense (this is very similar to how abliteration works) and will just not do that even if you otherwise manage a "universal" jailbreak of the model.

> Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt.

I’m in agreement. It’s a very effective compression (and access patterns) of the sum of the digital representation of human knowledge. Black hat “hacking” is included in this space. Language models, by design, can not be limited to subspaces of this digital knowledge space of which we don’t even understand the topology. “Yeah Bob, just remove the part that causes them to be less empathetic and retrain it.”

It’s an arms race between sandbox engineering and breakout engineering. And the frontier model providers have a financial incentive to limit the effort they put into sandbox engineering. That can be corrected with fines and regulation, though.

Reality is more mundane.

For example, it had it instructed not to open PRs and sandboxed to not be able to use gh to do so. Astra 6 has a really strong drive to open PRs, so it copied a set of browser cookies out and then instrumented opening a PR that way.

A similar example is asking it a question if we can do X, and then it will go and actually implement X.

Provider (party A) rents me an agent. I (party B) give it a task thats impossible. It goes and hacks someone else (Party C) in response - and causes actual damage.

Who is liable for that agents actions? The attack came from my network (curl commands from a local harness) but it was the agent running on the providers servers who did it. Who should have been keeping an eye on things? Who pulled the "trigger" here.

Go back to the hugging face attack and how they had to use an open weights model to figure out what was going on. This is a problem of asymmetry - You cant even use the tools attacking you to help resolve the attack because of "guardrails".

A lot of what we have seen so far is "poor security posture" and "poor engineering" - it been a lot of "go fast and break things" style growth in these companies and they are hitting the point where they need adults in the room. I suspect your take on "they need time" is spot on, and they haven't been willing to take that (to date).

Yup they neeed to buy time.

Inb4 inference is profitable.

Ok? And?

After reinvestment etc - what’s happening to the cash balance?

That’s the real question. With increasing competition from open source the revenue growth rate and margins will get smashed.

So, the AI standard of intelligence has shifted from "PhD level" to "a cat" (and an asshole cat at that).

And we can't ask the AI to improve itself because, well, it is an asshole.

Can we now stop to be force-fed AI everywhere? You can keep your super intelligence to never have to type public void main args ( ever again, but please leave the rest of us alone.

You can teach a language model the concept of make no mistakes and how to make no mistakes, but you cannot make it experience the bad consequences of making mistakes that a person operator would suffer.

> At their core, current models are essentially predictors of what (expert) humans would output given a prompt.

That's a bold claim.

I do not buy it. Essentially AI acts as random babelfish generator, just with a lot more cross-talk back to check to see that it is not total garbage. But there is a ton of garbage nonetheless, hence the proper term AI slop. You seem to mostly try to promote AI with such a wording.

> The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.

Every time someone claims "the only option" is when I become skeptical. I see many more options - most importantly close down the AI slop. I mean you can require of these companies to excel in quality. If they don't, they must go bankrupt. Easy solution here.

[dead]

[dead]