I think we should see this as simply silly behavior by a government.

Export control is not an effective tool for controlling a consumer facing technology developers everywhere want to use (see:VPNs) so there was no good faith policy justification for imposing an export control.

This is an administration that seems to be keeping track of who its friends are and aren't, and likes to be the center of every story. They also seem to like extracting concessions and reciprocal favors. We saw some of this behavior in the last administration too. US voters deserve better.

> Anthropic got what they deserved

Anthropic got the most rewarding hype ever in the history of mankind.

Imagine a private company invents a piece of technology soooo good that the US government has to issue a ban.

Did the government ban any models from Google or OpenAI? Nah, Russian/Chinese spies and ISIS are welcome to use those dumb models.

Anthropic will probably go for $2T IPO now.

It sounds right, but there is one caveat to me:

Training best models is hella expensive. Anthropic spent fortune to train it, and it definitely plans to make a fortune with it either. This US decision, if not reversed, would cause Anthropic potentially tens of billions of dollars of revenue loss. When company heads to IPO, and burning cash faster than it generates it, such moment can change their entire trajectory, plans for the future, hiring, new models development, etc.

Alright, one might say that “US will fund it directly and LLMs will move from free market to controlled and funded by government assets”.

But even then. Training new models is expensive not only in terms of computer, and not only in terms of utilities, data centers, etc. But in terms of talent either. It is hard to retain top talents with you when they should just train special models for government. I am not sure we are in 1945 again and that top tier AI researches will agree to sit in silo and work for models which only privileged selected organizations might use. Whenever government steps into control, freedom and creativity is affected.

P.S. Where I agree, though, is that we witness the start of government censorship of AI models. Imagine soon Anthropic open back access to Fable. Can we know what they put inside and which capability limitations, derived based on ID/IP, they enforce? No, we can’t. Here I agree that at least the era of government censorship begins.

> Do you think China is going to allow the strongest LLMs ... a year from now when they have Mythos capabilities

"Mythos capabilities" is not some magic threshold. This is exactly the type of language that people used about GPT-4 in 2023. Today, I can run models far stronger than GPT-4 on my laptop at speeds better than GPT-4 offered.

Anthropic are quite good at coining sticky phrases like "Mythos-class models", but these are manipulative attempts to shape the discourse for business purposes and should be identified as such.

I'm a European, the EU is supposed to be one of the closest allies of the US.

The US government found a jailbreak that allowed the user to make Fable do bad things, this is so dangerous that this model must be held back in areas that are not the US...

If this is so dangerous why allow US nationals access to it? Are there no evil people in the US?

Going back to my perspective: let's say I control a big enterprise or a government body, how should I view this or US technology? Should I be like: yes, let's use US tech, they are a reliable partner and would never abruptly cut us off! Or should I be like: there are competent alternatives out there and if your work hinges on wether or not you had access to Fable 5, then your business is probably not going to survive for long.

I suspect the big picture isn't just "governments restricting the availability of strong LLMs to the public", it's a group of tech lobbyists who have managed to push a narrative that's plausible enough to the majority, but serves their master's interests in stifling competition, whether that be from Anthropic or those who know how to use their tools effectively.

The fact that Anthropic are willing to dumb-down their own model responses to "Prevent foreign competitors from using the model to accelerate R & D and protect our leading position." [1] adds credence to this speculation. Anthropic are scared of their own model's power in the hands of competitors: it has nothing to do with security.

[1] https://eu.36kr.com/en/p/3848820681636481

I wonder how this is going to work given half the people working at the AI labs are Chinese foreign nationals, and even more interesting, DeepMind is based in the UK. Plus there is an awful lot of AI research going on all across Europe, especially Switzerland, that is feeding straight into the US major labs.

Banning foreign nationals from using your technology only makes sense if you don't rely on foreign nationals to build it in the first place.

Or are we so far along now we think we don't need them anymore.

I'm wondering if they might go for a restricted access model that goes beyond passport or citizenship, where people can still use it, but you have to be individually vetted, and put on a list to get security clearance.

Ironically, this is something that the restrictive EU AI regulations can help with. Had the Anthropic been in EU, they could not be restricted as long as they followed the laws which is essentially taking some precautions against obvious risks(no social profiling, emotional recognition in schools etc.).

That’s also the difference between being totalitarian government and laws and regulations based order.

The huge investment into LLMs at a loss is about having control of these tools and technology. Now we're seeing a state try to take some control.

But who do you trust more to make these decisions? A democratically elected government or a private company?

I think you are missing the bigger picture that is around the "bigger picture" you are seeing. AI proliferation is more dangerous than nukes proliferation, as any highly capable tech would enable destructive usecases as well. If nukes related material and knowledge was safeguarded, then AI requires it as well.

> I hate being told what technology I can and can't use

Ever since the original GPT-2 "it's too powerful to release!" I've realized that whatever is the current state of open models represents what we really have access to.

It's shocking to me how many people on HN, who engage in long conversations about LLMs and AI, have never actually run a model on their own hardware.

All you need is a reasonably good macbook pro/studio or an RTX [3-5]090 and you can run useful models in the >= 30 tokens/second range (much higher if you choose the GPU path). The difference between what you can run on this hardware and what you can run on hardware that costs 2-5x is not that big. Don't be fooled by people on Twitter/X claiming you need some outrageous setup.

It's also increasingly clear that frontier models are nowhere near close to pushing the limits of efficiency. Quantization, MoE, and other techniques have dramatically improved even in the last year.

For work, of course use OpenAI/Anthropic models, but for anything personal, anyone who considers themselves a "real engineer" should be running local models, using open harnesses and seeing what they can accomplish with these.

Even if open releases slow down or even stop, we have the foundation, right now, for smart engineers to squeeze something quite useful out of. Hopefully we'll one day figure out how to train large models in a federated way. But either way: not your weights, not your inference.

IMO: Its unacceptable that Anthropic be allowed the final say in what "safety" means for their products, and its extremely reasonable that the USG be allowed that say, for Americans. In other words: Anthropic cannot be allowed to distribute an unsafe product. It doesn't matter how much they "tried" to make it safe, by their own definition of safe.

That's separate from the question of whether Fable 5 and Mythos 5 are unsafe. I don't really know. Here's a few things that seem real, though: These models probably have some level of capability to assist with bioterrorism, Anthropic has self-admitted that their own safety measures are imperfect [1], so it should come as no surprise that jailbreaks seem far more possible than Anthropic is leading you to believe in this blog post [2].

[1] https://www.anthropic.com/news/fable-mythos-access: "We suspect that perfect jailbreak resistance is not currently possible for any model provider."

[2] https://x.com/elder_plinius/status/2064776322979676227

If Amazon sold a book that taught someone how to commit bioterrorism, would there be action against them to stop selling it? Its an imperfect analogy, but the parallels are there. LLMs don't get a free pass because they're also so good at writing typescript for beige CRUD apps and bedtime stories.

One thing I hope we align on: Synthetic safeguards (steering, rejections, etc) on top of models to block illegal/sensitive topics isn't good enough. Anthropic has self-admitted that it isn't good enough. We need the technology to lobotomize these capabilities the public deems too unsafe to allow out of the models at the most fundamental level. And, we need to align on what the scope of these forbidden fruit topics are. This is, actually, the only way open source continues to thrive. I want open source models to thrive, but they won't be allowed to thrive, nor should we want them to thrive, if they're teaching people how to engineer novel viruses and other horrible stuff.

China had already forbade their top researchers to even leave China.

Also foreign investments into Chinese AI labs have already been forbidden and asked to exit

I agree.

Honestly, and I don't say it lightly, long term this may have bigger impact on humanity as a whole than Iran war and its varying outcomes ( and consequences ). Separately, note how much this news was not really reported much today. Granted, a lot was happening, but it is telling.

> Will we be the poorer for that, or will we be safer? I think poorer, because I hate being told what technology I can and can't use, but I'm not certain.

I think this is bang on. The motives are kind of irrelevant, because now that the precedent has been set, I suspect they'll be much more likely to go here for future restrictions. It's very convenient (even if true) to just say "security reasons".

My guess is that Anthropic will either address the government's concern and get the export control removed or implement a citizenship verification (like passport upload or something).

I remember something with either ChatGPT or Claude, way early on, where I had to upload my passport to use some level of it (maybe it was the OpenAI API).

Anyway, there's no way they just shut this completely down, the revenue from mythos is huge. So if they can't get the government to budge they'll find a way to be compliant without completely shutting down.

I think this could kill LLM development. What's the point in pushing boundaries, when your business model is already hard to profit from, only to be blocked from selling your work to the entire world? Where's the incentive to continue?

I find it worrysome how often people value revenge over good. The same happened when traffic to SO cratered; as if the destruction of a valuable source of information was good just because the mods suck.

That's only if you believe this is actually motivated by safety, and not corruption. They won't block access to Grok, just watch. They'll probably allow ChatGPT too if it is censored in some way.

> If you think the solution here is going to be open source Chinese models and / or running on your own hardware, think again.

This logic is flawed. China had no incentive to release SOTA models to the world in the first place when OpenAI were milking everyone with closed source paid models. What changes now? Nothing. In fact, this is even more incentive for them to capture marketshare and dependence on Chinese models as the world will simply just use alternatives. Not bow down to restrictions. If your logic were correct that people would just comply, then the tons of VPN services wouldn't have a market in the first place.

A myopic view, but the government has generally not been heading in the direction of an educated populace over the last few decades. It doesn't surprise me that anything that's too intellectually capable is a threat.

The real story is that Anthropic went from being a "supply chain risk" to being a "national security risk."

I agree this is probably their thinking - they view frontier models (and the capability to build them) as a vital strategic edge that they want to keep to themselves.

The problem is that there are network effects at play - the more people you have using your models, the more training and fine-tuning data you're accumulating, so the faster you can develop the next frontier model. Not to mention the fact that more users means more revenue to fund your next-gen model training.

Perhaps the US administration is gambling that US citizens on their own provide enough of a training data and revenue flywheel for them to keep their AI development edge.

The next interesting question will be - will the US share this capability with her traditional strategic allies (e.g. five-eyes countries), or is it truly America First (or, 'America Alone')?

The other thing is what this will do to 1) the valuations of these companies, 2) their potential revenues and therefore the viability of the current datacenter buildout. Looking forward to the reaction of the market on Monday.

We are not missing the big picture, this is what Anthropic wanted. They made this bed, let them lay comfortable in it.

Personally, I assume that AI labs like Anthropic are high value targets for spies from other nations. I also assume that some of those spies have already had success in getting the model weights / source code / other such secrets.

So I doubt this action alone is enough to really stop other nations from getting access to state of the art AI. I think the US would have to go much further to really stop other nations from getting access to state of the art AI.

I think the Chinese don’t share the “AGI-pilled” understanding of AI that you see in some US companies and part of government.

Thus they are far less likely to do something like this.

AI companies business model depends on wide adoption. How will they survive if government closes access to their models?

I do not really like applying the "if we did it, they will too when they can!" logic to other government's.

China has flaws, plenty of them, but there's no real evidence to believe their motivations or mechanisms of pursuing motivations are that similar to that of the United States.

Is fable that good ? I was under the impression that it's just an incremental update, and not even a big one.

Government always restricted data, tools, technology. In France for instance you're not allowed to have a gun, but policemen have.

What's the difference ?

Imo china, and deepseek will keep its open source model because they invest in long term. At some point they could do something similar, but not now.

USA government is just hurting AI development in their country, and that's good news to me.

> These are heading in the direction of being powerful cybersecurity weapons and it will be in the interest of nation states to restrict and control them. In 2 years time, I would be surprised if the strongest LLMs are available for general use at all.

That sounds so great.

> Will we be the poorer for that, or will we be safer?

We will be not just safer but richer. These LLMs are like drugs that should absolutely not be cast freely into the highways and byways. My main worry is that this action will be a haphazard one-off and not part of a coherent plan of curtailing LLM propagation.

>The real story here is that this may be the beginning of governments restricting the availability of strong LLMs to the public, to you.

I can't agree more. This is a precedent not just in denial but possible vagueness. Judiciaries have 'vagueness doctrines' to counter such laws/directives but _these_ may be re-trumped by the deference given to national security.

If we don't get soon a framework by which models may be measured as 'too powerful' vs 'not too powerful' we supercharge the self-dealing (corruption) that this administration has brazenly adopted. Many fingers can be put on many scales; groks may be given a pass while others are held to higher "standards".

Will OpenAI now just asymptotically bump its versions to 5.99999999 to stay under a limit that nobody really understands?

I realize that this has all just happened and we might get some good rigorous clarification from our government.... sigh. We are living in a kakistocracy. Who am I kidding?

Yes. It’s really not a good idea to make this ban. When the US is gradually isolated in this way by its gov’s policy, the world becomes more and more dangerous. What worse, the traditional value of open to competition that Americans have hold for centuries seems to be substituted step by step. It’s absolutely a tragedy.

The scariest thing to me about AI is not what it can do, but that someday public access might be lost and governments/ billionaires would hold exclusive reign. Today could be the last time the public has any idea of the true capability of AI.

You can't use it if you're American either.

The whole thing is theatre.

Anthropic gets into argument with US government over model usage -> Release a model calling it too advanced for safe use -> release the model to public knowing well that this admin has thinnest of skins and will do something

Regulatory capture in roundabout way. Now it is going to take crying wolf over other companies/countries developing “Mythos grade model” to kick off action especially in next two years of this admin.

Companies will keep improving models because AI is not yet fully there. But it is incredibly naive to think governments were ever going to allow state of the art technology to be released to public or do things this publicly. Every company wants to show off and get publicly restricted because it shows off their strength.

I can only say well played Anthropic.

> In 2 years time, I would be surprised if the strongest LLMs are available for general use at all.

That's a bold prediction considering that's true today...

don't worry, these idiots can try, but it is too late for them :)

Repeating from the duplicated thread:

First I want to see them play video games at a high skill level, preferably without any access to game state beyond the same visual output that humans have access to, like a raster frame X number of times per second. One LLM model played Factorio, albeit at a very, very poor level, which can be seen if you slow the video to 0.25 playback speed and pause frequently.

https://old.reddit.com/r/factorio/comments/1u1blr6/claude_fa...

There have been streams of other games, where LLMs and AIs have likewise performed very poorly.

I recognize that LLMs might be better at language processing than these sorts of tasks. But being able to play video games is part of general capability. And this kind of hardcore video game playing, with no access to game state, is also a general task where feigning skill can be harder. If LLMs excel at pretending to be competent without actually being competent, like this AI training approach is arguably about

https://en.wikipedia.org/wiki/Generative_adversarial_network

Then some AIs might be trained and designed for deceiving humans instead of actually being competent and capable. And thus, one response is that they should be met with more difficult tests.

Basically, make tests that AIs or LLMs will not have an easy time cheating. Hopefully, that will engender research in greater LLM/AI competence, not in greater ability to cheat or deceive, neither for LLM/AI researchers and companies, nor for LLMs/AIs themselves.

I think it's too early to understand the ramifications but I agree this is a huge deal.

This is a very interesting perspective. However we always thought that the diffusion of ever stronger AIs was practically guaranteed by its competitive value- you might restrict what AIs are available in your country, but the impact on your economy can be dramatic if other countries have access to better models. In the end, it's hard to imagine governments blocking access to any AI that is just a bit better than what other countries have.

I lean libertarian but I can recognize the danger in having access to a machine that can craft pathogens to spec.

A pathogen with a very long incubation time and a high fatality rate would be about as bad as nuclear war. Maybe we need to figure out how to possibly defend against one person doing this before making it easy for anyone to do it.

Govts wont be able to do shit. Just like we saw with social media. This is just happening faster. Illusion of control theatre will continue for few years. Beyond which we might have totally different looking govts.

No different from encryption

> In 2 years time, I would be surprised if the strongest LLMs are available for general use at all.

It would be too naive to suppose that the strongest LLMs are available to plebs now.

It will just delay SOTA models to us by say 1 year. I’m actually ok with it given that’s it was entirely predictable any govt would do that to even strongish AI

> Will we be the poorer for that, or will we be safer? I think poorer, because I hate being told what technology I can and can't use, but I'm not certain. Maybe you think the government should restrict strong LLMs. Maybe you don't. But either way, this is big news and a rubicon has been crossed and a precedent set. That's true even if the motivation for this is just the government settling scores with Anthropic.

I mean, maybe in principle, but if the object is just hobbling Anthropic you might still get OpenAI's latest model without that much trouble.

Pepperidge farm remembers when they banned G4 Macs for export as well

The whole reason China open sourced its models in the first place was because nobody generally speaking really trusts China and Chinese deployed models (if they were proprietary)

and OSS models gave way to running it with freedom and security.

So OSS models have always tried to catch up to the frontier and lag behind 3-6 months. For my use cases, I am happy with current OSS models especially so if you let frontier-ish models design the plan with your input

If I were to suppose that China created a frontier model so good and far ahead, then I can understand if they don't open-source it. Qwen does it already with their Max models being closed-source.

but if you are suggesting that China in whole will remove itself from AI race, then 3 (or 4) possibilities can occur.

1. Some chinese companies might stop the production of OSS models if their names are known (z.ai etc.) but there are multiple other companies who are fighting with their research labs as well. They might create a decent model and OSS it to get known within world and China.

2. The whole Chinese economy (well similar to America, but to an even more extreme level from my understanding) depends on AI and is a bet on AI. They are funneling state and all bank money into these companies. From point 1, they wouldn't wish to be silent with frontier models and then lag behind and wait for other countries to catch up (point 3)

3. Europe(MistralAI)/India(SarvanAI? Kinda recent) will jump on the opportunity. (My point is that these two regions are trying to create their own models. How much they lack from the frontier is another thing but if China were to remove itself from the race, then they will have much more time to figure out how to make better models)

My point is that america and china are in arms race of closed source vs open source models. If china were to close source its models, they might simply lag behind and other countries will catch up.

4. Either that or you are right and we will have the current frontier OSS models and some more. IMO they are reasonably good as well and I used to wonder what would happen if say it would have been net good if AI was stuck at a similar level to sonnet 4.5 (IMO it was sweet spot), so I don't think that I am reasonably worried about it all. If absolutely need be, you can have an frontier model direct a plan and have OSS models do the grunt work.

“Fable was the strongest model on the market” - explain why anyone should believe that claim.

I’ve been trying to track LLM code generation adoption in the critical infrastructure world - as far as I can tell, it’s nill. Zero. Nada. Nobody is relying on these models to write secure code for anything where failure is catastrophic. Planes falling out of the sky. Nuclear reactors going into meltdown. Electrical grids loosing synchronicity. Lots of these BS claims from the marketing and investment crowd, but - it’s just a useful tool for non-critical areas. That’s all it is.