alt Hacker NewsGarry Tan wants US open-weight AI labs to 'distill' frontier models, too
Comments
> He also notes that the proprietary AI labs didn’t ask permission when they vacuumed up as much human knowledge as they could to train their models.
I think this should desactivate the moral high ground from which Anthropic is trying to speak. That they would want to make distillation orderly IMHO is fair, but to make it illegal is very rich from any AI frontier lab, really.
I think OpenAI and Anthropic will go bust, or at least be scrapped for parts in the next 5 years or so. It's clear that the extreme cost used up for training is impossible to recoup, as inference is already being subsidized.
It's also clear that, as Tan indicates, open-weight models will be (and basically already are) just as good as frontier models. It's all about the harness, baby. We will have two main forks in the road, and two new industries created:
- AI hardware (NVidia/Cerebras/etc.), the equivalent of Intel/AMD
- AI software (harnesses, assistants, etc.) the equivalent of Microsoft/Apple
> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider. “The nightmare scenario, the doomer scenario for AI is that there’s just one company,” he said. “It has the best access to capital. It has the best AI researchers. It runs away with it and suddenly there’s one company that’s monolithic. And that would be bad.
Well yes, as I think I said in a previous comment, on the current trajectory OpenAI and Anthropic will really stop releasing models due to distillation and regulatory pressures. Then, they would eat all knowledge work themselves, which would be the end of YC.
Controlling what users and customers do with API calls to closed weight models feels constraining, and there’s a role government can play here to normalize the fact that access to intelligence that was trained on broad public access data should itself also be more a form of a public good than something locked away behind restrictive terms of service
I do not agree with this man all that often, but that is very concisely put.
Not a lawyer but distillation sounds like a transformative work.
Same thing as Cliff Notes imo. In every other area of manufacturering and tech I can use a machine to build a new machine that competes with the original machine. Should Milwaukee be able to prevent DeWalt from using their drill to make a competing drill? Should Jetbrains ban Eclipse contributors from using their IDE?
Society as a whole has paid into this technology: through the theft of its intellectual property, through having to deal with the pillaging of so many commons (digital or otherwise) by it, through skyrocketing energy and computing device prices, and even just through ordinary investment. Democratize the technology! At the very least, don't step in legally to prevent this from happening.
I disagree, I think chinese distillation relies on making multiple accounts at a provider, signing Terms of Services and breaking them repeatedly, in addition to using fraud patterns like IP proxies and networks of credit cards.
I think that software execs should not incentivize users or other execs to break Terms of Services, or contracts of any kind.
An executive or manager of a company that breaks contracts is worth 0, there's no incentive to do business with them, if you know they will agree to doing or not doing something and then breaking that promise.
The word of a businessman is their most valuable asset, Tan is signalling that he is either misinformed on what Chinese distillation consists of, or that it's ok to do it.
FAQ:
- "But the frontier models do bad things too"
- An argument worthy of a 5 year old, one civil issue doesn't negate the other, bring it to a court if you have an actual claim against OAI or Claude, etc...
- "Companies have the right to reverse engineer"
- Ok, do it, but the moment you are creating 10K accounts in a Distributed fashion (Distributed as in the first D of DDoS), using IP proxies and stolen credit cards or your employees and employee family credit cards, you are not doing it because you believe you have a right, you are doing it despite not having a right to it.
EDIT:
Re(actually)reading the article, Tan's take is a bit more nuanced, he seems to be advocating for regulation to restrict the capacity of Foundation models to restrict usage, on the basis (or to the extent) that it was trained on public data, and therefore it belongs or attributes its success to a wealth of the commons.
My pre-existing quip is against those that want to solve this as-is by breaking the ToS. I think that's a weak version of Free Software position, it's very weak to complain that some software is proprietary and want to use it anyway, the strong FS position is that you don't even want to use it if it's proprietary, you won't catch a FS activist pirating proprietary software, they just don't use it and develop alternatives. Similarly it's not a FS position to distill a proprietary model (where you still wouldn't have source code at any rate).
I don’t think appeals to morality or ethics are required for this. You paid for the LLM’s output, you should be allowed to use it how you wish. The only reason distillation is a dirty word is the AI labs trying to spread FUD to protect their non-existent moat.
Frontier labs trained their models on the entirety of human knowledge and didn't ask permission. It's a "want" or "should" it's a moral imperative to distill their models.
Agreed! Allow US companies to innovate by creating an ecosystem of smaller, more efficient open weight models and it will be a net benefit for everyone. Distillation is a good thing.
Preventing token-consumers from developing competing products should be litigated as anti-competitive behavior.
Distilling frontier models is a brute force approach that rapidly hits diminishing returns after bootstrap because of the unevenness of the data. The simpler and more effective method is to have dedicated "teacher" frontier LLMs to generate targeted training data sets specifically for training new models and adjust on the fly based on feedback from the student model.
> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider.
Isn't this exactly what Dario wanted? He thought he knew what's best for the humanity...
Ok, but how do the economics of this work? Based on its settlement, Anthropic paid an average of $3000 per work they scanned based on their settlement (https://tech-insider.org/au/anthropic-copyright-settlement-2...). They and OpenAI pay billions per year for a mix of experts and normal people to label or create data. Why would they continue doing this if the value of this is immediately copied by open models? If your goal is to end the economics of generating and buying data for AI (and I recognize for some people this is really the goal) then sure, but if you want AI for various subfields of interest to continue improving then it's not workable.
Back when people made arguments for software privacy, the argument was usually "big business will still pay and consumers wouldn't have paid anyways so it's ok for us to pirate" - I actually think that was fine for business software but terrible for indie games, whose market was 0% businesses.
But in the AI case, it's not like they get to keep some of the value of their investment - it all gets cloned into models that businesses and consumers alike are happy to use. If someone knows how labs could continue to fund data creation and acquisition in this model, please do share!
I cannot feel anything but schadenfreude regarding anthropic having its IP stolen from it. Bravo Chinese labs, bravo
Eventually the top labs are going to collude and simply not release their best models to the public (if they aren't doing that already).
Net neutrality anyone? If AI is critical to getting work done in the modern era, its access should be guaranteed. Anyone banned from accessing frontier AI is being forcibly left behind. This includes distillation.
some people did bad things, now instead of punishing those people, we want rest of people all do bad things, because that's only fair.
Given the short-term pragmatic, conflicted way that AI tech adoption is happening... won't encouraging distillation effectively taint the entire space of open weights models, with the undisclosed biases of a few models that are under the influence of parties (certain billionaires and politicians) known for aggression and duplicity, and not for admirable ethics?
Following news of companies and projects increasingly moving to open weights models.
As AI gets more central to society, we really need to know how the weights were determined.
Open weights isn't just "free as in beer"; it can be "free as in the mystery drug that creepy guy chatting you up at the bar offered you". And maybe even he doesn't even know everything that went into the tablets, since he too was being worked, by an organ-theft ring who will be harvesting both of you tonight.
That's an analogy to get your attention. Your LLM probably isn't going to steal your organs. But in the current environment, it does and will have ideological biases determined by those with direct and indirect influence over it. And there will be a massive market for commercial influence biases (look at how previous generations of adtech invaded almost all technology companies). And there's incentive for military and spying capabilities to be buried in the models, perhaps as long-term sleepers. Maybe some organized crime trojans, too, depending which model you pick up.
In this low-trust environment of the current real world, we need genuine open source models, not closed "open weights", and not mindlessly distilling black boxes gifted by sketchy powerful interests.
Governments should be more concerned about the _people's_ personal data instead.
Ban data brokers before you ban distillation.
Yes and there's even a stronger argument that we could REQUIRE frontier model to be open weight / open source.
At the end of the day they were built from data that did not belong to them. So it would be fair that humanity REQUIRES to give back the output of that.
It's a bit like the free software thing: you can still make money from it and providing service to it, but if you build it based on another free stuff the derivative should be free.
Why not do the same for intelligence ?
Distillation is fair use
I can't believe to hear such a wisdom from Garry Tan.
you want smaller models with comparable capabilities. for resource efficiency, market efficiency, environmental conservation.
If it were so easy why aren't the frontier labs doing it themselves?
They should be called speakeasys
I see it as analogous to companies building fiber in the public ROW during the last big infrastructure bubble. Under the Telecoms Act, these companies had to allow competitors to use their fiber at a fair price.
Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!
Garry Tan and Sam Altman recently did this interview together. They seemed pretty friendly with each other during it. Wonder what Sam Altman would say about Tan advocating for OpenAI’s models to be distilled.
Then again this is the same OpenAI that has gotten into legal trouble recently regarding Apple’s IP so who knows
OAI and Anthropic remain the biggest heist ever in our lifetime.
Genuinely fucking crazy we pay money for fast access to autocomplete of stolen human remains.
Freefire
There were comparisons and Muse Spark is so very similar to Fable / Opus... so...
This is all based on the delusion that Chinese labs are mindlessly distilling the frontier.
I would love for a US lab to be at or near the frontier with an open weight model, but it’s going to take some serious elbow grease, and yes some distillation (which btw OAI, anthropic et al, also use distillation of other’s outputs in their training)
Like Gates saying there should be UBI, or Musk saying... well, whatever.
They know it won't happen, so arguing for it is 'effectively free' and purely personal marketing.
A bullshit game played by politicians and wannabes.
Garry also goes to Thiels silicon valley church.
[dead]
[dead]
[flagged]
I agree. The frontier models are based on training data from tons of copyrighted work. Some of that work was obtained illegally, even. They could not exist without strip-mining the commons. The labs have no moral or ethical ownership to the end result, and others should feel free to treat any company-imposed restrictions on their use as invalid.
I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct.
I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.