The Navier-Stokes fiasco made me push for local/controlled models very hard. "Can't rule out" that they stole data (backed up by their backdoor offers of sharing credit).
If these companies will steal from deep pockets like Disney or Sony (some of the most infamously litigious copyright trolls to ever exist), they won't think twice of stealing every bit of code you upload to them.
If your code passes through an AI company's servers, you can assume you just gave it to them. In turn, when your competitor tries to copy that new feature you just added, the AI is now trained in exactly how to copy you and eliminate your competitive edge. Unlike your employees, the AI isn't bound by the same rules and even if it were and violated them, your company probably doesn't have enough money to prove it in court (and that's if we somehow reverse some of the stupid "AI is the most transformative use of copyright I've ever seen" judges who have drunk the coolaid).
Most companies could build the compute to run GLM or Kimi models for way less than the potential loss due to IP theft from using third-party systems.
tbh to me if the AI company writes all of your code & your eng don't even review it anymore then… the AI company _controls your company_. maybe that's ok if you make widgets but less ok if you do anything in dev tooling, security, or [insert market they may suddenly decide to compete in].
If I'm not mistaken, they did rule out that they stole data from the mathematicians in question.
I would also add to this, there are ways to use customer data to improve your model outside of just using it as “training-data”.
A simple loophole, use the code to create an RLVR environment where the resultant code is the end goal / max reward. Technically the customer data is never trained upon, but effectively you’re using it. Even better, use the code as a seed to generate synthetic data similar to it and use that synthetic data as rewards in an RLVR model.
Unless you can host the ChatGPT model on your own servers, which I know some enterprises are doing, I don’t think there’s any hope of protecting your data / competitive advantage from these frontier companies. Better to be paranoid, than be commodified by these companies.