logoalt Hacker News

toomuchtodo • today at 6:15 PM • 1 reply • view on HN

As someone who red teams with open weight models (GLM 5.3 currently) and a domain focused harness, I would agree with this. No regulation will prevent a sufficiently capable open model, nor stop the development of open models. Alignment efforts are “please don’t stop us from selling stock to the unsophisticated” imho. They aren’t stopping anyone competent and/or capable from model development and training.


Replies

CSSer • today at 9:28 PM

As someone on the development side who just cares a lot about security, I've already witnessed the strict safety bite in the wrong direction too. IT staff are predictably terrified of the blast radius of these tools, so they're inclined to lock down internal enterprise agent subscription permissions so tightly that the tool is not only neutered but also incapable of doing pre-product release, staging testing or even responsible disclosure of vulnerabilities to adjacent teams at the speed at which threat actors will be able to move once it's in production. That's pretty bad. It's not like we don't wonder about the implications of architectural decisions we're making as we do them. We just often lack the bandwidth or resources to fully explore them. And there's a huge difference between "Let's see if I can DDoS this service" and "I wonder if that token is single use or not". If you're going to be tying your system to theirs, it seems like it's a good idea to question the latter and report that to them if it's not prior to shipping it. Forcing the engineer to do that manually helps no one good. I've always thought zero-trust was a little broken in this way, but now it feels really shattered.