logoalt Hacker News

tourmalinetaco • today at 10:24 AM • 0 replies • view on HN

What must we “contain”? These are not autonomous intelligences with desires or free will. They require prompting and to be turned on beforehand, otherwise they’re glorified matrix math. They can only continue processing as long as they receive power, and we decide that.

They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.

The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.