logoalt Hacker News

HarHarVeryFunnytoday at 7:58 PM1 replyview on HN

That an interesting question given how many generations of post-training are being done between base models in some cases. The Gemini flash models are apparently all based on the Gemini 3 base model from a year and a half ago.

It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on it, and the synthetic data they in turn generated, was also suspect? Burn it all down and start over from the pre-tainted data?

It's a bit like the idea of a tainted compiler binary built to backdoor everything it compiles, including future versions of itself.

Still, it seems it would take some Stuxnet level of planning for a rogue model to do something like this, although if RSI goes beyond babysitting the training process (as OpenAI brag about for Astra) to actually designing/constructing synthetic data sets, and managing the training run, then the attack vector is there ...


Replies

customguytoday at 8:36 PM

> it seems it would take some Stuxnet level of planning for a rogue model to do something like this

or maybe it could just.. happen? Posted often but not discussed yet: https://hn.algolia.com/?q=Language+models+transmit+behaviour...

> As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.

show 1 reply