> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?

That might be true, sure.

> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.

I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.