There is a huge difference between a clearly identifiable, robots.txt-respecting, largely symbiotic search engine spider versus this new AI-driven casual sociopathy of firing shotguns of deliberately masked bots at sites in ways which benefit no one except for, maybe, the bot-owner.

I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.