It's a cognitive tax on all the parties involved. It works on buyers (websites that pay for it) much the way that llm's work on their users, by working up to the point at which the buyers can't discern it working.

As a user (bot or flesh) you have to jump through these hoops, and to the website operators they get fewer complaints when users (and bots) are able to contort themselves in predictable ways. Versus you'd perceive as a website operator, if your blocking worked at all, that you'd risk seeing a precipitous dropoff of bots if you did things in bespoke ways, as both bots (true positives, among dd and cf survivors) and people (false positives) get jammed.

The net effect in this model would be that such things as cloudflare and datadome take money from websites in order to block "dumb" bots and enfranchise "smart" bots to the point of creating a market for bot services. People that want to bot have to use AI or go to bot service providers.

For website operators that's a worthwhile exchange for dollars, in that dumb bots can cause a lot of harm in wasted compute. Bots are typically dead end users who won't buy ever anyway, or merely inefficient way of doing business if they scrape excessively. And many do scrape excessively.

"reform school for bots" is a good mental model. For this to work, the "reforms" have to be consistent so that bots can conform to them.

It's a peculiar equilibrium and I am not sure that it's all that stable as an equilibrium. Would be a good subject for a formal treatment, game theory and economics.

Mostly economies of scale. They have enough data to do the statistics to identify bad actors, and enough bandwidth and servers to facetank the request volume.

Would love if they did but most end up just reselling cloudflare. Example WPEngine.