alt Hacker NewsIs cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too.
And they can pull it off because of their reach over the internet with the free DNS.
Replies
Their free DNS is only a small piece of the pie.
The fact that 30%+ of the web relies on their caching services, routablility services and DDoS protection services is the main pull.
Their DNS is only really for data collection and to front as "good will"
no? it takes 10 seconds to check:
> The /crawl endpoint respects the directives of robots.txt files, including crawl-delay. All URLs that /crawl is directed not to crawl are listed in the response with "status": "disallowed".
You don't need any scraping countermeasures for crawlers like those.
I think there's some space being absolutely snuffed by the countless bots of everyone, ignoring everything, pulling from residential proxies, and this, supposedly slower, well behavior, smarter bot.
Like there's a difference between dozens of drunk teenagers thrashing the city streets in the illegal street race vs a taxi driver.
Cloudflare has been trying to mediate publishers & AI companies. If publishers are behind Cloudflare and Cloudflare's bot detection stops scrapers at the request of publishers, the publishers can allow their data to be scraped (via this end point) for a price. It creates market scarcity. I don't believe the target audience is you and me. Unless you own a very popular blog that AI companies would pay you for.
They always have been.
They also use their dominant position to apply political pressure when they don’t like how a country chooses to run things.
So yeah, we’ve created another mega corp monster that will hurt for years to come.
Was it ever not one? They protect a lot of DDoS-for-hire sites from DDoS by their competitors. In return they increase the quantity of DDoS on the internet. They offer you a service for $150, then months later suddenly demand $150k in 24 hours or they shut down your business. If you use them as a DNS registrar they will hold your domain hostage.
If they ever sell or the CEO shifts, yes. For the meantime, they have not given any strong indication that they're trying to bully anybody. I could see things changing drastically if the people in charge are swapped out.
For a long time cloudflare has proudly protected DDoS-as-a-service sites (but of course, they claim they don't "host" them)
No: https://developers.cloudflare.com/browser-rendering/rest-api...