alt Hacker Newsmaybe stupid question but how is it possible to not be protected against supercrawlers like this, when you own large amounts of data, specially if all requests are coming from the same server ? Aren't cyberprotection services like CloudFlare work against this kind of "attacks" ?
Replies
Yeah, I can imagine it being a problem if it was from different IP addresses, but surely putting a limit on requests from a single IP would be a straightforward fix? Although maybe until it happens, you don't think about it.
> cyberprotection services
These services cost money. Scrapers also cost money in terms of resource utilization and downtime. There's always a tradeoff.
The ROI of services like Cloudflare bot protection, Datadome, HUMAN, AWS bot mitigation etc. is highly dependent on your income from the data. In the case of publicly hosted freely accessible data it can be hard to justify.
"supercrawlers like this"
then in the post "It came from a single Amazon Web Services (AWS) address. It had read the main page of 18,926 projects, which is practically our entire catalogue, in 26,429 requests over a span of 34 hours."
sounds like some R&D dude running solo to be honest, nothing about that is a supercrawler lol