logoalt Hacker News

skybrianyesterday at 4:26 PM1 replyview on HN

Apparently there’s a setting for each website to turn pay per crawl on or off, and they also control pricing:

> While publishers currently can define a flat price across their entire site, they retain the flexibility to bypass charges for specific crawlers as needed. This is particularly helpful if you want to allow a certain crawler through for free, or if you want to negotiate and execute a content partnership outside the pay per crawl feature.

https://blog.cloudflare.com/introducing-pay-per-crawl/

So it’s more like Cloudflare is enabling pay-for-crawl by its customers. There is a centralized implementation, but distributed price setting. This seems more like a market.

It’s unclear to me whether Cloudflare gets a cut.


Replies

angledtoday at 12:50 AM

Market makers always win…

Peak giving-Matt—the-headspins would be if JS stepped and made the crawler market for India.