alt Hacker NewsI'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse.
If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation criteria and don't spend over $20." then a lot of ecommerce stores are very very incentivized to be bot friendly.
And of course every "content" site is the reverse, wanting to not waste their bandwidth and hoping to score exclusive paid content deals with any entity that might want to scrape them.
Replies
I've already seen a small amount of transition in that direction. For instance:
Adafruit, last year: They blocked bots very heavy-handedly, including those that were there to gather information at the behest of a present-at-the-keyboard human. This made their products nearly invisible for those who were using ChatGPT instead of Google to find particular things to buy. Even when they were discoverable through forum postings, this added friction meant that I was unable to make a swift comparison betwixt their wares and those of others, and I wasn't buying anything from them because they were making deliberate choices that I felt were hostile to my process.
But that's changed.
Adafruit, at least a week or so ago: The bot was able to get in and find details about the stuff they sell. This lack of friction meant that I was able to make a swift comparison between their wares and those of others. I still bought the particular sensor-chip-on-a-board widget that I wanted elsewhere, but at least Adafruit's offerings were in the running this time.
Very few "user facing" applications will have to deal with real users any longer. From an author's point of view let's admit that it will make things easier.
Yeah, as soon as site operators implement that paid traffic scheme the Cloudflare has been pushing. It's all about being to monetize their compute/content.
My cgit instance is incredibly friendly to bots; I go out of my way to support git's fast "smart" HTTP protocol, and the clone URL is linked on every project's root page. I even included the full clone command on 429 error pages.
The end result was that I got millions of requests daily trying to download every HTML page (every variation of action, commit hash, branch, whatever else cgit allows) and basically no traffic to the proper clone route.
Turns out the bots just don't care about your bot-friendly access method and will instead throw more illicitly-proxied residential IPs at the problem.