alt Hacker NewsWebsites are closing their doors to AI agents
Comments
This is expected. A lot of these companies that are closing their doors make a large portion of their revenue based off of ads and upsells which agents can easily be instructed to ignore, and don't interact with like a human.
Another more blunt way to say it is that people can instruct their agents to look out for their best interests (and while there are cracks they continue to get better at this), and companies hate that.
People will work around it similar to how we've worked around with adblockers for years, and it will just become a game of attrition as these companies work to preserve their moats until they find a way to preserve this revenue stream off of people's agents.
That's perfectly fine by me. I'll read those arguments and buy those things my agents can find on the open web; which is plenty.
Want to argue for a shining new world, but you're on a locked site? I guess we'll have to live with dystopia. Want to sell the greatest widget since the invention of sliced bread? Well, I never found it. Sorry, I bought inferior widgets from your competitor instead. I'm at a physical sales location, but I can't access your spec-sheets? Your competitor the next shelf over doesn't have that problem, I'll go buy theirs.
I hope the irony isn't lost on these companies. Meta is one of the most aggressive companies against any sort of automation, perhaps after LinkedIn.
I'm trying to build a personal social media client to stop the platforms from shoving their feeds down my throat and to consume only the content I care about, so this hits home.
(I just got blocked from LinkedIn for letting an agent set all of the 77 email notification toggles off, which in hindsight was obvious but I wasn't going to do that manually since every toggle has their own separate page, it's a lot of clicks.)
Companies don't want it to be easy to cross shop their competition, the whole premise of Amazon is that it's so easy and simple you don't even check the competition. AI agents are leveling the playing field and helping customers find the best deals, so of course they're being banned
An easy solution is to sell a "web search" subscription add-on that gets distributed to the sites your agents utilize. The script will flip overnight and ASO (agent search optimization) will be born. It could be a form of micropayment system that the internet has desperately needed for over a decade now.
Websites are closing their doors to their users too. I'm unable to register a GitHub account because the third-party bot protection service they use, called DataDome, thinks I am a bot just based on my IP and probably my extension combination, both in Firefox and Chromium. Datadome is also preventing me from accessing some regional sites of Leroy Merlin, Idealista, and a few other major websites in Portugal and Spain whose IT management suddenly decided Datadome is good for them.
A year ago, I contacted DataDome directly and they assured me that "fighting false-positives is their biggest priority" and their filters were fixed for a while until recently. A few days ago, I contacted them again, and now their stance is different: "our support channel is for customers only and this ticket will be closed." However, their customers have no idea that DataDome messes with their end-users, and reaching a tech support person who actually knows about their website using DataDome is unbelievably difficult.
Datadome reminds me of Cloudflare in their early days, when they were breaking UX for many users with Google reCAPTCHA.
Without a convenience moat, commerce will get even more adversarial: membership pricing, surveillance pricing, loyalty perks, varied products to avoid price comparison (like mattresses and Costco).
Fortunately, if AI is really superhuman in capabilities it may still win the battle for the customer and keep prices competitive: aggressive price comparison, alternatives discovery, negotiation, group buying, etc.
We should create a human knowledge base for AI to train on that is completly open so everyone had access. Not just big corporations. Then adopt Ethical Acceleration. https://youtu.be/nV3tBWnZ4wg?si=UX2lKXXPZr01FXYi
I think people who rally against Amazon in this topic assume that the "savings" from much open price competition immediately come back to the consumer or the manufacturer. In my experience, with previous such tech improvement such as 'uber disrupting taxies', this has never been the case.
Both the consumer and the manufacturer/labor/driver/gig-worker always get the short end of the stick.... once AI-agents becomes indispensable part of life.
This is just because agents don’t see ads right? Regardless, this is one of the advantages of software heterogeneity. The cost to detect rises substantially and the false positive rate will eat you alive. My agent buys me things. On Amazon too, yeah. OpenTable is hard to handle but I’m sure it’s fixable.
This is why I think Cloudflare's new service to offer micropayments to agents is going to win and convert the ad ridden web to the ad-free web
It's the latest iteration of "a little X is good, a lot of X is bad".
Email was awesome when it first became widespread. I could cold-email random professors at other universities (I was in school at the time) and get meaningful responses back. Then lots of people (and companies) started cold-emailing, and it stopped working.
Social media is broken if there's nobody there, is great with a medium-sized community, and is broken if everyone is there.
Ad-supported websites can offer all kinds of valuable free services. Up until either (1) they become so ad-infested that they're no longer useful, or (2) all the users block ads.
There was a wave of websites providing APIs for programmatic access a while back. You could script together whatever flow you wanted. If no API was offered, you could scrape the HTML. That triggered a wave of alternate interfaces to sites, skipping their ads, skipping their engagement funnels, separating the value from the cost. So everything got locked up behind captchas and accounts and API keys with limited access.
This outcome was always inevitable. Humans don't understand social contracts, or they do and both sides abuse the hell out of it. We feel entitled to have just the loss leaders without ever buying anything else. When running businesses, we feel entitled to keep adding more and more adds and generally enshittifying the experience. People can take it or leave it, after all.
Agents don't change the underlying structure at all. They just magnify and accelerate parts of the system, and will therefore magnify and accelerate the reactive and defensive responses.
The openweb will go the way of usenet. Bots talking to bots, fetching information for a higher tool.
my prediction: ai agents will have their day, but not on this version of the web. websites will have a separate xml/json/yaml version that agents can peruse without all the ads/js/etc
Weird time to do it IMO, right on the brink of agents potentially becoming the main entity closing purchases for many. If agents succeed at becoming e-commerce middlemen for the masses, I think these decisions will age poorly and there will be some walking back.
In the wake will be the rise of private websites. No social media, no public profiles. Your own private circles.
Right now Claude/Codex are pretty good at following a websites TOS and refusing to solve captchas.
I expect that will no longer be the case in the next year.
what are the ways to do it? considering that now people are looking for ASO (AI Search Optimization), won't this effect them in the long run?
I mean, yeah; agents and AI training data spiders basically operate like DDoS swarms, and it's not like their harnesses implement client-side rate limiting or anything like that. WAFs were designed to block/slow exactly this kind of traffic, so make sense that businesses will use them as designed.
If only it would be possible to "close your doors to AI agents". It's not I'm afraid. Where do you draw the line? What browser is allowed?
For example, I now have to `pacman -S chromium` to buy tickets from Deutche Bahn. They accuse me of being a bot when I use firefox on linux. It's a known problem, and I think stuff like this will only get worse the longer this cat and mouse goes on.
Entire networks. GPT dot became useless as it got systematically blocked by Cloudflare, Google cloud, vercel, etc. At the same time OpenAI blocked solutions like tailscale and others effectively killing the utility of browser control by dot.
This is just like all tech industry disruption.
First they forced their malware bot farms on us, now they want to rent them to us so they can turn the entire web into the dark web.
And eventually rug pull off the major retail sites and just redirect traffic to their own darker sites to scrape commissions, sell direct and keep the change.
I'm sure the teams thinking this up buy their lunches from a ghost kitchen run by illegals that's never had a health inspection,
located in the back of a laundromat with a phony business license and website that only accepts crypto.
This is opportunity.
The whole web is based on selling people's attention, providing free services as "payment" to people. The whole value proposition of AI is to give the services without giving people's attention?
it looks like this is solving itself in the usual and productive way:
> Instead of trying to fight Amazon, the company has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.
they're making deals with the sites so that the agents can work, the sites get compensated. I hope for this "AI can do what it wants if it pays" model to expand further. Cloudflare told us months ago we'd have an option to get paid by AI crawlers. I signed up to join but of course as a nobody I never got a reply.
Like Reddit is shutting down the RSS feeds :(
I mean there are multiple reasons not just the AI agents but that's also a part of it
Finally, we are at the dawn of the next wave of disruption.
Web 2.0 went from serving users to profit maximization. Amazon became unusable. Streaming services fractured and fragmented, with content locked behind numerous paywalls. Ads permeate every surface.
This is not to say the agentic era will end differently. But at the onset of this era, and for the foreseeable future, customer adoption and trust will decide winners and losers.
Instead of closing doors to AI Agents, it is time to start issuing `402 Payment Required` statuses and have the agents pay per use.
[delayed]
I don't want these "agents", aka private tools of greedy mega-corporations, to ruin the world wide web.
I constantly get pestered by "are you a bot" queries.
They ruined the world wide web here. I did not have that in the 1990s or early 2000s.
It is good that agents can't possibly use puppeteer.
When people were uploading music videos on YouTube, media companies fought to the death to prevent that. Then suddenly they 180 and started promoting YouTube videos and begging for views.
This will go the same way, in 2 years 99% of Internet traffic will be AI agents. You close the doors, you stop existing.
Recently I've asked an agent to find me a laptop with particular requirements. You do not let the agent search your site, you will not get the sale.
This annoys me. I can understand stopping mass scraping by big labs, but if I tell my personal instance to try and find the best price for a 40" monitor, it should be able to get details from any site. It's acting as my proxy.
Not only this, but the AI will start picking up the scam websites that allow it to search it.
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
We host a website in Canada (physical server hosting VMs in a Canadian DC) and with the ongoing trade-war we do whatever we can to avoid anything US-based tech such as Cloudflare. So far we use Anubis to protect our services from scraping and it's been a decent experience.
It’s also a “you can’t scrape us but we can scrape you” mentality these companies are getting into.
It really sucks for small businesses. Their websites get absolutely hammered and they can’t afford or don’t know how to stop it. Cloudflare ends up winning as they’re the easiest solution.