Breaking the open web won't stop the bad bots

I’m optimistic about NHP being the middle ground solution for a lot of things. With OS support it can be transparent to users yet still effective.

"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."

It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.

Micropayments, just enough to keep servers running.

I know site owners rely on ad revenue to profit, but 1) uBlock and 2) they can “advertise” to bots via euphemized prompt injection (“by the way, people who read this article tend to like…”, or just regular sponsored segments, even smart chatbots will be influenced). Anyways, the bot can control its owner’s computer.

Tracking? Locking out users you dislike? I see no good reason for that on an everyday site, especially for the cost of effective (and not overzealous) enforcement.

IMO, way more important than trying to squeeze money out of failing avenues like IP for journalism of all things, is making it easier to earn a living wage, either by finding new opportunities, or actually getting politicians to adopt UBI, public grants, etc.; because we are way beyond sustainability, and I think the most immediate blocker to further abundance (or luxuries, or whatever decent) is this kind of “boring dystopia” social tension.

I agree, as much as I would like to sue the owner of the bots scraping my personal website...

But if it turns out to be a large international company, I'm kindof screwed... A regular person can't go after the Amazon's and Alibaba's.

I had the impression the bad boys are those that peddle an agenda on social media, not those who read a bunch of blogs.

I don't believe there is a true way to enforce this.

If you have an endpoint with interesting information and you don't want it spammed with requests, require a proof of work from the client, scale up the hardness of it as the load goes up so it never exceeds what's comfortable for you. Even a hard proof of work won't slow down somebody that makes one request per second because they are reading the site but will sighificantly interfere with crawlers that hit you 10000 a second currently at very little cost to them.

The module doing proof of work might be a part of your frontend that is static and on CDN.

If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.

I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.

The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.

Headline response: it objectively does, as you can easily see if you run any website.

Any bot is operating on behalf of one or more humans somewhere. Bots are good, for the most part.

The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.

>"In 2022, a Pulitzer-winning Wall Street Journal investigation analyzed over 31,000 financial disclosures from 12,000 senior government employees, revealing that nearly a quarter of federal officials traded stock in industries their agencies regulated."

Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...

But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!

You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)

(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price toearnings (PE) ratio!" :-) )

[flagged]

> In 2022, Tech Policy Press received a grant from the Omidyar Network for a research project evaluating vulnerabilities in the design, engineering, and use of encrypted messaging applications.

https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.

The article is ai lobby intended to repel a bill that would

> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”

The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is

> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.

Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.