alt Hacker NewsIt's 11 THOUSAND searches ON JUST THAT ONE TOPIC in one day when they are used to seeing less than 11 THOUSAND searches in one day TOTAL. None of these systems were built for this type of traffic. The total traffic from these crawlers is many times more than just 11,000. It's more like 11,000 searches on 100,000 topics.
But, per groups of queries, the worst they've seen is 11 thousand, and that's across many libraries. So, some tiny fraction of their users amplified their traffic by 11,000x by using a misconfigured LLM. I've done that a few times without LLMs (who hasn't passed --recursive to wget without remembering the -np?)
Also, if they just turned off all the user-hostile crap, they'd probably get at least 1000x back on their serving costs. How much does it cost to serve a login gate, captcha and rejection screen vs. a ~1MB static file?. What about the human cost where actual users cannot get access to the data they should have access to?
Just now, I ran a Kagi search. Its LLM summarized some page on the internet, but when I went to check the reference, I got a "sorry, humans only" access denied error message. I guess it's good that I could ask the LLM to quote the document verbatim for me in a follow up message, but what fresh hell is this? (Also, the error message screen was clearly longer than the document I'd requested.)