Agreed, this just reads like the universities have no idea how to actually serve data effectively. Just bad architecture and lazy or incompetent administrators.
No one is lazy or incompetent, but they built and use systems that are scaled for pre-LLM usage. Now they need to upgrade and they should. They challenge is of course who pays for the extra capacity?
Couldn’t possibly be that serving data to scrapers isn’t a core function for a university or that they can’t pay big tech salaries to draw sysadmins and engineers that know how to handle large scale deployments
I would encourage you to re-think your comments. You just labelled some people "lazy or incompetent" while having barely any knowledge of the situation, or what resources they have or don't have available. Libraries aren't start ups focusing on scalability, their focus is on preserving knowledge and helping academics and the public find it.
If you believe that they should focus on the scalability required to handle bot traffic perhaps you would be best donating money to them to help give them the resources to be able to do that, or volunteering to try to help understand their problem and fix it if you believe that it's a skill issue.
Reframing what you said in a more positive way could be: - I wonder what architecture they used for this and how it could be improved? - I wonder what resource limits they have that prevent them from handling this issue? - I wonder whats unique to a library that limits their ability to do what tech companies do?