alt Hacker NewsCurrent evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists.
See https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.
Replies
perching_aix • today at 7:49 PM
Is there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.
This is exactly the problem, unfortunately.
For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0].
Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are.
[0] https://dumps.wikimedia.org/