alt Hacker NewsThere's no way to just download all of it, is there? You have to scrape it slowly? I don't mind doing it that way, but flat downloads would be pretty neat. How do mirrors work? Do they run read-through caches?
Replies
Seattle3503 • today at 8:25 PM
I feel like I am coming in to the middle of a discussion. Towards what end?
➕ show 1 reply
brokensegue • today at 9:03 PM
there's some dumps on archive.org
i worked on a project a long time ago and downloaded a bunch of it
➕ show 1 reply
I have to imagine that's not available to avoid training by commercial entities. https://www.transformativeworks.org/ai-and-data-scraping-on-...