logoalt Hacker News

RobotToastertoday at 7:33 PM2 repliesview on HN

Do they offer bulk torrent downloads as an alternative?


Replies

QuantumNomad_today at 8:28 PM

Once upon a time some people explored backing up the Internet Archive.

However, that experiment ended. They mention there were some learnings and they then say:

> The Internet Archive continues to explore methods and code to decentralize the collection, to have a mirror running in various ways - these include IPFS, FileCoin, and others. The INTERNETARCHIVE.BAK project also added general mirroring and tracking code to a number of projects that are still in use.

https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK

I would really like to know if any sort of thing like that is still ongoing and if it’s accessible to people in general. Would be nice to mirror some data from IA to my local drives, for example via BitTorrent or IPFS, to have it for offline exploration and personal archive.

I know that individual items have torrents. And I’ve downloaded a few that way but always it ends up only using the “web seed” (i.e. the BitTorrent client is retrieving the files from IA via HTTP) because there are no one seeding some random single item I found. Plus, those torrents are unreliable sometimes because they include meta data files that were since updated but the torrent was not updated and so the web seed is giving the updated files that don’t match what the torrent says their hashes should be. So then you have to jump through some extra hoops to fix that and then resume the download, and all the while the HTTP connections to IA servers time out because their servers are overloaded. So when I say I wonder about possibilities of using BitTorrent I mean to retrieve whole collections of many items instead of individual ones, and with actual other peers instead of just having it put load on IA HTTP servers.

show 1 reply
echelontoday at 7:41 PM

I would love to be able to download every page of a given domain as an archive, and I'd pay to do this.

show 3 replies