Don't we need more than an index of Archive.org because whomever controls the domain could robots.txt these out of existence if they wanted to?

Bourdieu. The field has structure, the structure has logics, the logics shape what counts as a publishable story, a promotable journalist, a credible source, a "balanced framing".