alt Hacker NewsDecompress, scan as you go, discard. Having to read a few hundred GB and scan a terabyte is a nuisance. Not having to write a terabyte is priceless.
Decompress, scan as you go, discard. Having to read a few hundred GB and scan a terabyte is a nuisance. Not having to write a terabyte is priceless.
Could also maintain an in-memory index so that you can go back after the fact and extract individual files.