alt Hacker NewsWould a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.
Would a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.
I think so.
But developing the heuristics for choosing the appropriate compression algorithm for a stream of data is likely to need a very long time for compression tests of a lot of diverse training data, similarly to the training of a specialized ML model that classifies patterns.
Such heuristics should provide not only algorithm selection, but also parameter selection, when given only some simple input, e.g. the relative importances of compression ratio, decompression speed and compression speed.