alt Hacker News> 8MB I believe for high levels
Yep, i found it in the source here:
- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...
- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...
Also, zstd docs say:
> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.
That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
Replies
> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)
2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.
> They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?
It's actually 8 bits: https://www.rfc-editor.org/rfc/rfc8878.html#name-window-desc...
These command line parameters change the maximum the decompressor will allow. It's 128 MiB by default in the command line decompressor; other uses (like the "zstd" content coding for HTTP in web browsers) use a lower limit of 8 MiB (see https://www.rfc-editor.org/rfc/rfc9659.html).