I did. I read the code and saw that MiMo had replaced the zlib calls with a similar bz2 and compression.zstd call, removing the unavailable https://docs.python.org/3/library/zlib.html#zlib.Compress.co.... So I made sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.
This was the main change for bzip2:
@@ -33,19 +34,16 @@ def candidate_lengths(
level: int = 9,
pool: ThreadPoolExecutor | None = None,
) -> list[int]:
- """Compressed length of ``context + seq`` for each seq, sharing the context.
+ """Compressed length of ``context + seq`` for each seq.
- Compresses ``context`` once into a ``compressobj``, then clones its encoder
- state per candidate and feeds only that candidate. Identical to
- ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
- match search over ``context`` happens a single time.
+ Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
+ snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
+ whole block, so every candidate recompresses the full context. Threads
+ still scale because ``bz2`` releases the GIL.
"""
- base = zlib.compressobj(level)
- head = len(base.compress(context))
def length_for(seq: bytes) -> int:
- clone = base.copy()
- return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
+ return len(bz2.compress(context + seq, level))
if pool is not None:
return list(pool.map(length_for, sequences))
I did. I read the code and saw that MiMo had replaced the zlib calls with a similar bz2 and compression.zstd call, removing the unavailable https://docs.python.org/3/library/zlib.html#zlib.Compress.co.... So I made sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.
This was the main change for bzip2: