logoalt Hacker News

networkedtoday at 10:24 AM0 repliesview on HN

I did. I read the code and saw that MiMo had replaced the zlib calls with a similar bz2 and compression.zstd call, removing the unavailable https://docs.python.org/3/library/zlib.html#zlib.Compress.co.... So I made sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.

This was the main change for bzip2:

  @@ -33,19 +34,16 @@ def candidate_lengths(
       level: int = 9,
       pool: ThreadPoolExecutor | None = None,
   ) -> list[int]:
  -    """Compressed length of ``context + seq`` for each seq, sharing the context.
  +    """Compressed length of ``context + seq`` for each seq.
  
  -    Compresses ``context`` once into a ``compressobj``, then clones its encoder
  -    state per candidate and feeds only that candidate. Identical to
  -    ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
  -    match search over ``context`` happens a single time.
  +    Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
  +    snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
  +    whole block, so every candidate recompresses the full context. Threads
  +    still scale because ``bz2`` releases the GIL.
       """
  -    base = zlib.compressobj(level)
  -    head = len(base.compress(context))
  
       def length_for(seq: bytes) -> int:
  -        clone = base.copy()
  -        return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
  +        return len(bz2.compress(context + seq, level))
  
       if pool is not None:
           return list(pool.map(length_for, sequences))