Actually, given this, we believe that 4 pawns must have been captured to reach 182 bits. So at least 4 pieces no longer need colors. If we store the color mask at the end, I think we can make it variable length, and truncate when no further pieces need colors assigned.

So then we need maximum 182 - 4 = 178 bits

EDIT: Equivalently, we could suffix each non-empty piece in the sequence with an associated color bit