This is pretty cool, I wasn’t aware of these types of challenges. How does one even approach this?
Feels to me like it’s similar to dumping a binary with an image, the format being entirely custom.
And/or trying to decode a language or cipher, trying to recognize patterns.
TFA details a solution, it's pretty interesting. Basically the problem was to reverse engineer an absurdly obfuscated and slightly defect MD5 algorithm.
Study math/statistics/ML at a graduate level, to start.