Would PPO learn quickly from the hand engineered solution? Is PPO getting a terrible init?
It's interesting to see how tolerant of tedium the models are.