Nice analysis. The simplicity of this indicates that it should be happening during training via regularization.