Maybe taking this up one step to Hessian matrix estimation would improve the asymptotic speed over backprop which is already O(inference) per step, by opening the door for higher order methods.