actually, human latency is quite slow and distracted drivers often have 1sec+ latency.
it works because 99% of the time you don't need fast latency because you can accurately predict things.
that's why a standard recommendation is to drive 2+ seconds (time not distance) behind the car in front of you. also why experienced drivers instinctively move their hands/feet into position during tricky moments when they need to cut the latency.
fun exercise, try taking your foot off the gas and hitting the break - slower than you think!!
Distracted drivers having a large latency is obviously not the same thing as humans in general having a large latency...