> Many humans do a really bad job at driving, so I'm not sure we should try to emulate that

Agreed, but there are still really good human drivers, who still operate on sight alone. It's more about the upper bound, not the human average, that can be achieved with only sight.