In addition to vision, humans have proprioception and kinesthesia, hearing, inner ear receptors for balance, a sense of touch, sense of smell. We use all of these sensory inputs when we drive. And even then humans get into lots of accidents.
Lidar is superhuman vision.
How do you use your sense of smell to help you drive? I can't smell anything outside my car thanks to the air filter.
The real answer is very simple: safe drivers slow down to adapt to poor conditions (including visibility). Lidar might reduce how much it needs to slow down but I don't think it will be eliminated because you can't drive with only lidar.