There are fewer obstacles. That's the main reason it works, if you tried flying at 1 m above the ground it would become a lot more like driving, but without the benefit of friction. Flying requires less strict constraints on latency because it happens in straight line segments that are rather longer than the segments that you use when controlling a vehicle.