Even relatively small RGB -> depth models are pretty good. Which kind of implies depth is well encoded in the RGB, and adding depth would not really reduce entropy, while costing bandwidth.