it's a global visibility problem, unlike rasterization which is a mostly localized problem.

apart from all the updates and acceleration structures you have to do, you are also battling cache coherency during ray traversal, or as the saying goes: Primary rays cache, secondary trash.