Oh, it gets extremely annoying when you try to use those kinds of indirect, noisy, smeared, nonlinear, relative measurements for absolute sensing.

The power of modern AI robotics is that, much like the biological mess it's trying to replicate, it doesn't care that much about absolute sensing.

Imperfect sensors fused by a high performance control model with a good foundation of world dynamics and self dynamics? Gets you impressively close to "perfect sensors" in practice. Being able to take a big bundle of noisy nonlinear ambiguous measurements, and implicitly reference them off each other and over time to get to a usable control loop is powerful.

It's the kind of power human nervous system wields to force the horrid mess that is the human body to perform.

For "detect when you've accidentally picked up two items from a bin instead of one", a wrist camera is a far more natural sensor than a bundle of actuators - especially for lightweight items. Wrist cameras are awesome. But even a bundle of actuators, if fused well enough, can often do it.

The problem is that designing algorithms to sensor fuse to this level by hand is pretty much a no-go. The kind of sensor wrangling hell I would only wish upon an enemy. Which gets us right back to general purpose robotics being an AI problem at its core.