I think you're looking at the command line and saying a mouse is a solution looking for a problem.

Its not just about manipulating objects spatially. You could do that on a desktop screen with a wii-mote. The other aspect of the form factor is that its an always on, omni-present display, with awareness of the user's surroundings.

This unlocks the ability for apps to be locked to specific locations and contexts, to overlay information on the world, and to, as you stated, manipulate them in a spatial way.

Once the UX itself isn't an uncomfortable hassle, the use cases are really very easy to imagine.