>imagine whole movies shot in this format. The fly-on-the-wall fantasy of movies, without being locked in place for the whole shot.
That would be terrible. Framing is the major expressive feature in cinematography, and any interactive format needs a lot more thought put into it than just having a free camera. Literally the worst of both worlds.
Light field video streaming is a thing, however it's pretty niche. OTOY pioneered holographic codecs and light field videos with some degree of freedom and a sense of depth more than a decade ago.
Not terrible, per se, but not a movie. There are other formats that fit the task better like immersive theater and theater in the round.
This has been extensively explored with VR and games, though.
It wouldn't work for a movie, but it might be good for a concert video, stage play, dance, or sports, where you're capturing something happening in a defined space. You could pick a different seat in an auditorium.
Also consider connecting a virtual space with a real one, so it looks like something happening on a virtual stage that's connected to a room in your house.