alt Hacker NewsI don't know -- my conspiracy theory for many years has been that Apple isn't really very interested in multiple desktops at all, and there's just this one engineer who's made it his life's mission to bring the idea to the masses. But the masses also aren't really very interested in multiple desktops. So every two or three versions, he screws around with the UX and says "surely this time everyone will see the genius!" And they do not.
Personally, I almost never use more than two virtual desktops on macOS, which means most of the changes and bad decisions sail over my head. I don't think I really seriously used more than two virtual desktops on any OS later than BeOS, though -- which IIRC did it really well! -- which I suspect is less a function of macOS doing it badly than a simple function of rarely feeling like I'm cramped on a 27" 5K monitor. (Yes, I know half of you are like "I have three 24-inch monitors and each one needs nine virtual desktops to accommodate all of my windows." That's fine. You do you.)
I mean... I have a 34" widescreen, and yet, I spend most of my time on my laptop screen. So having multiple workspaces is pretty helpful for focus. I think your theory is a sound one. Once they realized their main business is a single screen with ONE APP focused at a time (yes I know, iPad can do more, Duo will also do more), they stopped caring about anything else.