It's just because they don't want to break anything in existing sites, sorta like how Microsoft doesn't generally want to break programs on Windows. So, changes are fairly incremental, and the quality is about what you'd expect from a piece of software that's decades old with no plan for what happens if it got this far.
But what do you do in that situation? If they change the structure too much, then either they make it impossible to upgrade an existing site, or potentially break a whole bunch of things said sites depend on (mostly themes and plugins). And that ease of upgrading is likely what stops a lot of people just migrating away to other solutions.
they broke tons of stuff with Gutenberg/block editor and didn't seem to be bothered by backwards compat
If the existing at-risk functionality is a security issue I feel like that’s a good time to throw the proverbial flag on the play and make it an issue. Letting consumers play host to malware and phishing portals is the worse move.
This isn’t some problem that has developed over time. It was shit from day one.
Even just introducing emulation layers for the old parts and making the new parts opt-in would be better than just giving up and keep on going. Plugin authors can choose the new subsystem as they publish updates or new plugins and themes, and in return they get speed and security guarantees.
And since the WordPress foundation controls the extension marketplace, they can reliably determine which parts of the API surface are in use, or even invest a chunk of money every month to send AI-written patches to plugin maintainers to ease the transition.
There would be so many ways to improve the situation (to the benefit of WordPress maintainers, customers, and ecosystem vendors alike, mind you!), but alas, they are stuck to their ways and will not.