I think it's a natural path of software life that compatibility often stands in the way of improving the API.

This really does seem like a rare thing that everything speeds up without breaking compatability. If you want a fast revised API for your new project (or to rework your existing one) then you have a solution for that with Polars. If you just want your existing code/workloads to work faster, you have a solution for that now.

It's OK to have a slow, compatible, static codebase to build things on then optimize as-needed.

Trying to "fix" the api would break a ton of existing code, including existing plugins. Orphaning those projects and codebases would be the wrong move, those things take a decade to flesh out.

This really doesn't seem like the worst outcome, and doesn't seem to be creating a huge fragmented mess.