> Folks who are okay with LLMs think the regression test suite is the spec and is the guarantee of stability. How else can it be? If you are depending on some behavior not covered in the regression tests, how do you know the next minor release won't break you?

I think this deserves a real response.

First, an analogy. I drive along a cliff with no guardrail. How do I not drive off the cliff? By knowing how to drive. Sometimes people mess up and drive off the cliff

In practice people are operating Postgres as a machine, more than an abstract spec. Minor releases exist, but the changes are made by people operating the machine and who have a fear of making the machine break.

There are also performance characteristics that are part of an informal spec. While you can definitely write regression tests on performance, theres loads of value in stability of internals because people who have problems look into the machine and discuss it.

The internals might change, but there’s a lot of friction. So… you can have a lot of informal knowledge about.

If everyone working on a software stack is bathing in this informal knowledge, then the decision making is based on that. Things like what is meant to be in a minor or major release is understood. And people make those judgement calls.

After all, even if you have regression tests if you’re making changes you’ll need to write new tests! How do you know your new tests are right? That the new behavior is right?

PG is the entire machine. Fortunately we have version numbers, migration strategies, etc. But “here’s a new binary that passes the test suite and… maybe changes everything not covered by tests maybe doesn’t”. Why bother suffering when there are totally reasonable gradual rewrite strategies?

And of course… every bug in the world… got through despite a test suite! What software out there doesn’t have bugs?

And if the behavioral difference in the rewrite does affect people… I guess that’s something right? “Oh this isn’t a bug because it wasn’t covered in regression tests” is not that tenable.

> If you are depending on some behavior not covered in the regression tests, how do you know the next minor release won't break you?

Decades of intrinsic knowledge. Which a rewrite lacks.

imho the issue is the language used. AI rewrites are cheap and cheap exercises require a great deal of scrutiny and proof of correctness. Simply using regressions test lacks the intrinsic knowledge of a decades old codebase stuck in developers minds.

Claiming a rewrite is better because it passes all tests is a flex, a new version requires a boatload of evidence for people to accept it as an improvement and not just "passes tests, written in rust via LLM so it must be better". Run it in production for a year in a sufficiently large system and you might be somewhere.

I think what folks want, but aren't quite able to articulate, is an ongoing community and effort that indicate a project will be healthy and maintained. We want to be able to rely upon the software that we are choosing to use.

Regardless of the technical choices, whether Rust is better or worse, whatever -- pgrust popped into existence thanks to one person driving an LLM through 7000 commits in ~2 weeks. It produced something that passes the regression tests. Even as an LLM-sceptic, I think that's amazing.

From that point, though, it appears to have been completely abandoned. There hasn't been a commit in a month, other than a brief tweak and a note that an as-yet unpublished version that's even betterer is in the works. IDK. I don't think we've acclimated to the shock of the change LLMs create, but if the outcome is a forest of exciting new projects that have a bus factor of 1 and little to no collaboration, I think that's a disservice to this profession.