alt Hacker NewsIn a project like PostgreSQL, those scars are reflected in unit tests demonstrating that they’re fixed. It’d be hard to pass its test suite and not be as robust as the original.
Replies
I dunno...I can envision something vibecoded prioritizing passing test suites producing something that does that, but isn't even functional in real-world production. Sort of like in the pre-AI world, where someone claims 'standards compliance' by way of passing compliance test suites, but can't actually interoperate well with other implementations of the standard. YMMV.
> It’d be hard to pass its test suite and not be as robust as the original.
This is not true, even in principle, even for Postgres itself. You'd be right to say that it'd be hard to pass the test suite and not be robust at all to some extent. But even in Postgres, I bet that you can quite easily introduce a change that will pass the whole test suite but reduce robustness compared to the latest release (for a somewhat silly example, add a call to `exit()` on a timer that's longer than the longest duration test in the suite - that will significantly reduce robustness while still passing the entire test suite).
Sure but these scars/tests are from the original implementation. Just because it doesn't have issues there doesn't mean it didn't bring its own set of issues
This is all well and good in theory, but the number of times I've seen tests that don't actually test what they say they're testing is hard to count. Yes even when you encourage the developers to ensure the test fails first and do TDD. Tests help you ship with confidence but there's usually at least a few that are just passing by pure luck.
So no, I wouldn't judge a rewrite as being equal just because it passes the tests. That said, I don't think that means you shouldn't do it. You just have to be pragmatic about it.
Passing a regression test suite only proves that those particular regressions aren't present. It proves nothing about robustness beyond that.
They ought to, but are they? In https://wiki.postgresql.org/wiki/Developer_FAQ I don't see a requirement to provide a regression test for a bug fix.
You immply that a testcase exists for every weird edge case. Especially filesystem and concurrency is things you can barely build test cases for.
Even a 100% test coversge is far away from verifying all behaviour.
Edsger W. Dijkstra:
"Program testing can be used to show the presence of bugs, but never to show their absence!"
Unit tests aren't useful for rewrites, only integration tests are. So there may be missing coverage. Also many things are simply difficult to test (eg performance under very specific conditions)