alt Hacker NewsYour CI has to be fully codified, stateless and possible to redeploy with a single command. That's the only way it can remain sustainable. No persistent hidden state, no manual configs (even as an option!) and automatically rebuilt on every release as the new version is deployed.
As a really big bonus, that also makes your CI testable.
In the previous job, we built such a thing: https://smarketshq.com/building-a-reproducible-ci-system-for...
Yes, totally agreed in theory, and it sounds like y'all built a great solution for your use case. But it takes substantial effort and discipline to do something like that at scale.
At some point, you develop complex interdependencies with other systems. You need sophisticated caching for optimum build performance. Techniques like GitOps are unsustainable at a certain number of engineers/commits per hour.