alt Hacker NewsMoving fast to make launch, we had missed a harness checkout step that would’ve caught a missing comms connection into an FPGA, and it was masked because our redundant comms channel made everything look nominal.
On orbit, we fixed it by pushing an FPGA update and adding software-level switching between the channels to prove the update applied and isolate the hardware path — which worked. Broader lesson, it is possible to design a sw stack capable of making updates to traditionally burned-in components.
> it was masked because our redundant comms channel made everything look nominal.
Hah, this has bitten me often enough I check for it in test suites now - ok, you’ve proven the system works and the backup works, have you proven the primary works? Another in the long list of ways you don’t expect a system to bite you until it does…