alt Hacker News"I don't know if we have the terminology to talk about this kind of failure mode."
We actually have and is called RISK.
RISK = Probability * Damage.
Applied to the seatbelt event we have a death level damage and a high probability of happening given recent studies, so using a simple belt could easily save you from deadly accidents.
Applied to any unrealistic scenario we have insane level damages but also an incredibly low probability (near 0) so RISK = ~0%
Replies
nme01 • today at 11:12 AM
This framework would be hard to apply to unknown unknowns. That's why in software engineering you'd apply canarying. Then the longer timeframe for potential negative effects, the slower the adoption of a new process should be.
Well that's the point about unknown unknowns though, we actually have no idea what the probability is. But we do know it's not low, that's the only unlikely scenario.
We only know it's a very complicated system with many interlocking dependencies, and based on what we know about complicated systems in general, as well as biology in particular, is that if any one of these unknown dependencies break, the whole system can fail catastrophically.
Therefore the probability that something will go wrong is very high, and the damage could easily be irreparable damage to the species, if not extinction. Does that not give an intolerably high risk?