alt Hacker NewsThere are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
> But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)