A Ph.D. is usually a collection of papers/chapters, so even if a fatal flaw is found in one of the chapters, it does not normally result in "failing the Ph.D." It just means that that particular paper isn't publishable. The bar for getting a Ph.D. is actually quite low; what is truly difficult is clearing the bar in terms of publications for academic positions and later tenure.

I guess there's no tradition for publishing "negative results" in mathematics? By that I mean not proof of something negative, but rather, "we tried this thing for ages and couldn't get it to work, but we couldn't prove that it could never work either".

Probably there are understandable reasons for that... But I think "negative science" is really important, and soft results like "we tried that for a long time and it wasn't very fruitful" are actually very important for progress even if they're poorly attested in the written record. I guess in mathematics they come informally from your thesis advisor...