When the damage accumulates across generations the natural selection has opportunity to weed out particularly harmful instances. You can get a feeling for how important avoiding the mitochondrial damage is and how hard it is to mitigate, by looking at how fiercely the reproductive process protects them from aging.