alt Hacker NewsI've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with this "icky" feeling. What parts of your work are compromised? Obviously this is far worse with things like binary files and structures than those built on readable text. I have normally just rebuilt my system from scratch in these instances (after fixing the issue of course), but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time. You might not find them until years later.
Replies
This is/was classic market segmentation. Want ECC? Pay up for Xeon (server/workstation CPU)
It's very important IMO that the ECC failures get reported and logged. The DDR5 on-die correction is not true ECC; the correction happens silently and is not reported or logged.
I have DDR4 ECC memory on my home server. Last week I got a crash... logged into my BMC and saw that one of the modules had "uncorrectable ECC". Then I looked over the logs and saw about 200 or so errors for the exact same module but "correctable ECC". They would happen over a short period of time followed by 4 months of no issues. Thanks to ECC and logging it was quick and easy to catch the issue right on the first occurrence, no data corruption. I have 4 modules in total, and only this one was having issues, so clearly a hardware fault.
Now imagine it was a regular non-ECC memory. There won’t be any crashes due to uncorrectable ECC; another 3 or more years would pass before corruption would become frequent enough for the issue to be noticed. At that point, there would be hundreds of corrupted files.
> I think people really downplay the impact of memory issues.
I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...
... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.
We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.
I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).
But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.
It's not happening. Never happened to anyone.
If you (or other) are really concerned, you can use my scheme:
- dsc09874.jpg
- dsc09874-b3-73b0835221.jpg
About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.
I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").
I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).
It's not mutually exclusive with ZFS / a system with ECC RAM.
> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time
I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.
Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".
But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.
> You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512.
And 9 times out of 10 (probably much more) it will be faulty storage, not RAM.
I'd say you are way overestimating impact of no-ECC.