alt Hacker NewsThis guide is also made from me (or some of the me from a couple years back). I haven't read the whole thing yet and it's probably clearly stated at some point (though one can deduce it with the beginning already) but the surprise for me was that this field is highly statistical. Before starting I had the (very) naive view that it was possible to read the genome as one reads a file and look at what's going on. But the sequencing technics (and accompanying algorithms) only allow to statistically read the genome. So variants/mutations found are only found with a given statistical certainty. If the sample wasn't well prepared for example it could be that this certainty is ultimately not high enough to do a proper analysis/diagnostic. It's a fascinating field (try to watch a video on sequencing by expansion, to feel how sci-fi this field actually is) that is very hard to approach with only high-school biology level and this guide is really well done to sort of bridge this first gap.
Replies
All one needs to do is look at the Claude Science thread here last week and note how many comments were surprised that it appeared to be a statistical/analysis tool.
And the foundations of those statistical approaches are built on heuristics and shortcuts.
For example, sequencing instruments include base quality strings in the output. Base qualities are estimates how likely the instrument got each sequenced base right. But most people don't want to store that much noise, especially when the actual data is highly compressible. So the base qualities get quantized using more or less principled methods that seem to work well empirically.
Read aligners make similar estimates of how likely they got the correct alignment for each read. Those estimates are typically based on simplistic models and a number of assumptions. There are two main components in the estimate. One is based on comparing the primary alignment the aligner chose to the secondary alignments it also found. Another is an estimate that the aligner didn't find the correct alignment, because that part of the sequenced genome is too different from the reference. The latter is obviously handwavy. And the aligner cheats in the former. Because people don't want to wait 10x or 100x longer for better results, the aligner gives up early and estimates how good secondary alignments it might have found if it had actually done the work.
And then there is variant calling. At some point, the state-of-the-art callers were statistical. But then people got better results with neural networks. Or at least the results were empirically better.
Biology is often an intensely statistics-heavy field. A remarkably large part of statistics was developed to study issues in biology, particularly dealing with evolution and ecology.
To expand this a bit, most sequencing methods are exact, and have a low error rate (except nanopores).
But they produce short reads, and because DNA is full of repetitive fragments, it's not always clear where the read came from.
We also have two copies of genes, which also further complicates matters.
The first startup where I worked, developed synthetic long reads on top of Illumina's hardware. We could stitch together 50kbp reads, which really helped with de-novo sequencing.
I'm working on a project in malaria genetics this summer, and I was shocked to find out that the entire analysis toolkit is entirely based on math and statistics (and some non-trivial stuff too, e.g. hidden Markov models to predict CNV). Genotype likelihoods throw an extra wrench into the process, since even basic stuff like predicting allele frequencies requires a maximum likelihood estimator instead of simple counting. This whole area was quite eye-opening, and I'm still amazed that reading billions of base-pairs in DNA sequencing reliably works.
Also gotta shout out to these incredible molecular animations by WEHI: https://www.youtube.com/watch?v=7Hk9jct2ozY