alt Hacker NewsIf those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?
Replies
Well keep in mind we receive from our mothers an entire functional cell. DNA is a bit overrated.
No room - or is there?
FractalDB proves that "synthetic data useful for training AI" can be produced by a very small program.
There is "no room" in DNA for a full dataset, or a full snapshot of model weights. But there's plenty of room for "this one little trick cuts the data required to learn a desirable thing X in half" reproduced thousands of times for different X. Low Kolmogorov complexity, but high performance gains.
The data is massively compressed and pruned. Where are the input tokens in an LLM?
Tokens being passed on would be what, you close your eyes and experience your ancestors lives?
DNA is probably the seed for the random initialization.
This is such a beautiful question. It looks like a miracle that a brain, let alone a whole body, can grow from such little data. Michael Levin has been doing a lot of research on this, you should check him out.