Does it matter? Evolution is the brain's very own "pre-training". Hundreds of millions of years of priors hardwired.
We can do that for AIs too - pre-train on pure low Kolmogorov complexity synthetics. The AI then "knows things" before it sees any real data. Advantageous sometimes. Hard to pick compute efficient synthetics though.
alt Hacker News
Does it matter? Evolution is the brain's very own "pre-training". Hundreds of millions of years of priors hardwired.
We can do that for AIs too - pre-train on pure low Kolmogorov complexity synthetics. The AI then "knows things" before it sees any real data. Advantageous sometimes. Hard to pick compute efficient synthetics though.