alt Hacker News> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.
Replies
Our entire genome would fit on a CD, and much of that is used for metabolism and overall body morphology, not specifically brain structure; even though it almost certainly gives us a whole bunch of stuff (e.g. enough to recognise smiles and thereby allow them to form a positive reward we can learn from, and some approximation of how to sense a potential reproductive partner), it is absolutely not enough to give us a universal language nor intuition about any tool more advanced than a sharp rock and possibly the wheel and fire.
On the other hand, even such basics may well make a huge difference.
Analogy: a cryptic Sudoku puzzle whose only clues are the rules, vs. a normal Sudoku puzzle with a few clues.
That 200M figure needs much more justification as well, humans recieve huge amounts of data through their senses, particularly vision and touch.
I agree with you. Also I really doubt we can so easily tokenize the input to the human brain.
You sound just like Noam Chomsky when it comes to this :) He always talked about the LAD!
If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?
there's also the fact that the brain doesn't process inputs in discrete, feed-forward steps. everything is running continuously and feeding back to earlier layers.
and then there's the abstraction of 'activation' used in neuralnets that is very useful computationally, but omits a lot of the complexity in actual neural signals. you don't get fine-detail in the timing of spikes or the fact that neurons use different types of neurotransmitters.