A nice source for this might be "The unbearable slowness of being: Why do we live at 10 bits/s?" [1].

A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year [2], according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.

[1] https://www.sciencedirect.com/science/article/pii/S089662732...

[2] Assuming 16 waking hours, and 10^9 bits/s of input that the paper mentions. 16*3600*365*10^9 = 21.024*10^15

Nah. The "huge amounts of data" humans receive are incredibly self-redundant.

Incorporating non-text data into AI training runs has repeatedly demonstrated that for general reasoning capabilities, no data type beats text in value per byte.

Learning something from a lecture, seminar or a book is close equivalent to learning from text/tokens.