logoalt Hacker News

ruszki • today at 3:55 PM • 0 replies • view on HN

I remember when about 15 years ago, I wanted to make my neural network home work based on proven math. I had to train a model to learn basic number recognition. The math said 100000 neuron is what's needed, so I typed in that, and my laptop immediately froze completely, and died a little later. I learned on that day that we know empirically that a few 10s are enough.

We have the same thing with LLMs. We don't know how they work. We don't understand them. We have only empirical, statistical indications, nothing is proven. We know how the small parts behave, we really understand those, and also the different algorithms. But they whys? No. We're getting better, but still far away.