alt Hacker NewsI wonder if we have an AI LLM equivalent to Moore's Law. Like how often do we expect improvement in this technology and with what timing?
Replies
According to Epoch AI:
> The cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023, or 13× per year. [0]
0. https://epoch.ai/publications/the-plunging-price-of-thought
I've heard tell about 100% of certain types of work being ended in batches of six months. For years. Truthfully, I'm skeptical, but accuracy wasn't prioritized.
reminds me of this blog post: https://campedersen.com/singularity
double the information density every 2 days?
serious bit: if you think about how these smaller models work, at the end of the day it seems that they are now capable of forgetting useless information because they're able to derive it in reasoning allowing models to become smaller at the cost of requiring more reasoning tokens to solve a task.
Hopefully enough runway for an existing model to train the next to be better than itself with absolutely no human intervention.
Yes -> every 18 months they've gotten 90% more efficient for the same level of quality for about 5 years. There's little sign that trend is slowing. If anything, there's reason to believe that System 1 models (plus potentially 1-2-3 workflows) may increase that over the next 3-5 years.
You'll know when the trend stops -> when the intelligence differential between smaller models like 7B starts to grow instead of shrink from 32B models -> that means 7B is getting about as smart as it can get. Then, 32B will follow next, then 70B, etc etc.
We haven't yet seen that at any size AFAIK.