alt Hacker NewsI don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.
Replies
In a lot of spaces, this is actually preferable.
Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc.
Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model.
Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained.
---
Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value).
I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please.
Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store.
If I'm doing auto-repair... "cars for dummies (version 3)". etc...
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet.
Self-learning/improving would be even better but that's still a long way to go.
At 125B + 51B I'd expect it to have some degree of world knowledge, clearly in the middle between small models like qwen 27B, and huge trillion parameter models.
For world knowledge, you'd want it to find and reference the source material to be sure. At that point, it doesn't matter if the knowledge is embedded.