alt Hacker NewsRight, the crazy thing is that much of the groundwork for the “rules-and-heuristics” mode of AI was laid down in the 70s and 80s, long before we had the raw compute power to reliably extract patterns from reality-scale inputs. Those early efforts failed miserably mostly because the rules had to be populated manually and in a ridiculously space-inefficient format (compared to the density of information in model weights).
So yeah, the next stage is models that basically do what humans do: encode causal models of the world in a composable, symbolic form that can be falsified and refined through interventional experiments.
I feel like the talk about "world models" is trying to reach at that, but cast it in different terminology. World model is just domain model, and once you're at domain model, there are multitudes of domains.
Unsupervised learning over domain rulesystems has the potential to let us define really well-defined, scoped models that behave a lot more deterministically and don't colour outside the lines, and reserve their weights for cleanly modeling the domain associations and relationships that matter.
I just asked codex the following question in the middle of my coding prompt:
Answer: As amusing as this may be, I really have no need or desire for my coding model to understand or be aware of ewoks and their relative strengths compared to jawans. Nor do I need it to understand the nuances of the races of middle earth. And prompt response of "I have no idea what you are talking about" to all of these would feel reassuringly scoped.Mixture-of-Experts seems like an attempt to do this - the domain structure being extracted into specific sub-models that are presumably trained on particular domain-associated content - but it feels like this is once again the beginnings of what is possible.