alt Hacker NewsThe context.
It's impossible to train on raw document > 1MB (I don't know the exact limit, but there is one). You can't vectorize weights (or can you?!) and the output context is limited. So you can't train a big model on it's previous weight or others models weights.
The output context can be "extended" using agents but I doubt anything coherent may be outputted using that technique.
Of course this is the problem. It's amazing that for all the commentary around AI models the number of people who really understand even the basics approaches zero.