Research on using LLMs and diffusion models to directly predict weights have been going on for a bit now for example https://arxiv.org/pdf/2402.18153