You can't really do a whole lot of inference in 16ms on consumer hardware. Not to say that inference isn't useful in realtime graphics, DLSS has proven itself well enough, but that's a very small model laser-targetted at one specific problem and even that takes a few milliseconds to do its thing. Fitting behemoth generative models into those time constraints seems like an uphill battle.