There is actually some work on doing this; after all, an imagegen model is simply a very large number of images that have been compressed together. Given a stable model and a means of inferring a prompt for an image, you can then generate a base image and store compressed deltas on top of it.

(of course the limit case of this is Samsung moon replacement)