You might want to take a look at https://diffusionillusions.com/ . You don't need a specialized models, little bit of traditional code for enforcing constraints on top of general purpose models can do quite a bit.