So what is the best way of stopping this to happen?

I am doing some experiments in forcing the LLM to use a process.

Start with a brand guidelines, followed by building wireframes, followed by merging the two together.

Think of it like chain of thought but for something visual.