> Or is the idea that what you write to them and how you use them doesn't matter, it's all up to the model/harness?

Yes, it is. Source: had models inplement complex things from scratch and bullshit regardless of whether it was a one-line prompt or a detailed "SOTA witchcraft magic spells that are guaranteed to work"