If you make a change in only one school, you end up with selection effects where interested parents move their children into (or away from) catchment areas based on vibes.

Then you can't really measure outcomes, because the strongest predictor of student performance is parents interest and resources.

You also run into issues with teaching skills and standards, you need a high level of planning and adherence to the supplied plan in order to measure outcomes; otherwise it's just vibes based on individual teachers.

Yeah, maybe we should have given a control group of kids infinite doomscrolling before we gave it to all them.

What you want is done all the time.

What happens a lot:

1) Someone (a researcher, usually) comes in and tries some radical new program in some school.

2) (sometimes) It works! It works great, in fact.

3) This new system or approach or framework gets publicized. This may include dissemination through academic channels, but also (and especially if it's really going to take off) through a kind of reform-grifting network that turns the whole thing into a bunch of stuff that can be sold, for actual money (training, materials, consultants). Turns out being an education researcher pays dick-all, but selling a "system" pays real cash dollars—for many researchers, admin, and curriculum-design folks, getting a windfall from being part of one of these is their most promising path to "making it" before they're old.

4) Some districts adopt the new thing, often with initial pilot programs. Some spend a lot of money doing it.

5) Few of them spend much time considering whether there are material differences between their schools and the one(s) where the system was proven (the experimental program was proven in a troubled inner city school? Surely our middling suburban school can expect similar improvements!). Expertise of and authority granted to the person or persons implementing the system also isn't considered as a factor (one or both are usually lacking, compared with the case or cases on which the promise of the system is judged).

(My personal "here's what to do if you want to fix schools" is "fix our justice/corrections system, worker protections, healthcare, and our social safety net". I think the biggest improvements to our schools would be found there. It's all stuff outside schools. That's why we keep struggling to make headway by monkeying around with schools themselves. It's why more money for schools doesn't help much. That the US finds it basically impossible to do anything constructive about any of those problems is... a sign we can expect not to see any huge across-the-board positive changes in US public school performance any time soon, I reckon)

Meanwhile, within and among districts, individual schools do pilot new programs, et c. All this stuff happens. Does it always happen with everything that turns into a broader reform? No, not always. Is this kind of activity constant, and common, in schools? Absolutely. Frankly it happens way too much (because people are desperate and flailing around to find a path to improvement through school reform, but see above about why I think they are doomed to remain desperate). There's an absolute shitload of process and curriculum churn in schools.

Consider also that while all the above is going on, you have the usual incompetence and principal agent problems you see in any organization. Important tasks are handed to the person an assistant superintendent's having an affair with (god, so common) for whom they invented a paid position. Systems are picked apart and bits adopted piecemeal while ones admin find too uncomfortable or scary are dropped up-front without even trying them, while anyone used to analyzing systems like this can see that the parts their dropping support and are necessary for the success of the parts they're keeping, dooming the reform before it's even implemented. Empire-building happens. Things get hijacked for personal gain. Powerful folks' own inept efforts at breaking into the reform-grift industry get pushed on those under them, as they try to get their own success story to sell. Superintendents or principals fall for obvious bullshit at one of their drinking-and-driving retreats er I mean conferences, because frankly most of them are kinda dumb, and then a whole district gets to suffer for a couple years. Et cetera. Same crap you see in big corporations.

But! Despite all that, lots of people are out there running experiments and reform pilot programs just as you suggest, and for the right reasons, and sometimes even competently. It's just that as soon as it goes past that, it tends to get caught up in all the above. However, even the best-considered reforms that show promise in early experiments and trials are rarely broadly-applicable enough, and familiar enough, and simple enough, and easy enough, and effective enough, to survive that process of wider application without being destroyed. Plus (to repeat, and IMO) I just don't think there are many big wins to be had with educational reform on its own, without working on things outside schools that are resulting in lots of hard-to-educate-in-a-classroom kids.

Every now and then, though, you get a really solid improvement, like, "hey that Whole Language thing that sure seemed to a lot of us to be backwards-ass garbage that really looked like it was making kids worse readers, in-fact, whatever its proponents claimed? Yeah, turns out it is backwards-ass garbage, we can improve reading markedly by knocking that off". (see process and systemic pitfalls outlined above for how it ended up widely in-use in the first place)