The study https://cepr.harvard.edu/sites/g/files/omnuum9881/files/2026... finds a ≈0.01 standard-deviation difference in test scores the year lockable phone pouches are adopted, with no detectable additional difference in the following year. Certainly nothing to sneeze at from a system-wide perspective, but easily overwhelmed by other factors for any individual student.
Individual test students are going to be hard to predict anyway. Though I think this is compared to the sample standard deviation, and for most students their improvement over time is going to be smaller than the difference between two random students.
That ≈0.01 SD effect is actually 0.0087 +/- 0.0043 (standard error), giving a 95% confidence interval of roughly 0.0003 to 0.0171 standard deviations.
So the effect could be tiny.
Worse, the report says it excludes observations outside the 0.5th and 99.5th percentiles of the 2021–22 fall-to-spring score-change distribution. I'm not sure how this is defensible. Either you trust your measurement instrument or you don't.
If you're using a difference-in-differences design, excluding observations can change what you're measuring, particularly if the exclusions differ between treatment and control groups or over time. Without trimming those observations, the estimated effect drops from 0.0087 SD to 0.0058 SD (and is no longer statistically significant).
The authors preregistered their study, but there's nothing in the report indicating that the trimming of 'extreme' values was part of that preregistration...