alt Hacker NewsThe increase in scores is really small. It’s 1.1 percentage points.
My interpretation is, the pandemic is a root cause of lower test scores for many reasons, one reason is that kids started using cell phones way more during the pandemic, and that new stuff on the phone (TikTok, let’s be real) causes lower test scores. Reducing usage during school is addressing a real problem, but it’s one of many real problems, and some are way bigger.
Replies
When you're dealing with large populations (here, the study include 230,065 students--a very large number), even small shifts due to some treatment can be significant. It is very hard to generate top-down policy interventions that shift the mean of a population in significant ways: if this treatment effect (banning phones) is real, 1.1 points represents a very big policy win that can easily be applied elsewhere. The devil is in the details, however: they exclude some recent data based on the pandemic, but baseline off of 2022-2023, which was still in the throes of the pandemic. The data they show looks to have around a 0.5-1 sigma variation in percentile from 2022-2024, so the shift from the baseline of around 1 to 4 definitely looks significant, but it will be interesting to see if sticks over time.
I think there is a typo in the paper, that was carried over to the article. These two sentences appear to contradict one another as written:
> Interestingly, we observe significantly improved student test scores in the second year of the ban (about 2-3 percentiles higher than the year before the ban) when suspensions revert to pre-ban levels.
> Overall, we show that student test scores improved by 0.6 percentiles, with the ban increasing spring test scores 1.1 percentiles in the second year relative to the spring test right before the ban took effect.
Instead, I think the 1.1 percentile gain should be about the first year, and a 2-3 percentile gain by the second year. That is consistent with the graph.
But yes, a fairly small gain. I agree that much of the gain could be recovering from losses during the pandemic. Also the FAST is a new test that started in the 2022-2023 school year, so some of this could also be due to students and teachers adjusting to the new test and improving over time.
I think you're correct, but another piece of this is that students (especially in HS, but probably also in MS) have realized they can accomplish most assigned tasks in a fraction of the time using online resources (whether copies of old tests, agentic AI, or other sites), so they lean on their phone for this. A big piece of the missing equation here is the fact that home PC/laptop use has also been consistently decreasing, in favor of phones & tablets, causing phones themselves to be even more indispensable for youth.
I read a position paper last week suggesting the solution to this is to take a zero tolerance policy in the classroom and move all course testing back to pencil & paper / bluebooks. I would support that (as a parent of two current high schoolers).