alt Hacker News> We found that high-dose supplementation was positively associated with 3 of 11 functions assessed: verbal memory, visual memory, and flexibility or set shift. However, the association with flexibility or set shift was not significant after correction for multiple testing.
They tested 11 different measures. Only 3 of those tests showed significance, but after they corrected for multiple testing the significance of 2 of those disappeared. Only 1 test remained below the significance threshold.
This is on a set size of about 500 children who completed the study, randomized to the two groups.
The only measure that remained statistically significant was visual memory. If you look at Figure 2 it's not even clear that there was a trend toward improvement because everything is so scattered, including a couple measures that were trending worse with higher Vitamin D levels.
This study isn't very convincing. It's a classic p-hacking trick to include many different smaller tests so if one of them pops up as an outlier you can claim that something was significant.
Replies
> It's a classic p-hacking trick
It's a hypothesis seeking study. It just invalidated 8 of them and picked 1 ok-ish candidate to run an actual study in.
The only thing wrong here is there's only one format for submitting a paper.
Where are you getting "after they corrected for multiple testing the significance of 2 of those disappeared"? The text you quoted says, of the three, one disappeared and two remained.
Relevant xkcd
Also it was an asthma prevention study, not cognitive functioning one, adding even more "researcher degrees of freedom".
Doing such side studies is fine in itself, but selling such shakey results as "This study suggests that high-dose vitamin D3 supplementation during pregnancy may be associated with improved cognitive functioning at age 10 years." is a stretch.