alt Hacker NewsGood discussion, but this conclusion:
> I don’t know. Maybe a little.
is not the right way to interpret the null result (insufficient evidence to reject null hypothesis that it has no effect) because the prior for supplements that people are trying to sell you is so low. In the absence of clear, strong evidence, you should assume the the whole thing is just an example of motivated reasoning. People want nootropics to be real, and other people really want to sell you readily available powders by claiming they have nootropic properties. In that environment, they were always going to trying to concoct a similar narrative about some supplement, and it just happened to be creatine. Those efforts were always going to result in a handful of "positive" studies that turn out to be non-reproducible, maybe because of p-hacking, maybe because of publication bias, maybe because of outright fraud. This is what the literature always looks like for stuff that just doesn't work. If it did work - if the effect size was large enough that you could personally detect it in your own life - then the papers would be trying to put error bars around the effect size, not trying (and failing) to barely distinguish it from a placebo.
“If your experiment needs statistics, you ought to have done a better experiment.” - Ernest Rutherford
Replies
I think this is quite different because (1) creatine has been proven to be completely harmless for people of all ages absent some rare thing like a kidney disorder and (2) it has a bunch of clearly measurable positive effects.
So this is more like trying to determine whether ginseng is good at keeping away colds or whatever. Maybe it does (actually quickly searching this, it definitely does but let's pretend it's as folksy as it sounds for a minute) but even if it doesn't then it's harmless and has a bunch of other effects, so when one takes it - the whole possible cold prophylaxis can just be seen as an added perk.
If you're being academically rigorous about it, sure, but I think it's fine as an informal conclusion.
The null is surely different in this case because we know that supplemented creatine goes to the brain and the brain uses it, which is definitely not a claim all supplements can make.
The fact that there's a biologically plausible mechanism plus mixed statistical evidence makes "maybe" a perfectly reasonable conclusion.
You are essentially trying to say that most studies regarding supplements are not valid
mandatory XKCD https://xkcd.com/882/ "Jellybeans cause acne"
I think the way you interpret the null result is downstream of considering creatine as "a supplement". You can make the null result say anything by changing your prior about creatine or supplements. That's the issue with priors.
You also seem to reject the possibility that some things help just a bit. The author has another article in the same vein about "things that can help maybe a bit but the evidence we have doesn't really help detecting small effects" https://dynomight.net/vitamin-d/
Heavily agreed. The supplement industry has such a low level of reliability that it has really soured me on so many practices people have. And I flat out do not understand how some of the supplement stores survive.
Okay, but it kind of sounds like you want to nanny-state the writer's language.
Their conclusion seems like an absolutely fair interpretation of what they saw, and it also strikes me as dangerous -- perhaps more dangerous -- to suggest "writer genuinely believes, or doesn't believe, a thing, but they should say it differently from their own interpretation because..people are dumb?" Something like that.
What I'm getting from all this is that we have an armchair expert explaining details about something they're not an expert in, and we have you reviewing the way they present this information, saying their very weak hedge wasn't strongly negative enough. Based on priors about the industry.
Basically, HN doing HN things - having a good deep think about topics outside their realm of expertise, then debating the minutiae. I'd just rather trust someone who has spent their career researching this stuff, thanks (unless that's what you do - in which case please tell us more).