alt Hacker NewsWhat does it really mean to properly understand p-value? What I remember is that if the p-value is less than 0.05, the research result is considered statistically significant. That's about as far as my memory goes. I know that's actually a misunderstanding, but that's how most people understand it. I'm not sure how much I need to know to say I truly understand it.
Replies
Agree. I think it's more important to understand statistical fallacies (selection bias, regression to the mean, survivorship bias, etc). Those are extremely common trip wires but to recognize them you don't need to memorize formal definitions.
It is mumbo jumbo (also people hearing “significant” treat this as “effect/change is large and important” which is in no relation to the actual amount of change).
Low p-value basically means how surprising your data would be if there were actually no effect (ie less than 5% of the time you’ll get this due to randomness if there is no change — which is rather impossible)
Sample size matters heavily. With more observations, estimates become more precise, so increasingly small differences can become statistically significant. With a large sample, you can therefore get a tiny, practically meaningless effect with a very small p-value.
Eg effect of $1 can be statistically significant (not random) which does not matter in practical terms if average is like $10000.
So the key point here is not only to look at the p-value but also at an actual change. If a drug gives you only 0.01% more hair, it doesn’t matter to you that it is guaranteed.
Fair play to you for answering, but that’s not what a P value is! A p value is the chance of getting a result greater than the result you actually got, _assuming you are testing a hypothesis_.
I’m also pretty sure I would fall in the camp of saying “nope don’t understand P values” as I can’t remember anything else about them.
P-values are almost always taught poorly, but it's not actually that difficult of a concept. I took statistics in high school, again in undergrad, and it wasn't until the third time in grad school that it actually made intuitive sense (thank you Julia Yang!). When you're testing a hypothesis in statistics, it's easier to formulate a "null hypothesis" which is the opposite of what you're testing, and then try to disprove that null hypothesis.
A p-value is the probability, assuming the null hypothesis is true, of obtaining a result at least as extreme as the one actually observed.
Put differently: if the null hypothesis were true, then for p=0.05 you'd see <things at least as far from the test statistic as what you just observed> at most 5% of the time.
Put differently again: If the null hypothesis you are testing is true, then for p=0.05 random sampling would not return an observation as far from the test statistic as you just observed, 95% of the time.
Say you go out and you see a sports bar and the FIFA world cup is on. France vs Sweden is playing on the TVs in the bar and the place is packed with fans of both countries. You look at a few people (wearing the national football colours) and you think to yourself "the French fans seem shorter than the Swedish fans". You decide to run an experiment to test this hypothesis. Lucky for you at half-time exactly n fans of each side agree to let you measure their height. You do this and of your sample of n fans, the French are 2cms shorter on average than the Swedish fans.
Now: the p-value is the probability of you getting a result at least this extreme assuming the null hypothesis (that there is no difference in average height between the general population of French and Swedish people).