logoalt Hacker News

abeppulast Thursday at 6:14 PM2 repliesview on HN

This seems sketchy in that they're giving themselves N*M opportunities to find significant correlations, which they acknowledge -- but with a bias that a significant correlation is not expected to be spurious if it aligns with a claim previously made elsewhere.

> Some of the additional statistically significant findings (e.g., the perception of Rubin’s Vase and openness, the Horse-Seal and intuitive decision making, and the Duck-Rabbit and extraversion and conscientiousness) appear somewhat isolated, are not related to previous research, or claims being made in social media posts and websites. As such, they may be the result of multiple analyses.

I know there's a body of work on "False Discovery Rate", and I think it would be more appropriate to use some of those tools to directly adjust for the number of attempts they're giving themselves.


Replies

nerdponxlast Thursday at 6:21 PM

Correct, FDR control procedures have been part of the statistics literature for at least 50 years, and are relatively easy to implement by hand. There's no excuse.

aqueueaqueuelast Thursday at 8:34 PM

Is this the green jelly beans xkcd?

show 1 reply