alt Hacker NewsI went through grad school in a very frequentist environment. We “learned” Bayesian methods but we never used them much.
In my professional life I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods. I’m sure other people’s experiences are different depending on the problems you gravitate towards.
In fact, I tend to get pretty frustrated with Bayesian approaches because when I do turn to them it tends to be in situations that already quite complex and large. In basically every instance of that I’ve never been able to make the Bayesian approach work. Won’t converge or the sampler says it will take days and days to run. I can almost always just resort to some resampling method that might take a few hours but it runs and gives me sensible results.
I realize this is heavily biased by basically only attempting on super-complex problems, but it has sort of soured me on even trying anymore.
To be clear I have no issue with Bayesian methods. Clearly they work well and many people use them with great success. But I just haven’t encountered anything in several decades of statistical work that I found really required Bayesian approaches, so I’ve really lost any motivation I had to experiment with it more.
Replies
A large portion of generative AI is based on Bayesian statistics, like stable diffusion, regularization, LLM as a learned prior (though trained with frequentist MLE), variational autoencoders etc. Chain-of-thought and self-consistency can be viewed as Bayesian as well.
I feel like I'm a polyglot here but primarily a native frequentist thinker.
I've found Bayesian methods shine in cases of an "intractible partition function".
Cases such as language models, where the cardinality of your discrete probability distribution is extremely large, to the point of intractability.
Bayesians tend to immediately go to things like Monte Carlo estimation. Is that fundamentally Bayesian and anti-frequentist? Not really... it's just that being open to Bayesian ways of thinking leads you towards that more.
Reinforcement learning also feels much more naturally Bayesian. I mean Thompson sampling, the granddaddy of RL, was developed through a frequentist lens. But it also feels very Bayesian as well.
In the modern era, we have Stein's paradox, and it all feels the same.
Hardcore Bayesians that seem to deeply hate the Kolmogorov measure theoretic approach to probability are always interesting to me as some of the last true radicals.
I feel like 99% of the world today is these are all just tools and we use them where they're useful.
Given your bias, why bother making this point on a thread about using Bayesian methods where they are applicable? Just seems like unconstructive negativity.
> I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods
Multilevel models are one example of problem were Bayesian methods are hard to avoid as otherwise inference is unstable, particularly when available observations are not abundant. Multilevel models should be used more often as shrinking of effect sizes is important to make robust estimates.
Lots of flashy results published in Nature Medicine and similar journals turn out to be statistical noise when you look at them from a rigorous perspective with adequate shrinking. I often review for these journals, and it's a constant struggle to try to inject some rigor.
From a more general perspective, many frequentist methods fall prey to Lindley's Paradox. In simple terms, their inference is poorly calibrated for large sample sizes. They often mistake a negligible deviation from the null for a "statistically significant" discovery, even when the evidence actually supports the null. This is quite typical in clinical trials. (Spiegelhalter et al, 2003) is a great read to learn more even if you are not interested in medical statistics [1].
[1] https://onlinelibrary.wiley.com/doi/book/10.1002/0470092602