The author makes a comparison to Haskell, which I think might be a little misleading.
Haskell is a little more complicated to learn but also more expressive than other programming languages, this is where the comparison works.
But where it breaks down is safety. If your Haskell code runs, it's more likely to be correct because of all the type system goodness.
That's the reverse of the situation with Bayesian statistics, which is more like C++. It has all kinds of cool features, but they all come with superpowered footguns.
Frequentist statistics is more like Java. No one loves it but it allows you to get a lot of work done without having to track down one of the few people who really understand Haskell.
I went through grad school in a very frequentist environment. We “learned” Bayesian methods but we never used them much.
In my professional life I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods. I’m sure other people’s experiences are different depending on the problems you gravitate towards.
In fact, I tend to get pretty frustrated with Bayesian approaches because when I do turn to them it tends to be in situations that already quite complex and large. In basically every instance of that I’ve never been able to make the Bayesian approach work. Won’t converge or the sampler says it will take days and days to run. I can almost always just resort to some resampling method that might take a few hours but it runs and gives me sensible results.
I realize this is heavily biased by basically only attempting on super-complex problems, but it has sort of soured me on even trying anymore.
To be clear I have no issue with Bayesian methods. Clearly they work well and many people use them with great success. But I just haven’t encountered anything in several decades of statistical work that I found really required Bayesian approaches, so I’ve really lost any motivation I had to experiment with it more.
> I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods
Multilevel models are one example of problem were Bayesian methods are hard to avoid as otherwise inference is unstable, particularly when available observations are not abundant. Multilevel models should be used more often as shrinking of effect sizes is important to make robust estimates.
Lots of flashy results published in Nature Medicine and similar journals turn out to be statistical noise when you look at them from a rigorous perspective with adequate shrinking. I often review for these journals, and it's a constant struggle to try to inject some rigor.
From a more general perspective, many frequentist methods fall prey to Lindley's Paradox. In simple terms, their inference is poorly calibrated for large sample sizes. They often mistake a negligible deviation from the null for a "statistically significant" discovery, even when the evidence actually supports the null. This is quite typical in clinical trials. (Spiegelhalter et al, 2003) is a great read to learn more even if you are not interested in medical statistics [1].
The evidence "actually supports the null" over what alternative?
In a Bayesian analysis, the result of an inference, e.g. about the fairness of a coin as in Lindley's paradox, depends completely on the distribution of the alternative specified in the analysis. The frequentist analysis, for better and worse, doesn't need to specify a distribution for the alternative.
The classic Lindley's paradox uses a uniform alternative, but there is no justification for this at all. It's not as though a coin is either perfectly fair or has a totally random heads probability. A realistic bias will be subtle and the prior should reflect that. Something like this is often true of real-world applicaitons too.
Thank you. The main problem with Bayesian statistics is that if the outcome depends on your priors, your priors, not the data determine the outcome.
Bayesian supporters often like to say they are just using more information by coding them in priors, but if they had data to support their priors, they are frequentists.
If they were doing frequentist inference they wouldn’t be using priors at all and there is nothing frequentist in using previous data to construct prior distributions.
Not true. In frequentist statistics, from the perspective of Bayesians, your prior is a point distribution derived empirically. It doesn't have the same confidence / uncertainty intervals but it does have an unnecessarily overconfident assumption of the nature of the data generating process.
I agree Bayesian approaches to multilevel modeling situations are clearly quite useful and popular.
Ironically this has been one of the primary examples of, in my personal experience, with the problems I have worked on, frequentist mixed & random effects models have worked just fine. On rare occasions I have encountered a situation where the data was particularly complex or I wanted to use an unusual compound probability distribution and thought Bayesian approaches would save me. Instead, I have routinely ended up with models that never converge or take unpractical amounts of time to run. Maybe it’s my lack of experience jumping into Bayesian methods only on super hard problems. That’s totally possible.
But I have found many frequentist approaches to multilevel modeling perfectly adequate. That does not, of course, mean that will hold true for everyone or all problems.
One of my hot takes is that people seriously underestimate the diversity of data problems such that many people can just have totally different experiences with methods depending on the problems they work on.
These days, the advantage is that a generative model can be cleanly decoupled from inference. With probabilistic languages such as Stan, Turing or Pyro it is possible to encode a model and then perform maximum likelihood, variational Bayes, approximate Bayesian inference, as well as other more specialized approaches, depending on the problem at hand.
If you have experienced problems with convergence, give Stan a try. Stan is really robust, polished, and simple. Besides, models are statically typed and it warns you when you do something odd.
Personally, I think once you start doing multilevel modeling to shrink estimates, there's no way back. At least in my case, I now see it everywhere. Thanks to efficient variational Bayes methods built on top of JAX, it is doable even on high-dimensional models.
A large portion of generative AI is based on Bayesian statistics, like stable diffusion, regularization, LLM as a learned prior (though trained with frequentist MLE), variational autoencoders etc. Chain-of-thought and self-consistency can be viewed as Bayesian as well.
I feel like I'm a polyglot here but primarily a native frequentist thinker.
I've found Bayesian methods shine in cases of an "intractible partition function".
Cases such as language models, where the cardinality of your discrete probability distribution is extremely large, to the point of intractability.
Bayesians tend to immediately go to things like Monte Carlo estimation. Is that fundamentally Bayesian and anti-frequentist? Not really... it's just that being open to Bayesian ways of thinking leads you towards that more.
Reinforcement learning also feels much more naturally Bayesian. I mean Thompson sampling, the granddaddy of RL, was developed through a frequentist lens. But it also feels very Bayesian as well.
In the modern era, we have Stein's paradox, and it all feels the same.
Hardcore Bayesians that seem to deeply hate the Kolmogorov measure theoretic approach to probability are always interesting to me as some of the last true radicals.
I feel like 99% of the world today is these are all just tools and we use them where they're useful.
Given your bias, why bother making this point on a thread about using Bayesian methods where they are applicable? Just seems like unconstructive negativity.
I think Rafael Irizarry put it best over a decade ago -- while historically there was a feud between self-declared "frequentists" and "Bayesians", people doing statistics in the modern era aren't interested in playing sides, but use a combination of techniques originating in both camps: https://simplystatistics.org/posts/2014-10-13-as-an-applied-...
I agree... I feel like "The Elements of Statistical Learning" was possibly one of the first "postmodern" things where "well, frequentist and Bayesian are just tools in the toolbox, we now know they're not so incompatible."
After Stein's paradox it became super hard to be a pure frequentist if you didn't have your head in the sand.
As a data scientist, I find applied Bayesian methods to be incredibly straightforward for most of the common problems we see like A/B testing and online measuring of parameters. I dislike that people usually first introduce Bayesian methods theoretically, which can be a lot for beginners to wrap their head around. Why not just start from the blissful elegance of updating your parameter's prior distribution with your observed data to magically get your parameter's estimate?
I think it would be interesting if frequentist stats can come up with more generative models. Current high level generative machine learning all rely on Bayesian modeling.
I'm not well versed enough, but what would a frequentist generative model even mean?
The entire generative concept implicitly assumes that parameters have probability distributions themselves that naturally give rise to generative models...
You could do frequentist inference on a generative model, sure, but generative modelling seems fundamentally alien to frequentist thinking?
I am more familiar with Bayesian than frequentist stats, but given that they are mathematically equivalent, shouldn't frequentist stats have an answer to e.g. the loss function of a VAE? Or are generative machine learning inherently impossible to model for frequentist stats?
Though if you think about it, a diffusion model is somewhat (partially) frequentist.
I guess you have me thinking more... things like Parzen window estimators or other KDEs are frequentist...
But while it's a probability distribution, to a frequentist they are estimating the fixed parameters of a distribution.
The distribution isn't generative, it just represents uncertainty - and I think that's a bit of the deep core philosophical divide between frequentists and Bayesians - you might use all the same math, but you cannot possibly think of it as being generative.
> In Bayesian statistics, on the other hand, the parameter is not a point but a distribution.
To be more precise, in Bayesian statistics a parameter is random variable. But what does that mean? A parameter is a characteristic of a population (as opposed to a characteristic of a sample, which is called a statistic). A quantity, such as the average cars per household right now. That's a parameter. To think of a parameter as a random variable is like regarding reality as just one realisation of an infinite number of alternate realities that could have been. The problem is we only observe our reality. All the data samples that we can ever study come from this reality. As a result, it's impossible to infer anything about the probability distribution of the parameter. The whole Bayesian approach to statistical inference is nonsensical.
Haskell is a little more complicated to learn but also more expressive than other programming languages, this is where the comparison works.
But where it breaks down is safety. If your Haskell code runs, it's more likely to be correct because of all the type system goodness.
That's the reverse of the situation with Bayesian statistics, which is more like C++. It has all kinds of cool features, but they all come with superpowered footguns.
Frequentist statistics is more like Java. No one loves it but it allows you to get a lot of work done without having to track down one of the few people who really understand Haskell.
In my professional life I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods. I’m sure other people’s experiences are different depending on the problems you gravitate towards.
In fact, I tend to get pretty frustrated with Bayesian approaches because when I do turn to them it tends to be in situations that already quite complex and large. In basically every instance of that I’ve never been able to make the Bayesian approach work. Won’t converge or the sampler says it will take days and days to run. I can almost always just resort to some resampling method that might take a few hours but it runs and gives me sensible results.
I realize this is heavily biased by basically only attempting on super-complex problems, but it has sort of soured me on even trying anymore.
To be clear I have no issue with Bayesian methods. Clearly they work well and many people use them with great success. But I just haven’t encountered anything in several decades of statistical work that I found really required Bayesian approaches, so I’ve really lost any motivation I had to experiment with it more.
Multilevel models are one example of problem were Bayesian methods are hard to avoid as otherwise inference is unstable, particularly when available observations are not abundant. Multilevel models should be used more often as shrinking of effect sizes is important to make robust estimates.
Lots of flashy results published in Nature Medicine and similar journals turn out to be statistical noise when you look at them from a rigorous perspective with adequate shrinking. I often review for these journals, and it's a constant struggle to try to inject some rigor.
From a more general perspective, many frequentist methods fall prey to Lindley's Paradox. In simple terms, their inference is poorly calibrated for large sample sizes. They often mistake a negligible deviation from the null for a "statistically significant" discovery, even when the evidence actually supports the null. This is quite typical in clinical trials. (Spiegelhalter et al, 2003) is a great read to learn more even if you are not interested in medical statistics [1].
[1] https://onlinelibrary.wiley.com/doi/book/10.1002/0470092602
In a Bayesian analysis, the result of an inference, e.g. about the fairness of a coin as in Lindley's paradox, depends completely on the distribution of the alternative specified in the analysis. The frequentist analysis, for better and worse, doesn't need to specify a distribution for the alternative.
The classic Lindley's paradox uses a uniform alternative, but there is no justification for this at all. It's not as though a coin is either perfectly fair or has a totally random heads probability. A realistic bias will be subtle and the prior should reflect that. Something like this is often true of real-world applicaitons too.
Bayesian supporters often like to say they are just using more information by coding them in priors, but if they had data to support their priors, they are frequentists.
Ironically this has been one of the primary examples of, in my personal experience, with the problems I have worked on, frequentist mixed & random effects models have worked just fine. On rare occasions I have encountered a situation where the data was particularly complex or I wanted to use an unusual compound probability distribution and thought Bayesian approaches would save me. Instead, I have routinely ended up with models that never converge or take unpractical amounts of time to run. Maybe it’s my lack of experience jumping into Bayesian methods only on super hard problems. That’s totally possible.
But I have found many frequentist approaches to multilevel modeling perfectly adequate. That does not, of course, mean that will hold true for everyone or all problems.
One of my hot takes is that people seriously underestimate the diversity of data problems such that many people can just have totally different experiences with methods depending on the problems they work on.
If you have experienced problems with convergence, give Stan a try. Stan is really robust, polished, and simple. Besides, models are statically typed and it warns you when you do something odd.
Personally, I think once you start doing multilevel modeling to shrink estimates, there's no way back. At least in my case, I now see it everywhere. Thanks to efficient variational Bayes methods built on top of JAX, it is doable even on high-dimensional models.
I've found Bayesian methods shine in cases of an "intractible partition function".
Cases such as language models, where the cardinality of your discrete probability distribution is extremely large, to the point of intractability.
Bayesians tend to immediately go to things like Monte Carlo estimation. Is that fundamentally Bayesian and anti-frequentist? Not really... it's just that being open to Bayesian ways of thinking leads you towards that more.
Reinforcement learning also feels much more naturally Bayesian. I mean Thompson sampling, the granddaddy of RL, was developed through a frequentist lens. But it also feels very Bayesian as well.
In the modern era, we have Stein's paradox, and it all feels the same.
Hardcore Bayesians that seem to deeply hate the Kolmogorov measure theoretic approach to probability are always interesting to me as some of the last true radicals.
I feel like 99% of the world today is these are all just tools and we use them where they're useful.
After Stein's paradox it became super hard to be a pure frequentist if you didn't have your head in the sand.
The entire generative concept implicitly assumes that parameters have probability distributions themselves that naturally give rise to generative models...
You could do frequentist inference on a generative model, sure, but generative modelling seems fundamentally alien to frequentist thinking?
Though if you think about it, a diffusion model is somewhat (partially) frequentist.
But while it's a probability distribution, to a frequentist they are estimating the fixed parameters of a distribution.
The distribution isn't generative, it just represents uncertainty - and I think that's a bit of the deep core philosophical divide between frequentists and Bayesians - you might use all the same math, but you cannot possibly think of it as being generative.
https://arxiv.org/pdf/2510.18777
But that doesn't mean a frequentist views a VAE as a generative model!
Putting it another way, Gaussian processes originated as a frequentist technique! But to a frequentist they are not generative.
To be more precise, in Bayesian statistics a parameter is random variable. But what does that mean? A parameter is a characteristic of a population (as opposed to a characteristic of a sample, which is called a statistic). A quantity, such as the average cars per household right now. That's a parameter. To think of a parameter as a random variable is like regarding reality as just one realisation of an infinite number of alternate realities that could have been. The problem is we only observe our reality. All the data samples that we can ever study come from this reality. As a result, it's impossible to infer anything about the probability distribution of the parameter. The whole Bayesian approach to statistical inference is nonsensical.