I'm new to GrowthBook and I wasn't able to find in the documentation how does it handle multiple primary metrics for the Bayesian Approach? Is there something similar for the Bonferroni correction for Bayesian Metrics?
Thanks in advance! 🙏
05/16/2023, 3:35 PM
Hi Pedro, we don’t do multiple testing corrections for Bayesian currently
05/16/2023, 5:02 PM
Thanks for the answer. Out of curiosity, do you have plans to implement corrections on multiple metrics/hypotheses for Bayesian or is there a good reason to not adjust them?
05/16/2023, 5:07 PM
05/16/2023, 5:11 PM
There aren't immediate plans because there isn't a direct analog in the Bayesian framework. The Bayesian systems that traditionally help deal with comparing many things all at once require additional scaffolding about the relationship between tests and assuming priors for those relationships, which is somewhat more involved.
Some people would argue that multiple comparisons aren't a problem for Bayesian analysis, but I would say those arguments are mostly theoretical, and practically one should indeed be concerned with this. Unfortunately building a general system to handle this is harder for Bayesian analysis than for frequentist analysis.