Is there any explicit adjustment for multiple testing in growthbook? I know the danger of multiple testing/comparisons in bayesian inference is debated, so just wondering if there’s an official take on that or best practice recommendations.
11/29/2022, 9:13 PM
We haven’t built in any multiple testing correction so far. When we show dimensional analysis with high cardinality we show a warning message and turn off the statistical analysis by default. We encourage users to use dimensional analysis as data points for followup experiments, and not to use them causally
11/29/2022, 9:38 PM
Thanks (and agree about caution with dimensional analysis). One follow-up — as far as we can tell the risk thresholds are set for a given metric wherever is is used. It’s not possible to adjust the thresholds for a given metric for a specific experiment. Is that right? Is there any functionality to that affect that we’re missing? (cc @wide-jordan-14758)
11/29/2022, 10:03 PM
at the moment, you can set risk thresholds per metric, but not per experiment
@stocky-vr-71737 your project doesn’t include this, does it?
11/30/2022, 3:31 AM
This is correct, currently we do not have per-experiment overrides for risk threshold. Although that is certainly something we could add (Adding would be quite similar to our per-experiment metric conversion window overrides, launching soon...)