Final questions that I had today: I’m seeing a lot of discussion of Type I rate in the docs, e.g. when I define dimensions.
1. Is the long-term Type I error rate in a Bayesian world fixed like in a frequentist world? If so, what is the rate in Growthbook?
2. Does Growthbook correct for the Type I error rate when I split data by dimensions?
06/11/2022, 8:01 PM
We do not do any multiple testing corrections. So if you add a lot of metrics or drill down by dimensions your false positive rate will increase.
06/12/2022, 12:07 AM
awesome thanks. Is #1 a valid question, i.e. is the long-term false positive fixed in the same way with the Bayesian approach?