Hello GrowthBook team!
I have a question regarding multiple testing.
It seems to be that multiple testing correction takes into account both
number of groups and number of metrics.
Is there a possibility to use only number of groups in corrections?
h
helpful-application-7107
08/12/2024, 4:55 PM
Hey Aleksei, we don't currently have that option. By Groups do you mean Dimension slices or do you mean Variations? (e.g. in a 3 variation test you have 2 tests, A v B and A v C)?
@helpful-application-7107 Hi! By number of groups I mean variations 🙂 And it seems that multiple testing corrections take into account number of variations and number of used metrics. And my question was about the ability to apply corrections using only number of variations in one test.
h
helpful-application-7107
08/13/2024, 6:00 PM
Yep, understood. Thanks for clarifying, it helps me understand your use case. We don't currently have this as an option, but it's a reasonable ask. I've opened an issue but I'm not certain we will prioritize it: https://github.com/growthbook/growthbook/issues/2868
Can I ask one more question: do you have one key metric you want to apply corrections to, and the rest you don't care as much about, or do you have 2-3 key metrics and you want to correct within each of those, but not across?