https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
b

billowy-window-97200

12/12/2023, 10:35 AM
Parallel A/B Testing on the same page
Hi there folks, I have a question: Does Growthbook have any feature to handle 2 or more experiments running on the same page at the same time? Or can we somehow setup this on the user side to account for multiple comparisons, for example?
f

future-teacher-7046

12/12/2023, 12:40 PM
The vast majority of the time, running 2 experiments on the same page at the same time is perfectly fine. Interaction effects between experiments are very rare. Plus, the benefit you get from running more experiments tends to outweigh any downsides. That said, there are some situations where you need tests to be mutually exclusive. We support namespaces for this. https://docs.growthbook.io/features/rules#namespaces
b

billowy-window-97200

12/12/2023, 1:49 PM
Thanks @future-teacher-7046
@future-teacher-7046 I read the topics you shared but I have a question still. If I define a multiple comparison correction, how to I also "connect" it to the other experiment running at the same page? Do I define these corrections for the 2 experiments running in the same page or only in one of them ?
f

future-teacher-7046

12/13/2023, 3:14 PM
Multiple comparison correction is only done within a single experiment results. If you have multiple metrics and/or a dimension drill down selected, we'll apply the correction to all of the p-values.
b

billowy-window-97200

12/13/2023, 4:37 PM
@future-teacher-7046 thanks for the reply! Just to ensure I get it right: Suppose there are 2 experiments, X and Y, running on the same page. Each of them have control and variation (Ax, Ay, Bx and By). When Applying the corrections, I understand the methods should account for the number of tests that are running in parallel (in this case 2), but I don't see this option in Growthbook. How is Growthbook defining the number of tests running in parallel and that need the correction?
f

future-teacher-7046

12/13/2023, 5:07 PM
Our multiple testing correction only operates on a single experiment and doesn't take into account any other experiments that may be running on the page.
Controlling multiple comparisons across experiments tends to be impractical and prohibitively costly in terms of statistical power. So we just focus on a single experiment's results view instead
b

billowy-window-97200

12/14/2023, 7:32 AM
Ahh okay, got it! Thanks @future-teacher-7046 for explaining 🙂
4 Views