Hi everyone. Have a question that is not directly related to Growthbook but rather to A/B testing in...
h

Home Kralych

over 2 years ago
Hi everyone. Have a question that is not directly related to Growthbook but rather to A/B testing in general. Precisely, sample size estimation (now that you have Frequentiest framework in your product, it’s even more relevant). So, when estimating the required sample size for the experiment, one usually needs the base conversion rate so they can estimate the required sample size. The problem we have is that our conversion rate fluctuates wildly, so even if the historical data shows the base conversion rate of 9%, it can significantly drop or get higher during the experiment due to some external factors (like, we started getting less quality traffic from our PPC ads). Are there any ways to take this into account so that we still have the statistical power of at least 80% even if the “real” conversion rate drops during the experiment for both variants? My first guess would be to “underestimate” base conversion rate, but i’m not sure I’m on the right track here. Say, we looked at the data for the last 3 months and discovered that our conversion rate was 9%, would it be statistically and mathematically valid to specify the base conversion rate to be 8% instead of 9%, so that we “safeguard” ourselves against conversion drop during the experiment? Of course, it would require larger sample size, but that’s a sacrifice we are willing to take.