Hey there! My team would like to run a non-inferiority test using GrowthBook but I'm having trouble setting this up. Is this type of experiment supported by the platform?

Hello, I am a data scientist at GrowthBook. There are a couple of options: 1. if you just want to make sure something doesn't break, and you less focused on learning about your effect of your feature, then Safe Rollouts are a great option: https://docs.growthbook.io/features/safe-rollouts 2. if you do want to learn about the effect on some of your metrics, then you could do something like the following. Suppose revenue is your metric of interest, and you want to make sure revenue doesn't decrease. If you usually use false positive rate of

alpha = 0.05

, then you could instead use false positive rate of

alpha = 0.1

and reject the null that there is no effect only if the confidence interval is statistically significant and the effect is negative. This approximates the decision from a one-sided interval. Hope this helps, Luke

Thanks Luke! I'm curious about the Safe Rollout feature. I took a look at the docs you linked and wanted to ask how you would recommend teams to choose their monitoring period? Is there something like a power calculator for safe rollouts?

Hi Xing, You are right that GrowthBook's power calculator can help determine duration. While our power calculator assumes you are running an Experiment rather than a Safe Rollout, if you use the correct metrics and target the correct population, it will provide similar results. The results should be slightly conservative (i.e., it may suggest you run your Safe Rollout slightly longer than you need to). Additionally, I would monitor the feature over a long enough period to ensure that it is appropriately "battle tested." For example, if you get most of your user traffic on Monday and Tuesday, make sure that the monitoring period includes Monday and Tuesday. Hope this helps, Luke