This message was deleted.

It's likely just a false positive. One thing you can do is break down the results by different dimensions and see if one of them explains the change. For example, maybe one of your variations randomly happened to get more US visitors in it and they might convert at a much higher rate for that metric. That would tell you that when looking at that metric, you should make sure to take that dimension into account before making conclusions for an experiment.

Thanks, that helps. Is there a fixed false positive rate in Growthbook? e.g. 5% of experiment-metric results will be false positives? I'm not sure how to reason about this so if you have other rules of thumb when interpreting results it would be great to hear!

Bayesian methods focus more on minimizing loss as opposed to worrying about the false positive rate directly. If you make the wrong decision, but the risk/loss is super low, it's not really that bad. That being said, you are likely using some decision rules for Bayesian experiments like "if chance to beat control goes above 95%, call it a winner". In that case, you are in essence choosing to have a 5% false positive rate across experiments.

So if Growthbook shows a 99.99% chance of a variant beating control and the risk has stabilized at 0, is there truly only a .01% chance that result is a false positive? If we can use 100% - chance to beat control as a heuristic for a “false positive threshold” then that definitely helps us reason through it

Yes, I think that's the right way to think about it. The one thing GrowthBook does not take into account right now is a multiple testing correction. So if you have lots of metrics, or look at lots of different dimension breakdowns, you're false positive rate will increase since you are in essence doing a lot of mini experiments. If you have 20 independent metrics and each one shows 95% chance to beat control, it's likely one of them is a false positive.

Thanks! That makes sense.