https://www.growthbook.io/ logo
m

mammoth-librarian-62538

03/31/2023, 4:40 AM
Hi team, Could someone from GrowthBook team please provide information on what implications can we imply from
Chance to Beat Control
when running experiments? Would be great if you can also help elaborate on what this parameter means. I am running an experiment and yesterday I was seeing a lot higher value of
Chance to Beat Control
whereas today, when more users are flowing in, this
Chance to Beat Control
went down to 50% though the percent change of the two variants is greater than yesterday (51.4% vs 46.3%).
f

fresh-football-47124

03/31/2023, 4:41 AM
Hi, that doesn’t look right
let me bring it up with the team
m

mammoth-librarian-62538

03/31/2023, 4:56 AM
Thank you 🙂
h

helpful-application-7107

03/31/2023, 5:46 AM
Hi Phrare. Sorry about this. This is likely the result of the large amount of variance in your metric due to the small number of observations and some large outliers. In these situations, when the sample size is so small (<30 in this case), often our Bayesian engine cannot quite give you reliable variance estimates. Unfortunately, our UI doesn't do a great job of telling you this. You can either just wait for some more data, or you can try editing the metric to add a cap that will winsorize some large outliers. That could work in the short-run, but the cap will also change your estimates and you may want to remove it once you've collected more data.
1
m

mammoth-librarian-62538

04/04/2023, 10:36 AM
Hi @fresh-football-47124 @helpful-application-7107, Seems like it’s happening again, now with more sample size. Is there something I should be aware of?
h

helpful-application-7107

04/04/2023, 2:40 PM
I suspect it is still the same issue. It all depends on the variance of your estimates. and it looks like it is likely quite large (but that your variation 1 is clearly better).
m

mammoth-librarian-62538

04/05/2023, 3:16 AM
@helpful-application-7107 thank you for the clarification - is this something Growthbook team needs to further investigate? There’s more volume flowing in, but the percent change result is still the same.
h

helpful-application-7107

04/05/2023, 3:54 PM
I don't think so. I think it's unfortunate the case when there are large outliers, like I suspect, and small (<100) observations. If you want, you can click the three dots in the top right and share the main_sum and main_sum_squares statistics with me and I can confirm the issue. You can get those numbers like so: https://www.loom.com/share/3a64819e50474b21a8fd8c2c2664677f