Hi team, Could someone from GrowthBook team please provide information on what implications can we imply from

Chance to Beat Control

when running experiments? Would be great if you can also help elaborate on what this parameter means. I am running an experiment and yesterday I was seeing a lot higher value of

Chance to Beat Control

whereas today, when more users are flowing in, this

Chance to Beat Control

went down to 50% though the percent change of the two variants is greater than yesterday (51.4% vs 46.3%).

Hi, that doesn’t look right

Thank you 🙂

Hi Phrare. Sorry about this. This is likely the result of the large amount of variance in your metric due to the small number of observations and some large outliers. In these situations, when the sample size is so small (<30 in this case), often our Bayesian engine cannot quite give you reliable variance estimates. Unfortunately, our UI doesn't do a great job of telling you this. You can either just wait for some more data, or you can try editing the metric to add a cap that will winsorize some large outliers. That could work in the short-run, but the cap will also change your estimates and you may want to remove it once you've collected more data.

Hi @fresh-football-47124 @helpful-application-7107, Seems like it’s happening again, now with more sample size. Is there something I should be aware of?

I suspect it is still the same issue. It all depends on the variance of your estimates. and it looks like it is likely quite large (but that your variation 1 is clearly better).

@helpful-application-7107 thank you for the clarification - is this something Growthbook team needs to further investigate? There’s more volume flowing in, but the percent change result is still the same.

I don't think so. I think it's unfortunate the case when there are large outliers, like I suspect, and small (<100) observations. If you want, you can click the three dots in the top right and share the main_sum and main_sum_squares statistics with me and I can confirm the issue. You can get those numbers like so: https://www.loom.com/share/3a64819e50474b21a8fd8c2c2664677f