Hey there! 👋 I’m trying to understand (i.e. replicate in a spreadsheet) how the

Chance to Beat Control

and

Risk of Choosing

statistics are calculated. Is there a toy example that shows how these calculations are done? (say on a sample dataset)

We have docs which include a white paper on the formulas used: https://docs.growthbook.io/statistics

Got it. I have seen that it’s possible download a jupyter notebook of the results and also that the gbstats code is available in github. What would be really helpful though is a more intuitive explanation/breakdown of that math. Anyway thanks!

we’re making videos, perhaps we can do one on the statistics - what would make it more intuitive?

Basically I was imagining something that would start from a small sample dataset, then walk through the steps necessary to arrive at the resulting statistic (in this case, “chance to beat control”). Currently what I’m planning to do now in order to understand the calculation better is just to unpack the code for analyze_metric_df using my own sample data, and then see how the variables are populated at each step.

cool, if you come up with a good visualization I’d love to see it

By the way, it might also be helpful to mention that the driving reason for this is that we are receiving somewhat “unexpected” results for an AA experiment that we are running. I would expect that in an AA test scenario with sufficient data, most metrics would converge to ~0% for “Percent Change” and ~50% for “Chance to Beat Control”. Is this the correct intuition, or am I missing something? Similar to the example that was posted here: https://growthbookusers.slack.com/archives/C01T6PKD9C3/p1660436632723409

you are correct - what is your sample size?

We’re at roughly 250k users in each group