https://www.growthbook.io/ logo
l

little-cpu-3178

09/14/2022, 9:08 AM
Hey there! 👋 I’m trying to understand (i.e. replicate in a spreadsheet) how the
Chance to Beat Control
and
Risk of Choosing
statistics are calculated. Is there a toy example that shows how these calculations are done? (say on a sample dataset)
f

fresh-football-47124

09/14/2022, 9:13 AM
We have docs which include a white paper on the formulas used: https://docs.growthbook.io/statistics
if you make a cloud account, you can load the sample data set
you can also download a jupyter notebook of the results which includes our stats package
l

little-cpu-3178

09/14/2022, 9:42 AM
Got it. I have seen that it’s possible download a jupyter notebook of the results and also that the gbstats code is available in github. What would be really helpful though is a more intuitive explanation/breakdown of that math. Anyway thanks!
f

fresh-football-47124

09/14/2022, 9:46 AM
we’re making videos, perhaps we can do one on the statistics - what would make it more intuitive?
l

little-cpu-3178

09/14/2022, 9:50 AM
Basically I was imagining something that would start from a small sample dataset, then walk through the steps necessary to arrive at the resulting statistic (in this case, “chance to beat control”). Currently what I’m planning to do now in order to understand the calculation better is just to unpack the code for analyze_metric_df using my own sample data, and then see how the variables are populated at each step.
f

fresh-football-47124

09/14/2022, 9:52 AM
cool, if you come up with a good visualization I’d love to see it
👌 1
l

little-cpu-3178

09/14/2022, 9:55 AM
By the way, it might also be helpful to mention that the driving reason for this is that we are receiving somewhat “unexpected” results for an AA experiment that we are running. I would expect that in an AA test scenario with sufficient data, most metrics would converge to ~0% for “Percent Change” and ~50% for “Chance to Beat Control”. Is this the correct intuition, or am I missing something? Similar to the example that was posted here: https://growthbookusers.slack.com/archives/C01T6PKD9C3/p1660436632723409
f

fresh-football-47124

09/14/2022, 10:00 AM
you are correct - what is your sample size?
l

little-cpu-3178

09/14/2022, 10:01 AM
We’re at roughly 250k users in each group
5 Views