#announcements
Title
# announcements
l

little-cpu-3178

09/14/2022, 9:08 AM
Hey there! đź‘‹ Iâ€™m trying to understand (i.e. replicate in a spreadsheet) how the
``Chance to Beat Control``
and
``Risk of Choosing``
statistics are calculated. Is there a toy example that shows how these calculations are done? (say on a sample dataset)
f

fresh-football-47124

09/14/2022, 9:13 AM
We have docs which include a white paper on the formulas used: https://docs.growthbook.io/statistics
if you make a cloud account, you can load the sample data set
you can also download a jupyter notebook of the results which includes our stats package
l

little-cpu-3178

09/14/2022, 9:42 AM
Got it. I have seen that itâ€™s possible download a jupyter notebook of the results and also that the gbstats code is available in github. What would be really helpful though is a more intuitive explanation/breakdown of that math. Anyway thanks!
f

fresh-football-47124

09/14/2022, 9:46 AM
weâ€™re making videos, perhaps we can do one on the statistics - what would make it more intuitive?
l

little-cpu-3178

09/14/2022, 9:50 AM
Basically I was imagining something that would start from a small sample dataset, then walk through the steps necessary to arrive at the resulting statistic (in this case, â€śchance to beat controlâ€ť). Currently what Iâ€™m planning to do now in order to understand the calculation better is just to unpack the code for analyze_metric_df using my own sample data, and then see how the variables are populated at each step.
f

fresh-football-47124

09/14/2022, 9:52 AM
cool, if you come up with a good visualization Iâ€™d love to see it
đź‘Ś 1
l

little-cpu-3178

09/14/2022, 9:55 AM
By the way, it might also be helpful to mention that the driving reason for this is that we are receiving somewhat â€śunexpectedâ€ť results for an AA experiment that we are running. I would expect that in an AA test scenario with sufficient data, most metrics would converge to ~0% for â€śPercent Changeâ€ť and ~50% for â€śChance to Beat Controlâ€ť. Is this the correct intuition, or am I missing something? Similar to the example that was posted here: https://growthbookusers.slack.com/archives/C01T6PKD9C3/p1660436632723409
f

fresh-football-47124

09/14/2022, 10:00 AM
you are correct - what is your sample size?
l

little-cpu-3178

09/14/2022, 10:01 AM
Weâ€™re at roughly 250k users in each group
17 Views