Hey there! :wave: I’m trying to understand (i.e. r...

# announcementsl

little-cpu-3178

09/14/2022, 9:08 AMHey there! 👋 I’m trying to understand (i.e. replicate in a spreadsheet) how the

`Chance to Beat Control`

and `Risk of Choosing`

statistics are calculated. Is there a toy example that shows how these calculations are done? (say on a sample dataset)f

fresh-football-47124

09/14/2022, 9:13 AM

We have docs which include a white paper on the formulas used: https://docs.growthbook.io/statistics

fresh-football-47124

09/14/2022, 9:16 AM

if you make a cloud account, you can load the sample data set

fresh-football-47124

09/14/2022, 9:17 AM

you can also download a jupyter notebook of the results which includes our stats package

l

little-cpu-3178

09/14/2022, 9:42 AMGot it. I have seen that it’s possible download a jupyter notebook of the results and also that the gbstats code is available in github. What would be really helpful though is a more intuitive explanation/breakdown of that math. Anyway thanks!

f

fresh-football-47124

09/14/2022, 9:46 AM

we’re making videos, perhaps we can do one on the statistics - what would make it more intuitive?

l

little-cpu-3178

09/14/2022, 9:50 AMBasically I was imagining something that would start from a small sample dataset, then walk through the steps necessary to arrive at the resulting statistic (in this case, “chance to beat control”).
Currently what I’m planning to do now in order to understand the calculation better is just to unpack the code for analyze_metric_df using my own sample data, and then see how the variables are populated at each step.

f

fresh-football-47124

09/14/2022, 9:52 AM

cool, if you come up with a good visualization I’d love to see it

👌 1

l

little-cpu-3178

09/14/2022, 9:55 AMBy the way, it might also be helpful to mention that the driving reason for this is that we are receiving somewhat “unexpected” results for an AA experiment that we are running.
I would expect that in an AA test scenario with sufficient data, most metrics would converge to ~0% for “Percent Change” and ~50% for “Chance to Beat Control”. Is this the correct intuition, or am I missing something?
Similar to the example that was posted here: https://growthbookusers.slack.com/archives/C01T6PKD9C3/p1660436632723409

f

fresh-football-47124

09/14/2022, 10:00 AM

you are correct - what is your sample size?

l

little-cpu-3178

09/14/2022, 10:01 AMWe’re at roughly 250k users in each group

35 Views