#announcements

Title

l

little-cpu-3178

09/14/2022, 9:08 AMHey there! đź‘‹ Iâ€™m trying to understand (i.e. replicate in a spreadsheet) how the

`Chance to Beat Control`

and `Risk of Choosing`

statistics are calculated. Is there a toy example that shows how these calculations are done? (say on a sample dataset)f

fresh-football-47124

09/14/2022, 9:13 AM

We have docs which include a white paper on the formulas used: https://docs.growthbook.io/statistics

if you make a cloud account, you can load the sample data set

you can also download a jupyter notebook of the results which includes our stats package

l

little-cpu-3178

09/14/2022, 9:42 AMGot it. I have seen that itâ€™s possible download a jupyter notebook of the results and also that the gbstats code is available in github. What would be really helpful though is a more intuitive explanation/breakdown of that math. Anyway thanks!

f

fresh-football-47124

09/14/2022, 9:46 AM

weâ€™re making videos, perhaps we can do one on the statistics - what would make it more intuitive?

l

little-cpu-3178

09/14/2022, 9:50 AMBasically I was imagining something that would start from a small sample dataset, then walk through the steps necessary to arrive at the resulting statistic (in this case, â€śchance to beat controlâ€ť).
Currently what Iâ€™m planning to do now in order to understand the calculation better is just to unpack the code for analyze_metric_df using my own sample data, and then see how the variables are populated at each step.

f

fresh-football-47124

09/14/2022, 9:52 AM

cool, if you come up with a good visualization Iâ€™d love to see it

đź‘Ś 1

l

little-cpu-3178

09/14/2022, 9:55 AMBy the way, it might also be helpful to mention that the driving reason for this is that we are receiving somewhat â€śunexpectedâ€ť results for an AA experiment that we are running.
I would expect that in an AA test scenario with sufficient data, most metrics would converge to ~0% for â€śPercent Changeâ€ť and ~50% for â€śChance to Beat Controlâ€ť. Is this the correct intuition, or am I missing something?
Similar to the example that was posted here: https://growthbookusers.slack.com/archives/C01T6PKD9C3/p1660436632723409

f

fresh-football-47124

09/14/2022, 10:00 AM

you are correct - what is your sample size?

l

little-cpu-3178

09/14/2022, 10:01 AMWeâ€™re at roughly 250k users in each group

5 Views