Title
o

orange-train-515

12/13/2021, 9:32 AM
Hi. My understanding is that the statistical significance for non-binomial metrics is calculated somewhat different and uses different statistical approaches, is that correct?
f

fresh-football-47124

12/13/2021, 9:33 AM
Hi Eugene, yes, you can read about the stats we use here: https://www.growthbook.io/docs/GrowthBookStatsEngine.pdf
f

future-teacher-7046

12/13/2021, 1:00 PM
Both binomial and non-binomial use the same basic statistical approach, just with different Bayesian priors. Binomial uses beta-binomial priors and everything else uses gaussian priors.
b

breezy-crowd-53224

12/13/2021, 1:30 PM
sorry to ‘highjack’ this question , but should we assume any type of distribution of the data for continuous data? should not matter if I’m correct?
f

future-teacher-7046

12/13/2021, 2:10 PM
With enough samples, the central limit theorem should apply. If your data is extremely skewed, we recommend adding a cap to the value (e.g. normal orders are \$10, but you get a \$1000 bulk order occasionally)
b

breezy-crowd-53224

12/13/2021, 2:42 PM
k that’s helpful. Using frequentist approach I do notice quite a difference between using Mann Whitney U versus regular TTest. So central limit theorem apparently is not the complete story or doesn’t apply somehow 😉 Was wondering how this is handled in bayesian testing
I presume capping should be done in our data / SQL right?
f

future-teacher-7046

12/13/2021, 2:51 PM
You can set a capped value in the metric behavior settings. We're looking into adding more prior options for extremely skewed distributions
b

breezy-crowd-53224

12/13/2021, 3:33 PM
oh that’s awesome 🙂
13 Views