Hi. My understanding is that the statistical significance for non-binomial metrics is calculated somewhat different and uses different statistical approaches, is that correct?

Hi Eugene, yes, you can read about the stats we use here: https://www.growthbook.io/docs/GrowthBookStatsEngine.pdf

Both binomial and non-binomial use the same basic statistical approach, just with different Bayesian priors. Binomial uses beta-binomial priors and everything else uses gaussian priors.

sorry to ‘highjack’ this question , but should we assume any type of distribution of the data for continuous data? should not matter if I’m correct?

With enough samples, the central limit theorem should apply. If your data is extremely skewed, we recommend adding a cap to the value (e.g. normal orders are $10, but you get a $1000 bulk order occasionally)

k that’s helpful. Using frequentist approach I do notice quite a difference between using Mann Whitney U versus regular TTest. So central limit theorem apparently is not the complete story or doesn’t apply somehow 😉 Was wondering how this is handled in bayesian testing

You can set a capped value in the metric behavior settings. We're looking into adding more prior options for extremely skewed distributions

oh that’s awesome 🙂