This message was deleted.

Oof, so sorry about this. This looks like a bug. One second.

Thanks. It’s not an urgency so I’ll probably wait for the fix

Should be landed now, if you download a new notebook. Please let me know if you get a chance to check it out!

Checked it. Seemingly works without hickups. By the way, I’m now accessing the statistical model you use by using synthetic data and put hundreds of tests through the model in the jupyter notebook and what strikes me as odd is that even with small sample sizes of 500 or 1000 users the credible interval holds exceptionally well, even though the original population distributions are extremely skewed. I’d expect credible intervals be less reliable with small samples taken from extremely skewed population distributions but somehow everything I’ve seen so far performs well and I can’t explain it. I tried to sample the distribution and given the skew of the population distribution and the sample size, the distribution of the sample mean doesn’t converge to normal. Still, credible intervals look reliable. I am kind of surprised why it’s the case.

Beyond the fact that the CLT is pretty strong even at small sample sizes, the bayesian engine explicitly models the data as beta (for binomial) or normally (for count metrics) distributed. This is an assumption of our bayesian (and basically all A/B testing platforms that have a bayesian engine).