https://www.growthbook.io/ logo
#announcements
Title
# announcements
s

some-planet-44104

07/14/2023, 8:28 PM
Hi! I’ve been wondering why in some cases the credible interval has a visible skew like in this case with my revenue per user metric. I kinda get the general idea and intuitively it’s somewhat understandable, but maybe someone can give me more detail? I’m well versed in confidence intervals in Frequentist approach but they d”n’t have this concept of “thickness”
f

fresh-football-47124

07/14/2023, 8:47 PM
@helpful-application-7107 ^
h

helpful-application-7107

07/14/2023, 9:12 PM
In the Bayesian engine, we show a posterior distribution, so these literally represent some density curve about what your posterior beliefs about uplift should be. The posterior probability of uplift being < 0% will be (roughly, due to some vis constraints) the pct of the area of this graph that is to the left of 0. So that's nice! So why is it skewed? • A reasonable experimenter might assume it's because we have some informative prior that skews results towards zero; but it isn't this reason. We use non-informative priors, so we shouldn't see any weight from the prior dragging results towards zero. • The real reason, is that we are showing you the posterior distribution of
uplift = (treatment_mean - control_mean) / control_mean
using a lognormal distribution. We approximate
uplift
using
log(treatment_mean / control_mean)
which simplifies computation of the posterior. Using this log-transformation results in the skew that you see.
s

some-planet-44104

07/15/2023, 7:42 AM
Thanks for the explanation. I’m not very familiar with this approach and it was only yesterday that I read about log transformation but in the context of making the distribution less “skewed”. I checked the whitepaper but didn’t find anything about this approach in Growthbook.
I also read that transforming the distribution like that we are modeling the distribution of the median and not the mean, something along those lines
8 Views