https://www.growthbook.io/ logo
Join Slack
Powered by
<@U01T6HCHD0A> why do we use 95% credible interval...
# announcements
f
@fresh-football-47124 why do we use 95% credible interval and not 89% . I was reading in some papers for bayesian framework 89% CI would be a better compared to 95% though arbitrary?
f
Better is subjective
Depends what you are optimizing for
we have a change coming soon which will let you adjust it
🙏 1
f
@fresh-football-47124 Can you clarify is that is the mean of posterior distribution or MAP? (Percent Change)
f
@helpful-application-7107^?
h
Given the distributions we're using, I'm pretty sure the mean of the posterior is the same as the MAP.
The percent we display is definitely the mean difference, which given the uninformative prior is the same as the mean of the posterior.
f
Hi @helpful-application-7107 / @fresh-football-47124 Conversion for Control : 2.84 % (134/ 4721) Treatment : 3.43% (160/4621) And Chance to beat control : 95.6% Risk : 0.19% Percent change : 21.8% With all the above data , i want to roll-out treatment with the variability is so high. 1. What is the risk if i roll-out treatment? 2. What is the 95% credible interval for treatment 3. What is the gain/loss of users if i go ahead with treatment?
f
Hi Nishant: 1. Depends a bit on what you mean by risk, but there is a 95.6% chance your treatment is an improvement over the control, and a 4% chance it’s actually not - and if it’s actually the 4% case, the likely impact to your metric is 0.19%. 2. We show the 95% CI in the graph/violin plot, do you need the specific values? 3. The likely improvement to this metric is around 21%. But as the overall numbers are fairly low, I imagine the error bars are pretty wide for your CI.
f
To the point number 2, yes i would like to calculate the specific values for 95% CI . To the point number 3, the error bars are very wide for the CI ranging from -0.2% to 52%
@fresh-football-47124 In the current experiment we added a new feature (notification feature) . It led to improvement in conversion. The relative uplift is 21% . Now my question is if I roll the new feature and i am wrong . In that scenario what does risk tells me. Does it tell me that since control has the chance to better by 4% , in that scenario my treatment conversion would be the baseline conversion * (1 - 0.0019). Here baseline conversion is 2.84%( control). So risk tells me that if i choose treatment and it is worse in that scenario my treatment conversion is likely to drop at 2.84*(1- 0.0019)= 2.8346 . So risk is less than 1% if i roll out new feature.
f
yes, correct - Risk shows you the expected loss if you’re wrong
f
Hi @fresh-football-47124 does it make any sense to understand Type 1 error and Type 2 error in bayesian framework ? If i want to interpret Type 1 error or Type 2 error can we make any inferences from Bayesian framework?
f
@helpful-application-7107?
h
Type 1 and Type 2 errors can be strictly defined in both frameworks, it's just that the frequentist framework, when executed as expected, will provide you with particular controls over Type I errors.
If you're using the framework to make ship/no-ship decisions, then it inherently will have some type I and type II errors, it can just be hard to know up front with the Bayesian framework (and it can be hard to know in the Frequentist framework if you stop early, test multiple metrics, don't know the power of your test, etc.).
72 Views
Open in Slack
PreviousNext
Powered by