#announcements
Title
f
fast-continent-33282
02/23/2023, 3:50 AM@fresh-football-47124 why do we use 95% credible interval and not 89% . I was reading in some papers for bayesian framework 89% CI would be a better compared to 95% though arbitrary?
f
fresh-football-47124
02/23/2023, 4:13 AM
Better is subjective
Depends what you are optimizing for
we have a change coming soon which will let you adjust it
🙏 1
f
fast-continent-33282
02/23/2023, 2:15 PM@fresh-football-47124 Can you clarify is that is the mean of posterior distribution or MAP? (Percent Change)
f
fresh-football-47124
02/23/2023, 3:46 PM
@helpful-application-7107^?
h
helpful-application-7107
02/23/2023, 4:26 PMGiven the distributions we're using, I'm pretty sure the mean of the posterior is the same as the MAP.
The percent we display is definitely the mean difference, which given the uninformative prior is the same as the mean of the posterior.
f
fast-continent-33282
02/27/2023, 2:46 PMHi @helpful-application-7107 / @fresh-football-47124
Conversion for Control : 2.84 % (134/ 4721)
Treatment : 3.43% (160/4621)
And Chance to beat control : 95.6%
Risk : 0.19%
Percent change : 21.8%
With all the above data , i want to roll-out treatment with the variability is so high.
1. What is the risk if i roll-out treatment?
2. What is the 95% credible interval for treatment
3. What is the gain/loss of users if i go ahead with treatment?
f
fresh-football-47124
02/28/2023, 4:48 AM
Hi Nishant:
1. Depends a bit on what you mean by risk, but there is a 95.6% chance your treatment is an improvement over the control, and a 4% chance it’s actually not - and if it’s actually the 4% case, the likely impact to your metric is 0.19%.
2. We show the 95% CI in the graph/violin plot, do you need the specific values?
3. The likely improvement to this metric is around 21%. But as the overall numbers are fairly low, I imagine the error bars are pretty wide for your CI.
f
fast-continent-33282
02/28/2023, 10:28 AMTo the point number 2, yes i would like to calculate the specific values for 95% CI .
To the point number 3, the error bars are very wide for the CI ranging from -0.2% to 52%
@fresh-football-47124 In the current experiment we added a new feature (notification feature) . It led to improvement in conversion. The relative uplift is 21% . Now my question is if I roll the new feature and i am wrong . In that scenario what does risk tells me. Does it tell me that since control has the chance to better by 4% , in that scenario my treatment conversion would be the baseline conversion * (1 - 0.0019).
Here baseline conversion is 2.84%( control).
So risk tells me that if i choose treatment and it is worse in that scenario my treatment conversion is likely to drop at 2.84*(1- 0.0019)= 2.8346 .
So risk is less than 1% if i roll out new feature.
f
fresh-football-47124
02/28/2023, 4:54 PM
yes, correct - Risk shows you the expected loss if you’re wrong
f
fast-continent-33282
03/02/2023, 2:21 PMHi @fresh-football-47124 does it make any sense to understand Type 1 error and Type 2 error in bayesian framework ? If i want to interpret Type 1 error or Type 2 error can we make any inferences from Bayesian framework?
f
fresh-football-47124
03/02/2023, 5:05 PM
@helpful-application-7107?
h
helpful-application-7107
03/02/2023, 5:16 PMType 1 and Type 2 errors can be strictly defined in both frameworks, it's just that the frequentist framework, when executed as expected, will provide you with particular controls over Type I errors.
If you're using the framework to make ship/no-ship decisions, then it inherently will have some type I and type II errors, it can just be hard to know up front with the Bayesian framework (and it can be hard to know in the Frequentist framework if you stop early, test multiple metrics, don't know the power of your test, etc.).
8 Views