#announcements

Title

f

fast-continent-33282

02/23/2023, 3:50 AMf

fresh-football-47124

02/23/2023, 4:13 AM

Better is subjective

Depends what you are optimizing for

we have a change coming soon which will let you adjust it

🙏 1

f

fast-continent-33282

02/23/2023, 2:15 PMf

fresh-football-47124

02/23/2023, 3:46 PM

h

helpful-application-7107

02/23/2023, 4:26 PMGiven the distributions we're using, I'm pretty sure the mean of the posterior is the same as the MAP.

The percent we display is definitely the mean difference, which given the uninformative prior is the same as the mean of the posterior.

f

fast-continent-33282

02/27/2023, 2:46 PMHi **@helpful-application-7107** / **@fresh-football-47124**
Conversion for Control : 2.84 % (134/ 4721)
Treatment : 3.43% (160/4621)
And Chance to beat control : 95.6%
Risk : 0.19%
Percent change : 21.8%
With all the above data , i want to roll-out treatment with the variability is so high.
1. What is the risk if i roll-out treatment?
2. What is the 95% credible interval for treatment
3. What is the gain/loss of users if i go ahead with treatment?

f

fresh-football-47124

02/28/2023, 4:48 AM

Hi Nishant:
1. Depends a bit on what you mean by risk, but there is a 95.6% chance your treatment is an improvement over the control, and a 4% chance it’s actually not - and if it’s actually the 4% case, the likely impact to your metric is 0.19%.
2. We show the 95% CI in the graph/violin plot, do you need the specific values?
3. The likely improvement to this metric is around 21%. But as the overall numbers are fairly low, I imagine the error bars are pretty wide for your CI.

f

fast-continent-33282

02/28/2023, 10:28 AMTo the point number 2, yes i would like to calculate the specific values for 95% CI .
To the point number 3, the error bars are very wide for the CI ranging from -0.2% to 52%

f

fresh-football-47124

02/28/2023, 4:54 PM

yes, correct - Risk shows you the expected loss if you’re wrong

f

fast-continent-33282

03/02/2023, 2:21 PMHi **@fresh-football-47124** does it make any sense to understand Type 1 error and Type 2 error in bayesian framework ? If i want to interpret Type 1 error or Type 2 error can we make any inferences from Bayesian framework?

f

fresh-football-47124

03/02/2023, 5:05 PM

h

helpful-application-7107

03/02/2023, 5:16 PMType 1 and Type 2 errors can be strictly defined in both frameworks, it's just that the frequentist framework, when executed as expected, will provide you with particular controls over Type I errors.

If you're using the framework to make ship/no-ship decisions, then it inherently will have some type I and type II errors, it can just be hard to know up front with the Bayesian framework (and it can be hard to know in the Frequentist framework if you stop early, test multiple metrics, don't know the power of your test, etc.).

8 Views