s

some-planet-44104

01/11/2023, 8:58 AM
Hi everyone. Am I correct to assume, that if I wanted to test for the “chance to be different” instead of the “chance to beat control” with the 95%. certainty, it would be correct to look at the probability interval of “2.5%-97.5%” instead of “0%-95%“? I understand it might not be something that fits into Bayesian world, the question is purely hypothetical, to help me better understand what’s going on
h

01/11/2023, 3:46 PM
It depends on what you mean by "chance to be different," because that might be tricky to define in a really meaningful way. In what way will you be using this interval? When we say "chance to beat control" we aren't looking at a range 0-95%, we're summing the posterior distribution that is greater above 0.
s

some-planet-44104

01/12/2023, 3:13 PM
Hi Luke. My understanding, if the 95% of the posterior distribution is above 0, we are reasonably confident that our variation is better when testing for an improvement. But when testing for a difference instead of an improvement, we can reject the null hypothesis if variation is extremely better or extremely worse, but in order to account for 95% of the distribution, we reject null hypothesis if either 97.5% of the distribution is above 0 (in which case the variation is better) or only 2.5% is above 0 (in which case the variation is worse then control). That’s my understanding, but I need a validation that I’m thinking in the right direction. Thank you for taking your time to respond to my question!
h

01/12/2023, 3:16 PM
Hi Yevhan. So you're right that if we were to map the bayesian posterior on to frequentist concepts like null hypothesis testing, we could make statements like the ones you're making above. Unfortunately, the two concepts don't map cleanly to one another; your standard null hypothesis significance test doesn't have a direct relationship to the bayesian framework.
The Bayesian framework should be attractive in that it explicitly lets you say "There is a 95% chance that variant B is better than variant B". You could make your personal decision making criteria higher or lower than 95%, but it isn't exactly the same as setting a 0.05 or 0.1 significance level in the frequentist framework.
s

some-planet-44104

01/12/2023, 4:40 PM
But can Bayesian Framework answer questions like “There is a 95% chance that variant B is DIFFERENT from variant A?
h