Hi everyone. Am I correct to assume, that if I wanted to test for the “chance to be different” instead of the “chance to beat control” with the 95%. certainty, it would be correct to look at the probability interval of “2.5%-97.5%” instead of “0%-95%“? I understand it might not be something that fits into Bayesian world, the question is purely hypothetical, to help me better understand what’s going on

It depends on what you mean by "chance to be different," because that might be tricky to define in a really meaningful way. In what way will you be using this interval? When we say "chance to beat control" we aren't looking at a range 0-95%, we're summing the posterior distribution that is greater above 0.

Hi Luke. My understanding, if the 95% of the posterior distribution is above 0, we are reasonably confident that our variation is better when testing for an improvement. But when testing for a difference instead of an improvement, we can reject the null hypothesis if variation is extremely better or extremely worse, but in order to account for 95% of the distribution, we reject null hypothesis if either 97.5% of the distribution is above 0 (in which case the variation is better) or only 2.5% is above 0 (in which case the variation is worse then control). That’s my understanding, but I need a validation that I’m thinking in the right direction. Thank you for taking your time to respond to my question!

Hi Yevhan. So you're right that if we were to map the bayesian posterior on to frequentist concepts like null hypothesis testing, we could make statements like the ones you're making above. Unfortunately, the two concepts don't map cleanly to one another; your standard null hypothesis significance test doesn't have a direct relationship to the bayesian framework.

But can Bayesian Framework answer questions like “There is a 95% chance that variant B is DIFFERENT from variant A?

Technically, there is a near 100% chance that variant B is different form variant A, because the distribution is continuous and the probability that B is exactly equal to A is 0. You could definitely make a decision rule for yourself where you said "If 97.5% of the posterior distribution is above 0 OR 97.5% of the posterior distribution is below 0, then I will firmly believe that B is DIFFERENT from A." Another alternative would be to say "I think that any difference that is from -2.5 to 2.5% is not meaningful" and then make sure that 95+% of the posterior distribution was < -2.5% or > 2.5% and then say that "There is at least a 95% chance that variant B is 2.5% better or worse than variant A".