I am trying to figure out why the confidence interval isn’t shown for the revenue metric, is it because of the “suspicious result” notification or some other reason? Usually it’s because there’s not enough data, but when there isn’t, there is a notification that mentions it
Hi Luke. Yet, it was lower than that. However, the tag disappeared but I’m still seing this “Chance to beat control” that is suspiciously stuck at 50%,
it haven’t moved at all since the beginning of the experiment
04/14/2023, 1:47 PM
This is probably the result of our particular bayesian approach causing problems when a group mean could potentially being negative (large variance and enough of the posterior is negative) and the 50% is a default return value when we can't compute uncertainty accurately. I can confirm this if you share the users, sums and sums of squares from the raw query.
We're aware just returning 50% is a bad user experience and improving it is our intention.