I have this problem every so often - sometimes when updating the stats, continuous metrics show “chance to beat control” around 50% and the credible intervals aren’t displayed. When I switch to the Frequentist framework - everything is displayed correctly
07/17/2023, 3:35 PM
Because of the aforementioned log-approximation that we use in the bayesian engine, if the average in either group is statistically close to zero, sometimes our engine cannot properly compute the uplift posterior. This is likely what is happening here and should resolve if the variance decreases, but we should definitely improve the error messaging in this part of the app. We could also increase the tolerance to allow statistics through even in cases where the log approximation will work more poorly.