#announcements
Title
f
fast-continent-33282
08/16/2023, 2:48 PMHi @helpful-application-7107
I have been running and experiment and it seems the chance to beat control is not correct.
These are my numbers for the current experiment.
Control : 1287 / 2384 (53.98%)
Treatment : 1271 / 2234 (56.89%)
And Relative Mean Uplift is 5.4 %
Risk if i choose Treatment is : 0.53%
The Growthbook outputs 82% chance to beat control but when i calculated it on the notebook it is 97%.
Can you check these numbers please?
@fresh-football-47124
h
helpful-application-7107
08/16/2023, 3:59 PMWhat notebook are you using to calculate it, can you share it?
Can you also share a screenshot? I get 97% as well when passing the numbers through our stats engine
f
fast-continent-33282
08/16/2023, 4:09 PMScreenshot 2023-08-16 at 9.37.06 PM.png
Copy code
import numpy as np
from scipy.stats import norm, beta
from scipy.special import digamma, polygamma, roots_hermitenorm # , roots_sh_jacobi
from scipy import linalg
from scipy.special import betaln, eval_jacobi, roots_sh_jacobi Copy code
def update_beta_prior(x_a, n_a, x_b, n_b,alpha_0 = 1, beta_0=1):
alpha_a = alpha_0 + x_a
beta_a = beta_0 + n_a - x_a
alpha_b = alpha_0 + x_b
beta_b = beta_0 + n_b - x_b
return alpha_a, beta_a, alpha_b, beta_b Copy code
def calculate_beta_binomial(alpha_a, beta_a, alpha_b, beta_b):
log_beta_mean = lambda a, b: digamma(a) - digamma(a + b)
var_beta_mean = lambda a, b: polygamma(1, a) - polygamma(1, a + b)
d1_beta = norm(loc=beta.mean(alpha_b, beta_b) - beta.mean(alpha_a, beta_a),
scale=np.sqrt(beta.var(alpha_b, beta_b) + beta.var(alpha_a, beta_a)))
d2_beta = norm(loc=log_beta_mean(alpha_b, beta_b) - log_beta_mean(alpha_a, beta_a),
scale=np.sqrt(var_beta_mean(alpha_b, beta_b) + var_beta_mean(alpha_a, beta_a)))
return d1_beta, d2_beta Copy code
x_a, n_a = 1287, 2384 # converted & total users in A
x_b, n_b = 1271, 2234 # converted & total users in B Copy code
alpha_0, beta_0 = 1, 1 # Beta prior
alpha_a, beta_a, alpha_b, beta_b = update_beta_prior(x_a, n_a,x_b, n_b, alpha_0, beta_0)
d1_beta, d2_beta = calculate_beta_binomial(alpha_a, beta_a, alpha_b, beta_b)
print(f'The probability the conversion in B is higher is {round(d1_beta.sf(0) * 100, 2)}%')
print(f'The 95% credibility interval of (p_b/p_a-1) is {(np.exp(d2_beta.ppf((.025, .975))) - 1)}')Thus is
Itamar Farhanh
helpful-application-7107
08/16/2023, 4:39 PMI'm not able to recreate this within GrowthBook, so something is happening between the data being passed to our stats engine and the data shown in the results ui
Can you share the metric settings (the right panel on the metric page) and the analysis settings in the Config tab?
And can you also click the three dots next to the results, select view queries, and share the full query response for this metric?
f
fast-continent-33282
08/16/2023, 5:02 PMcc: @helpful-application-7107
h
helpful-application-7107
08/16/2023, 5:39 PMTwo things: are you self hosted? This looks like an old set of queries.
You have a custom denominator which may be causing the difference and our front end might not be showing the right denominator for some reason but it might be hard to debug.
f
fast-continent-33282
08/16/2023, 5:41 PMYes we do have custom denominator.
h
helpful-application-7107
08/16/2023, 5:43 PMYeah so the variance can get complicated in that case depending on the denominator type, but it's going to be hard for me to debug as your GrowthBook looks like it's 6+ months old.
I don't think it's a mistake in the calculation of chance to win. Is the denominator a count or a binomial?
f
fast-continent-33282
08/16/2023, 5:44 PMIt's count.
First solution is to update Growth book.
h
helpful-application-7107
08/16/2023, 5:45 PMYeah so the calculation isn't wrong
The variance is different for ratio metrics. It relies on the Delta method to produce a different variance estimate than if it were a regular binomial metric.
There's no bug here, it's just a little unclear that this computation is different because it's a ratio metric