Hi <@U03GC3G9E8M>
I have been running and experime...

# announcementsf

fast-continent-33282

08/16/2023, 2:48 PMHi **@helpful-application-7107**
I have been running and experiment and it seems the chance to beat control is not correct.
These are my numbers for the current experiment.
Control : 1287 / 2384 (53.98%)
Treatment : 1271 / 2234 (56.89%)
And Relative Mean Uplift is 5.4 %
Risk if i choose Treatment is : 0.53%
The Growthbook outputs 82% chance to beat control but when i calculated it on the notebook it is 97%.
Can you check these numbers please?
**@fresh-football-47124**

h

helpful-application-7107

08/16/2023, 3:59 PMWhat notebook are you using to calculate it, can you share it?

helpful-application-7107

08/16/2023, 4:01 PMCan you also share a screenshot? I get 97% as well when passing the numbers through our stats engine

f

fast-continent-33282

08/16/2023, 4:09 PMScreenshot 2023-08-16 at 9.37.06 PM.png

fast-continent-33282

08/16/2023, 4:14 PM Copy code

```
import numpy as np
from scipy.stats import norm, beta
from scipy.special import digamma, polygamma, roots_hermitenorm # , roots_sh_jacobi
from scipy import linalg
from scipy.special import betaln, eval_jacobi, roots_sh_jacobi
```

Copy code

```
def update_beta_prior(x_a, n_a, x_b, n_b,alpha_0 = 1, beta_0=1):
alpha_a = alpha_0 + x_a
beta_a = beta_0 + n_a - x_a
alpha_b = alpha_0 + x_b
beta_b = beta_0 + n_b - x_b
return alpha_a, beta_a, alpha_b, beta_b
```

Copy code

```
def calculate_beta_binomial(alpha_a, beta_a, alpha_b, beta_b):
log_beta_mean = lambda a, b: digamma(a) - digamma(a + b)
var_beta_mean = lambda a, b: polygamma(1, a) - polygamma(1, a + b)
d1_beta = norm(loc=beta.mean(alpha_b, beta_b) - beta.mean(alpha_a, beta_a),
scale=np.sqrt(beta.var(alpha_b, beta_b) + beta.var(alpha_a, beta_a)))
d2_beta = norm(loc=log_beta_mean(alpha_b, beta_b) - log_beta_mean(alpha_a, beta_a),
scale=np.sqrt(var_beta_mean(alpha_b, beta_b) + var_beta_mean(alpha_a, beta_a)))
return d1_beta, d2_beta
```

Copy code

```
x_a, n_a = 1287, 2384 # converted & total users in A
x_b, n_b = 1271, 2234 # converted & total users in B
```

Copy code

```
alpha_0, beta_0 = 1, 1 # Beta prior
alpha_a, beta_a, alpha_b, beta_b = update_beta_prior(x_a, n_a,x_b, n_b, alpha_0, beta_0)
d1_beta, d2_beta = calculate_beta_binomial(alpha_a, beta_a, alpha_b, beta_b)
print(f'The probability the conversion in B is higher is {round(d1_beta.sf(0) * 100, 2)}%')
print(f'The 95% credibility interval of (p_b/p_a-1) is {(np.exp(d2_beta.ppf((.025, .975))) - 1)}')
```

fast-continent-33282

08/16/2023, 4:14 PMThus is

`Itamar Farhan`

code from medium that i use to double check these numbers.h

helpful-application-7107

08/16/2023, 4:39 PMI'm not able to recreate this within GrowthBook, so something is happening between the data being passed to our stats engine and the data shown in the results ui

helpful-application-7107

08/16/2023, 4:39 PMCan you share the metric settings (the right panel on the metric page) and the analysis settings in the Config tab?

helpful-application-7107

08/16/2023, 4:40 PMAnd can you also click the three dots next to the results, select view queries, and share the full query response for this metric?

f

fast-continent-33282

08/16/2023, 5:02 PMcc: **@helpful-application-7107**

fast-continent-33282

08/16/2023, 5:02 PMh

helpful-application-7107

08/16/2023, 5:39 PMTwo things: are you self hosted? This looks like an old set of queries.
You have a custom denominator which may be causing the difference and our front end might not be showing the right denominator for some reason but it might be hard to debug.

f

fast-continent-33282

08/16/2023, 5:41 PMYes we do have custom denominator.

h

helpful-application-7107

08/16/2023, 5:43 PMYeah so the variance can get complicated in that case depending on the denominator type, but it's going to be hard for me to debug as your GrowthBook looks like it's 6+ months old.

helpful-application-7107

08/16/2023, 5:44 PMI don't think it's a mistake in the calculation of chance to win. Is the denominator a count or a binomial?

f

fast-continent-33282

08/16/2023, 5:44 PMIt's count.

fast-continent-33282

08/16/2023, 5:45 PMFirst solution is to update Growth book.

h

helpful-application-7107

08/16/2023, 5:45 PMYeah so the calculation isn't wrong

helpful-application-7107

08/16/2023, 5:46 PMThe variance is different for ratio metrics. It relies on the Delta method to produce a different variance estimate than if it were a regular binomial metric.

helpful-application-7107

08/16/2023, 5:47 PMThere's no bug here, it's just a little unclear that this computation is different because it's a ratio metric

26 Views