https://www.growthbook.io/
Join Slack
Hi <@U03GC3G9E8M> I have been running and experiment and it seems the chance to beat control is not ...
# announcements
f
Hi @helpful-application-7107 I have been running and experiment and it seems the chance to beat control is not correct. These are my numbers for the current experiment. Control : 1287 / 2384 (53.98%) Treatment : 1271 / 2234 (56.89%) And Relative Mean Uplift is 5.4 % Risk if i choose Treatment is : 0.53% The Growthbook outputs 82% chance to beat control but when i calculated it on the notebook it is 97%. Can you check these numbers please? @fresh-football-47124
h
What notebook are you using to calculate it, can you share it?
Can you also share a screenshot? I get 97% as well when passing the numbers through our stats engine
f
Screenshot 2023-08-16 at 9.37.06 PM.png
Copy code
import numpy as np
from scipy.stats import norm, beta
from scipy.special import digamma, polygamma, roots_hermitenorm  # , roots_sh_jacobi
from scipy import linalg
from scipy.special import betaln, eval_jacobi, roots_sh_jacobi
Copy code
def update_beta_prior(x_a, n_a, x_b, n_b,alpha_0 = 1, beta_0=1):
  alpha_a = alpha_0 + x_a
  beta_a = beta_0 + n_a - x_a

  alpha_b = alpha_0 + x_b
  beta_b = beta_0 + n_b - x_b

  return alpha_a, beta_a, alpha_b, beta_b
Copy code
def calculate_beta_binomial(alpha_a, beta_a, alpha_b, beta_b):
  log_beta_mean = lambda a, b: digamma(a) - digamma(a + b)
  var_beta_mean = lambda a, b: polygamma(1, a) - polygamma(1, a + b)

  d1_beta = norm(loc=beta.mean(alpha_b, beta_b) - beta.mean(alpha_a, beta_a),
                 scale=np.sqrt(beta.var(alpha_b, beta_b) + beta.var(alpha_a, beta_a)))
  d2_beta = norm(loc=log_beta_mean(alpha_b, beta_b) - log_beta_mean(alpha_a, beta_a),
                 scale=np.sqrt(var_beta_mean(alpha_b, beta_b) + var_beta_mean(alpha_a, beta_a)))

  return d1_beta, d2_beta
Copy code
x_a, n_a = 1287, 2384  # converted & total users in A
x_b, n_b = 1271, 2234  # converted & total users in B
Copy code
alpha_0, beta_0 = 1, 1  # Beta prior

alpha_a, beta_a, alpha_b, beta_b = update_beta_prior(x_a, n_a,x_b, n_b, alpha_0, beta_0)
d1_beta, d2_beta = calculate_beta_binomial(alpha_a, beta_a, alpha_b, beta_b)
print(f'The probability the conversion in B is higher is {round(d1_beta.sf(0) * 100, 2)}%')
print(f'The 95% credibility interval of (p_b/p_a-1) is {(np.exp(d2_beta.ppf((.025, .975))) - 1)}')
Thus is 
Itamar Farhan
code from medium that i use to double check these numbers.
h
I'm not able to recreate this within GrowthBook, so something is happening between the data being passed to our stats engine and the data shown in the results ui
Can you share the metric settings (the right panel on the metric page) and the analysis settings in the Config tab?
And can you also click the three dots next to the results, select view queries, and share the full query response for this metric?
f
cc: @helpful-application-7107
h
Two things: are you self hosted? This looks like an old set of queries. You have a custom denominator which may be causing the difference and our front end might not be showing the right denominator for some reason but it might be hard to debug.
f
Yes we do have custom denominator.
h
Yeah so the variance can get complicated in that case depending on the denominator type, but it's going to be hard for me to debug as your GrowthBook looks like it's 6+ months old.
I don't think it's a mistake in the calculation of chance to win. Is the denominator a count or a binomial?
f
It's count.
First solution is to update Growth book.
h
Yeah so the calculation isn't wrong
The variance is different for ratio metrics. It relies on the Delta method to produce a different variance estimate than if it were a regular binomial metric.
There's no bug here, it's just a little unclear that this computation is different because it's a ratio metric
79 Views
Open in Slack
PreviousNext
Powered by

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.

Powered by