https://www.growthbook.io/ logo
b

billowy-hospital-28386

04/17/2023, 9:15 PM
In a recent experiment, we observed a notable contrast between users who were part of the experiment and those who were not. The biggest difference we saw between the in-experiment and out-of-experiment populations was in the percentage of new user accounts - there were significantly less 0D users (playing on the day of registration) in the experiment population, and more 1 year+ users. Is there a reason why this might have been the case?
f

fresh-football-47124

04/17/2023, 10:34 PM
Was this an A/A test?
b

billowy-hospital-28386

04/18/2023, 2:16 PM
It was an A/B test.
f

fresh-football-47124

04/18/2023, 3:23 PM
okay - curious as to how do you that the difference was due to a sampling imbalance vs the effect of the experiment?
👍 1
g

gray-crayon-34592

04/19/2023, 7:48 AM
Have you checked the pre-exposure of the groups? Meaning, was this difference already present before the experiment?
👍 1
b

billowy-hospital-28386

04/20/2023, 5:23 PM
We're going to dig a little deeper. Thanks guys.