In a recent experiment, we observed a notable contrast between users who were part of the experiment and those who were not. The biggest difference we saw between the in-experiment and out-of-experiment populations was in the percentage of new user accounts - there were significantly less 0D users (playing on the day of registration) in the experiment population, and more 1 year+ users. Is there a reason why this might have been the case?
04/17/2023, 10:34 PM
Was this an A/A test?
04/18/2023, 2:16 PM
It was an A/B test.
04/18/2023, 3:23 PM
okay - curious as to how do you that the difference was due to a sampling imbalance vs the effect of the experiment?
04/19/2023, 7:48 AM
Have you checked the pre-exposure of the groups? Meaning, was this difference already present before the experiment?