In a recent experiment, we observed a notable contrast between users who were part of the experiment and those who were not. The biggest difference we saw between the in-experiment and out-of-experiment populations was in the percentage of new user accounts - there were significantly less 0D users (playing on the day of registration) in the experiment population, and more 1 year+ users. Is there a reason why this might have been the case?

Was this an A/A test?

It was an A/B test.

okay - curious as to how do you that the difference was due to a sampling imbalance vs the effect of the experiment?

Have you checked the pre-exposure of the groups? Meaning, was this difference already present before the experiment?

We're going to dig a little deeper. Thanks guys.