Hi! I have a request to adjust the retention metric calculation. Currently, the denominator includes all users, which I believe is incorrect for longer-term metrics. Take Week 1 Retention, for example. During the first 7 days after the experiment starts, the metric is 0 divided by the total number of users, and only gradually starts increasing afterward. This creates a misleading picture, as the metric naturally grows over time. I think the denominator should include only users who had a chance to reach 7+ days in the app — in other words, those whose exposure date allows for observing their Week 1 behavior. This way, we can avoid the issue of the metric artificially inflating throughout the experiment just due to time passing.

Hi Vladislav, I'm a data scientist at GrowthBook. Thanks for your question. You are right that the denominator incorrectly includes all users for retention metrics. We do this because: 1. it reduces query costs 2. the bias due to incorrectly including these users attenuates over time If desired, you can see how big of an impact these exclusions have on retention: 1. for your experiment, go to the

Overview

tab 2. select

Edit

3. under

Metric Conversion Windows

select

Exclude In-progress Conversions

We may update this logic down the road. Luke

Hi Luke, Thanks for the explaination.

the bias due to incorrectly including these users attenuates over time

That's true, and I didn't really pay attention to the Retention Daily 1-7 metric, realizing that there was only one day that wasn't counted. The problem is visible on Retention Weekly 1, for example. Let's say the experiment runs for three weeks, and we only have two weeks of retention for the first week. This means that one-third of the denominator is incorrect. Of course, you can extend the experiment to several months and smooth out this effect, but from a business perspective, this is not reasonable.

I agree that it is not ideal. If your traffic uniformly arrives, then you are right, your denominator is too big by 1/3. If you have logged-in experimentation (many repeat users), then perhaps most of your new users will arrive at the beginning of the experiment, and the bias is not too strong.