Hi Team I have question. We ran an experiment and had a clear winner but some of our metrics have 7 day conversion window. We released the feature to 100% users within few hours of conclusion of the experiment. On day of conclusion the chance to beat control was 98% with 27% mean relative uplift. The same day the feature was released to 100% users including those who are part of the control group. We see that after release the gain is minimising We have 2 day conversion window and 7 day conversion window for same metrics. After 48 hours of the experiment conclusion - the mean relative uplift is 15.9% with probability to beat the control is 95.83% 1. Will it affect our metrics as the users from the control group are also released to the new the feature and some of them may still fall under the 7 day conversion window? 2. Can this be due to the users who are under the control group are now exposed to the new feature and are converting hence leading to the reduction in the gain that we had initially?

Hi Nishant - did you turn off the experiment or change the winning variation to 100%?

Steps that i followed: 1. Stopped the experiment and declared treatment as winner. 2. Disabled the A/B experiment rule under the feature flag. 3. Set the default behaviour of the feature flag to New feature.

okay, cool

No

but yes, it’s possible that some users who were in the control, will now have events caused by the rolled out new variation

We have different conversion window for different metrics. One that we focus on has 2 day and 7 day conversion window.

there are a couple of ways around this

The feature is a clear winner in our case. The discussion is around the actual uplift. The day of conclusion the lift was 27% with Prob(to beat control) was 98% and risk to be around 0.01%. After few days of feature roll-out the initial gain that we had has reduced to 15% uplift with Prob(to beat control) is 95% with risk of 0.1%.

you could change the exp end date back 7 days… but it might not give the same results

The discussion is around the gain we see in the control ->is it due to feature roll-out to 100% of users and some users from the control are taking the test and converting quickly hence minimising the gap between variation and control.

yes, it seems like it

I will look at all the suggestions provided by you. Thanks a ton for clarifying the query.