We ran a experiment and have a clear winner. We want to roll-out the feature to next set of users to the winner variant but we want to keep the experience of previous set of devs who are still part of the experiment same. We do not want to change the experience of devs who are still part of the experiment.
Can we do it ? If yes, how?
07/11/2023, 6:46 PM
Unfortunately, we don't currently have a way to roll out a feature that keeps the old user's experience while enrolling all new users in one of two variants.
It's possible you could make something like the following work, but it may be difficult to execute:
1. Create a forced rule for users with "new to experiment" attribute that get the winning variant, and let the rest go through to the existing experiment rule.
2. Create this "new to experiment" attribute. This is the difficult part, because knowing if a user was exposed to an experiment requires logging somewhere for that user if they've seen the experiment before and using that as an attribute you can pass to the SDK. An alternative is to at least force users with some account creation date later than X, where X is the date that you add the forced rule, to all get the winning variant. This would make all new users (who could not have seen the experiment, assuming it was a logged-in experiment) go to the winning variant. Then all old users, regardless of whether they were exposed to the experiment or not, would go to the experiment.
We have on our roadmap changes to Features that should make this kind of rollout easier in the future, but they are not coming in the short term.
There may be other options depending on what % of users you had in the experiment to begin with, but you will still face the above issue even if you only had 10% of traffic in your experiment.