We are running an experiment . We have allocated 10% of traffic to the A/B experiment. Now we want to ramp up the traffic to 50% without disturbing the user experience? I have few question.
1. Will the ramping up will lead to resampling of the users?
2. Is there any chance that previous users who are part of the experiment may be thrown out of the experiment?
3. How do we ensure that resampling does not happen and users are not thrown out of the experiment?
01/11/2023, 7:35 PM
If you keep the traffic split between the variations the same, then there won't be any resampling as you scale up. For example, if you started with 10% of users and a 50/50 split, then want to scale up to 50% of users, still with a 50/50 split, it will work fine as expected.
But if you keep the rollout percent the same and change from a 90/10 split to a 50/50 split, then users will be resampled.
So bottom line, keep the traffic split the same and just adjust the percent rollout.
01/12/2023, 1:55 AM
Thanks for the response @future-teacher-7046. Next follow up question is what happens when we scale down.
For example keep the traffic split between variations same but instead of scaling up from 10% to say 50% , i scaled down from 50% to 10% -> how this would impact the experiment?
1. Will the users be resampled?
2. Or the GB will exclude 40% of users from the experiment and serve those 40% users the default behaviour?
01/12/2023, 2:10 AM
It's #2. If you keep the traffic split the same, it will never do resampling.