Hi! I have a question about user bucketing:
• For an ongoing test, if we’d like to add more test cells, can we ensure that users already bucketed in certain cells don’t get moved to another test cell? Is there a way to only re-distribute users from a “control” cell into the new test cells, or does adding more cells re-distribute the bucketing from the whole test?
(I asked a similar question a few months ago, not sure if anything has changed since then)
01/25/2023, 3:10 PM
It's still not possible today, although we've been talking a lot recently about ways we can safely enable changes like this mid-test while keeping the statistics accurate and user experience smooth.
01/25/2023, 4:08 PM
Follow up question just for my understanding: Would you be fine "restarting" the analysis from the new phase where you added new test cells (and just making sure that old users did not change experience)? Or would it be very important to keep their old conversion data in your experiment?
01/25/2023, 7:36 PM
We would be fine restarting the analysis from the new phase. The goal is really just not to confuse users too much by switching them in and out of cells too often
01/25/2023, 7:37 PM
Right, so as Jeremy said this still isn't supported but we're actively investigating this kind of mid-test change. Thank you for clarifying the use case.