Hi! I’m trying to understand a bit more of the namespace/grouping of experiments, and how does it (or how is it supposed to) work if new experiments are added to a namespace dynamically?
That’s usually how it works on my setup, an experiment starts and after couple of days/weeks, another overlapping one starts while the first one hasn’t finished yet.
How can I deal with that?
07/05/2022, 11:20 AM
Ideally, you can predict that ahead of time and leave room in the namespace. For example, if you want to allow for 4 simultaneous tests, limit tests to 25% of the namespace.
It's possible to start a test covering 100% of the namespace and change it halfway through to only include 50%. It could introduce some noise to the results and might open you up to Simpson's paradox though.