wide-jordan-14758
03/09/2023, 8:53 PMhelpful-application-7107
03/09/2023, 9:06 PMwide-jordan-14758
03/10/2023, 9:26 PMhelpful-application-7107
03/10/2023, 9:38 PMFrom our PoV, we do not need the conversion window. After a user is exposed to a test, we do not want to filter out any data.Good news: our refactor will handle this use case out of the box and is about 80% the runtime on our test data. We are finalizing the naming, but it will be something like
Full History
or First Exposure until End of Experiment
that we intend to use to replace Multiple Exposures
.
Bad news: we don't really have an existing way to do the analysis you want to do. You're correct about the way it works now; each day will have all users exposed on that day + the conversion window starting on that day (each day a user is bucketed on kind of treats that user as if they were a totally new user). After our refactor, each day will be just users bucketed for the first time on that day and their metrics will be from either their first conversion window, or from the start of that window until the end of the experiment depending on whether you choose First Exposure
or Full History
Therefore, the time series will show you how effects have changed over when a user first entered an experiment, but not the average difference between group A and group B (for all users in that group) on day X.