brash-raincoat-8859705/30/2022, 8:00 AM
brash-raincoat-8859705/30/2022, 8:30 AM
brash-raincoat-8859705/30/2022, 9:05 AM
I'd be a bit concerned about the causal nature of the metric if you're only looking at the last dayHmm.... why is that? 🤔 Maybe there's something I'm missing. So in this specific experiment, we randomly split a segment of the users into 2 groups (control & variant) for 2 weeks. The variant is exposed to a new app feature which does not exist in control. At the end of the 2 weeks period, we would like to see if there is an uplift in the score distribution of the variant compared to the control. The score is re-calculated every day for every user in our population, so there will be one score per user on the last day. Assuming there's no other experiment/feature flag changes running in the 2 weeks period, shouldn't it be ok to only look at the score once the experiment ends?
brash-raincoat-8859705/30/2022, 9:27 AM