Ahoi GrowthBook-people, I have a question regarding the feature for CUPED. Our users at idealo evaluated a test with and without CUPED. We observed a change in the relative difference between the means of the baseline and variant in the visualisation (see the two attached screenshots). This surprised our users, and I can’t really confidently explain why this happens. We use frequentist statistics. From your documentation and looking at the original paper linked therein, I understood that CUPED does not change the estimate for the absolute difference between the observed group means. It does, however, change the estimate for the group means themselves. Hence, can you explain why the relative difference changes, i.e. what you normalise the relative difference on? We have even seen trends being inverted due to this behaviour. I’ll link @fresh-football-47124 and @helpful-application-7107 from GrowthBook, and @adventurous-dream-15065 from our side.

First of all, those are huge decreases in your confidence interval widths, so CUPED is doing a ton of work to reduce variance and this seems like a really good use case for it. CUPED absolutely changes the difference in variation means. By changing the estimate for the group means, we necessarily also change the estimate for the difference in those means (unless, by some very random chance, the change in the group means cancels out across both groups to some rounding error). In the original paper, you can see

\Delta_cv

is the CUPED lift estimate, and

\Delta_cv

is going to be different from

\Delta

(introduced right below equation 1 in section 2.1 in their paper). This is because the raw variation averages (in their paper written as

Ybar

) are different from the adjusted variation averages (in their paper written as

Yhat_cv

). That's just to help you see how the quantities are different in their paper. Intuitively, in order for CUPED to work, our estimate of the lift itself has to move. Think of it this way: in some respects CUPED reduces variance by eliminating "improbable" randomizations. If you end up with a randomization (that works well) but for some reason puts slightly less active users in your test variation, then CUPED uses that pre-experiment knowledge that these were less active users to reduce that imbalance's impact on the estimate. That would mean shifting the estimate to account for this chance imbalance. Because this imbalance is due to chance, you're correct that on average, across many runs of the same experiment, the CUPED estimate would be on average the same as the non-CUPED estimate. However, in a single run, the estimate will always change. (BTW, this applies to whether you use relative or absolute effects, because the shift occurs in the variation averages, which is used for both estimates.)

Thanks for the explanation! I think I mixed up a single realisation of the estimator with its expected value. To make sure I understood it, could you please double check whether the math ion the attached picture is correct? It’s very brief. Assuming my scribble is correct, I have another question from the user’s perspective. How should one interpret the corrected relative uplift? Formally, it describes the relative uplift of the corrected group means. Our users, however, tend to interpret it naively as the uplift of the uncorrected means, that are stated in the Experiment Analysis panel. This corrected uplift of n% is then reported as “we observed an uplift on n% in our metric”, which is wrong.

Yes, your math looks right and is a clear exposition!

Ahoi Luke, feel free to take from the exposition what you need for your FAQ.

Hey Peter, sorry I meant to respond to this. From my last investigation, ratio metric CUPED requires an additional unit level self-join to do well, which can be a prohibitively slow query at scale. We are looking in to using experiment dimensions to add post-stratification to our CUPED estimator to reduce variance for both mean and ratio metrics.

Thanks for following up! We will reach out with some lead-time, when the topics gets more traction in our organisation. Looking forward to your implementation and nevertheless, though 😉 FYI @adventurous-dream-15065