hey @helpful-application-7107 - I wanted to get some clarification on this change: https://github.com/growthbook/growthbook/pull/2160 Does this mean that CUPED wasn't previously being applied in the case described? We had some data scientists bring up that they saw "% change" values change after we updated our self-hosted deployment to pull in this commit. Would the difference in the two screenshots (one where % change was positive is from before deploying that commit, where one with negative % change is after) be expected behavior from having CUPED enabled?

Hey Mike.

Does this mean that CUPED wasn't previously being applied in the case described?

Unfortunately yes. I thought the magnitude of the issue was quite small because the override had to have been saved with the value set to off, and then disabled. But maybe I missed some edge case where this applied more broadly and we should have alerted more folks. I'd be happy to do any debugging here to help you figure out how this affected your data (the worst case scenario is you just thought you were getting variance reduction when you actually weren't for some subset of metrics).

Would the difference in the two screenshots (one where % change was positive is from before deploying that commit, where one with negative % change is after) be expected behavior from having CUPED enabled?

Yes, those screenshots would be consistent with left being no CUPED, right being CUPED. The CI is quite a bit tighter on the right image, and the % change isn't a direct computation of the group means, which, while potentially confusing, is also the way it will look with CUPED. Lastly, this sign flip is totally reasonable given the uncertainty in the left image.

sweet, no worries. I can forward that on. really appreciate the context