some-planet-44104
02/01/2023, 1:59 PMfresh-football-47124
helpful-application-7107
02/01/2023, 4:52 PM(c) the variance of your metric
and increase your power. But it will also likely decrease (b) the effect size of your experiment
because in reality it is often hard to move very small numbers that happen infrequently, and this would decrease your power.
So in this case, it isn't totally clear what effect it will have. If you want to be more conservative, you'll want to explicitly either (a) just increase that sample size, (b) decrease your expected effect size, or (c) increase the expected variance of your metric.
would it be statistically and mathematically validEstimating power is almost always a guessing game. Tweaking your historical data to be more or less conservative doesn't undermine any statistical validity, it just can change the risk you're taking on when running an experiment. In general, frequentist statistics will be valid so long as you run the experiment to conclusion. Of course, those statistics won't be very useful if they come from an underpowered experiment where you will likely just get a null result. So to that end, I think being more conservative in the situation you're in makes the most sense.
some-planet-44104
02/01/2023, 5:26 PMhelpful-application-7107
02/01/2023, 7:17 PM9% * 0.1 = 0.9%
or 0.09 * 0.1 = 0.009
)
With a base conversion of 8%, a relative effect of 10% is the same as an absolute increase in 0.8 percentage points, or an increase in the proportion converting of 0.008 (8% * 0.1 = 0.8%
or 0.08 * 0.1 = 0.008
).
Therefore, you need more users in the second case since you're actually looking to detect a smaller absolute lift in the proportion of users converting.some-planet-44104
02/02/2023, 3:10 PMhelpful-application-7107
02/02/2023, 3:46 PMsome-planet-44104
02/03/2023, 10:53 AM