https://www.growthbook.io/ logo
b

brash-raincoat-88597

10/04/2022, 1:00 AM
Would an uneven split of control/variant makes a difference in the Chance to Beat Control? I have an experiment with a 25/75 control/variant split in a 400k segment, which I thought would have enough samples given the effect we expected to see (~5-10%), but it turns out I’m seeing only a ~26% chance to beat control
f

future-teacher-7046

10/04/2022, 1:01 AM
If your experiment does not have an effect on users, the "Chance to Beat Control" will stay around 50% forever, no matter how much data you collect.
If there is a difference, it will tend towards either 0% or 100% as you gather more data. Given you have plenty of users and it's still in the middle means the experiment likely is not affecting your metric.
b

brash-raincoat-88597

10/04/2022, 1:10 AM
Ahh right, it’s a chance… okay I understand the 100% part, but when would it reach 0%?
f

future-teacher-7046

10/04/2022, 1:10 AM
If it's worse than the control, it will tend towards 0%
b

brash-raincoat-88597

10/04/2022, 1:11 AM
Got it, thank you very much
2 Views