Would an uneven split of control/variant makes a d...
# announcementsb
brash-raincoat-88597
10/04/2022, 1:00 AMWould an uneven split of control/variant makes a difference in the Chance to Beat Control? I have an experiment with a 25/75 control/variant split in a 400k segment, which I thought would have enough samples given the effect we expected to see (~5-10%), but it turns out I’m seeing only a ~26% chance to beat control
f
future-teacher-7046
10/04/2022, 1:01 AM
If your experiment does not have an effect on users, the "Chance to Beat Control" will stay around 50% forever, no matter how much data you collect.
future-teacher-7046
10/04/2022, 1:04 AM
If there is a difference, it will tend towards either 0% or 100% as you gather more data. Given you have plenty of users and it's still in the middle means the experiment likely is not affecting your metric.
b
brash-raincoat-88597
10/04/2022, 1:10 AMAhh right, it’s a chance… okay I understand the 100% part, but when would it reach 0%?
f
future-teacher-7046
10/04/2022, 1:10 AM
If it's worse than the control, it will tend towards 0%
b
brash-raincoat-88597
10/04/2022, 1:11 AMGot it, thank you very much
31 Views