prehistoric-king-1395010/26/2023, 11:42 AM
helpful-application-710710/26/2023, 3:15 PM
The drift always seems to overweight the control,I'd love to see other evidence of this, but I'm not seeing anything too crazy here. This kind of imbalance in that screenshot is fairly normal. The associated p-value is around 0.14.
For guardrail metrics, its making all variants look worse due to lower conversions.It shouldn't cause lower conversions on average which is what our metrics look at. Sometimes chance imbalance causes spurious results, but just a slight imbalance in numbers shouldn't cause the average conversion rate to be different.
prehistoric-king-1395010/27/2023, 9:24 AM