Hello everyone, We've been running an A/A test for over a week now but the results are not the expected for a healthy A/A. • P Value is doing fine • but the %Change is in negative (which I understood from the documentation that it should be 0-100) • the chance to win is Very low when it should've been around 50%. the implementation was double checked from our devs but we're still not sure what could be the issue here. any tips?

Hi Auns, thanks for your questions. Regarding "P Value is doing fine": Is this the Experiment Balance Check p-value you are referring to? If so, this number tests if the observed proportions in control and treatment (49.33% and 50.67%) match a 50/50 split across control and treatment. This p-value is 0.41, and we do not see evidence for imbalance. Regarding "but the %Change is in negative (which I understood from the documentation that it should be 0-100)": if the control mean is larger than the variation mean, then % change can be negative. For example, if the control mean is 100, and the variation mean is 50, then the % change is (50 - 100) / 100 = -0.5 = -50%. Regarding "the chance to win is Very low when it should've been around 50%.": The interval shown for

ATC_mini_miswag

is almost wholly below 0, and the chance to win is appropriately very small. If this is an AA-test, you may want to check your implementation, as the variation mean is much smaller than the control mean. This can occasionally happen by chance, but it's worth checking your implementation a third time, or potentially rerunning another AA test. Please see our documentation. Thanks, Luke