We're running our first A/B-test, it's only been running for two days. Getting this "suspicious result" in one of the variants, (for one of the metrics). Should we be worried or is it normal in the beginning of the test period? I'm guessing there's nothing to worry about but just want to double check.
05/31/2023, 2:50 PM
Suspicious results mean that the change in the metric is so great, it might be running into Twyman’s Law
you can adjust the ‘max percent change’ threshold for that metric if you believe the results are real