I am seeing this "value changed too much" notification and while I understand the concept and the possible problem with the experiment premises and setup, I'd like to know how I go about this problem. After checking the experiment setup, everything looks fine and the tracking works OK. Should I wait for the "suspicious results" get less suspicious or is it the final verdict?
Looks like our user behaviour is rather different depending on the day of week, so maybe that's that. Should I wait for more data?
12/14/2021, 11:20 PM
You can change the suspicious threshold on a per-metric basis if you need to. Some metrics have a lot of variance and will result in false positives with the default threshold.
If you edit a metric and go to the "Behavior" tab, you can set the "Max Percent Change" value to something bigger.
12/14/2021, 11:21 PM
Oh. Right. Seen this. Didn't know there was a connection. Thanks for the tip!
Yeah. 100% probability of outperforming the original is not something I've seen too often