#announcements
Title
h
helpful-carpenter-3208
03/22/2022, 1:19 PMHi, a general question. what’s this
suspicious resultf
future-teacher-7046
03/22/2022, 1:30 PM
Hmm, that's not supposed to show up with small amounts of traffic since there's usually a lot of fluctuation at the start of experiments. I'll fix that. Once you collect a lot of data, it's extremely rare to see huge changes in metrics. Sometimes it's real, but often it's the result of a bug somewhere.
If you do have metrics with a lot of variability, you can change the suspicious data threshold on a per-metric basis. The default is set to 50%. If you edit the behavior settings for the metric and go into advanced options you should see it.
h
helpful-carpenter-3208
03/22/2022, 2:26 PMOk, this experiment was just start half day ago and still collecting more data. What I’m curious about what’s the underlaying concept of such suspicious things. and I can see there are some advanced settings on the metric itself, but I don’t know what’s those advanced settings means
Like what’s the risk thresholds means here.
f
future-teacher-7046
03/22/2022, 2:29 PM
The threshold for suspicious data is called "Max Percent Change"
The concept behind all of our checks is to make sure you aren't making decisions based on faulty data. If conversion rate doubles, it's possible it's just an amazing experiment, but most likely there's a bug (e.g. double firing the conversion event). It comes from Twyman's Law:
the more unusual or interesting the data, the more likely they are to have been the result of an error of one kind or another
🙌 1
11 Views