https://www.growthbook.io/
Join Slack
Hi, a general question. what’s this `suspicious result` means? Is there a explanation doc for these ...
# announcements
h
Hi, a general question. what’s this 
suspicious result
means? Is there a explanation doc for these kind of labels?
f
Hmm, that's not supposed to show up with small amounts of traffic since there's usually a lot of fluctuation at the start of experiments. I'll fix that. Once you collect a lot of data, it's extremely rare to see huge changes in metrics. Sometimes it's real, but often it's the result of a bug somewhere.
If you do have metrics with a lot of variability, you can change the suspicious data threshold on a per-metric basis. The default is set to 50%. If you edit the behavior settings for the metric and go into advanced options you should see it.
h
Ok, this experiment was just start half day ago and still collecting more data. What I’m curious about what’s the underlaying concept of such suspicious things. and I can see there are some advanced settings on the metric itself, but I don’t know what’s those advanced settings means
Like what’s the risk thresholds means here.
f
The threshold for suspicious data is called "Max Percent Change"
The concept behind all of our checks is to make sure you aren't making decisions based on faulty data. If conversion rate doubles, it's possible it's just an amazing experiment, but most likely there's a bug (e.g. double firing the conversion event). It comes from Twyman's Law:
the more unusual or interesting the data, the more likely they are to have been the result of an error of one kind or another
🙌 1
112 Views
Open in Slack
PreviousNext
Powered by

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.

Powered by