Hi - for one of our tests I get the following representation of data, it looks a bit off (risk of choosing = 0% and the graph is not shown) - what could be the issue for that?
also chance of beating is 50%, which does not make sense if the decrease is 29.6%
06/29/2022, 2:23 PM
Looks like the number of conversions is really low. 8 vs 13, so it's way too early to draw any conclusions from the data.
The stats kind of break down when you don't have enough data, which is why we set a minimum sample size of 150 by default. In certain circumstances, it's fine to lower it a little bit, but not that much. You can control the minimum sample size in the metric behavior settings.
06/29/2022, 4:05 PM
mhh… but for all our other tests this did not look like this even though traffic was similarly low…
06/29/2022, 4:29 PM
if the number of conversions is very low, the results will be unreliable. So sometimes it may look fine, sometimes it may look like this. We recommend setting a higher threshold for minimum sample size so that the results are more reliable
06/29/2022, 4:36 PM
Okay, thanks 👍
I mean we will wait anyways for at least one week that the data becomes more reliable but it just looked like this for the first time - so just wanted to be sure if this could also be a bug or so - thanks for your help 👍