Is it safe to stop the experiment when 95% threehold has been reached?
I am not big on statistics but prior to Growhbook I've always calcualated the required sample size and probably used what's known as "Frequentist approach", but I imagine things work different in Growhbook since it uses bayesian approach. From my understanding, this approach allows to stop the experiment when the threeshold has been reached but I am not sure.
01/14/2022, 4:43 PM
There are lots of factors to consider when stopping an experiment.
1. You probably want to run for at least a full week to get both weekend/weekday traffic (unless it's looking really bad, then you can stop earlier)
2. If the Risk column is really low for your key metrics, it's safe to stop since even if you're wrong it won't have a bad impact.
3. If Chance to Beat Baseline for your key metric is above 95% or below 5% and the Risk column isn't too big, it's also a good indication you can safely stop.
4. If there are any external factors like a big marketing campaign or holiday that are affecting results, you might want to stop the test to avoid conflicts.
01/14/2022, 5:27 PM
Got it. Looks like the figures are good indication for stopping the experiment as long as the audience characteristics remain consistent. Thanks!