This message was deleted.

If a minimum sample threshold is reached and one of the goal metrics for a variation passes either a 5% or 95% level for "chance to beat control", Growth Book sends an email to you. We want to improve this heuristic in the future to be a lot smarter.

hmm, that's not really the stopping condition I want. I would prefer something that considers cost and improvement in % from each additional sample.

We do show the Bayesian Risk (or expected loss) for each metric, which is often used as the stopping condition. There are usually many variables and external factors that I don't think the stopping decision can be entirely left up to an algorithm. For example, if one metric is up, one is down. Or the metrics are slightly down, but you believe the lower maintenance cost will be worth it.

thanks for the answer. I don't agree at all, because my background in behavioral economics espouses a pessimistic view of human judgment in statistical environments with low probabilities, but your perspective is informative and stems from a valid viewpoint.