Hello, I am exploring the Experiment Analysis part of Growthbook and maybe I am missing something in the configuration. I do not see any kind of sample size calculation in ex-ante analysis of experiments. results (and p-values in frequentist engine or probabilities in bayesian) appear right after having hit the minimum sample size, but I do not see any advice in the UI regarding the fact that the sample size has not been reached yet. So for a non-expert user the risk of peeking is high. Is there some configuration that I am missing? What I am expecting here is that based on historical data, there is some possibility to alert the user that the conclusion are premature to look at.

@helpful-application-7107?

Hi Alessandro. You're correct that this does expose people to peeking risk. Three things. 1. We plan to, for both our Bayesian and Frequentist engines, build in estimates of needed sample size to achieve some level of certainty (e.g. power for some effect size in the Frequentist case). This work is being planned and should arrive later this year. 2. For now, you can use the sequential testing (see your org settings) to prevent false positives from peeking, all though this comes at a large cost to power 3. You can also set a Minimum Sample Size on a metric that will hide results until some metric value has been reached. This is a blunt instrument, unfortunately, but it is all that exists. I think you have found this. As for "but I do not see any advice in the UI regarding the fact that the sample size has not been reached yet" we do have something for the Bayesian engine but not the Frequentist engine. I'll open a ticket for this, although I suspect we'll wait until we have a robust power calculator in the app to resolve this issue.

Thank You very much guys for the prompt response and glad to hear that Luke! It will be really cool and I guess that in the meanwhile I will stick to bayesian and/or higher threshold in metrics to minimize the temptation of peeking for some users! thank you