Another thing, I feel like it wound be great to have an option to automatically stop an experiment when the pre-defined sample size has been reached. Now I have to do it by hand and sometime even have to edit the query to “revert” sample size to the pre-defined one so that I don’t inflate my p-values

I've also been thinking some sample size controls would be great. At least to track progress toward a defined sample size and an alert if reached if not the option to auto stop when reached. But why @some-planet-44104 would you try to drop from your sample if it goes over? I'm not aware of any reason that would bias your p-values. Perhaps power your test above an effect size of interest but that doesn't seem like a bad thing.

Hi Meritt. @billions-house-96196 Sorry for the late reply. Monitoring the results and calling the shots past the sample size the experiment has been designed for inflates false positive rate in much the same way it does if we analyze the experiment and call the shots before the set sample size has been reached. Of course it’s more of a question of whether we wait past the sample size for the results to get significant, but I feel safer to follow strict protocol.