Hi Growthbook team! May I have a bit of advice? I wanted to start using experimentation in order to help make more data-informed decisions within my company. The company suffers from the classic “knee jerk” feature-build response to problems so, I wanted to introduce more rigour into what work is prioritised and subsequently built. Crappy ideas are cheap to demand, expensive to build! However, I believe the client’s sites just simply aren’t doing enough volume for quant methods to resolve and I have a stubborn “not enough data” lozenge within our test experiment results. Unbeknownst to me I had predominately been using “Frequentist” statistical approaches in the past at companies with a bit more scale in terms of ecommerce (Just Eat). I’m unfamiliar with Bayesian approaches (as in I haven’t hand-calc’ed any Bayesian approach to verify a set of results like I have with basic Chi-Squared or even simpler T-tests) but I just read your white-paper on the topic. So, my question; I got started with GrowthBook by doing my usual A/A test integration to ensure distribution and to correct our analysis SQL. What should I see if I were to pass the “not enough data” barrier (12 days remaining!). Is it a violin plot with a narrow distribution around 0%? The reason I ask is that I need to buy more time so that I can actually get onto the A/B without the whole idea being scrapped! More generally, what kind of volume does a site/app need in order to have these experiments resolve under 30 days. Is that even a fair question to ask of Bayesian engines?

Unbeknownst to me I had predominately been using “Frequentist” statistical approaches in the past at companies with a bit more scale in terms of ecommerce (Just Eat). I’m unfamiliar with Bayesian approaches (as in I haven’t hand-calc’ed any Bayesian approach to verify a set of results like I have with basic Chi-Squared or even simpler T-tests) but I just read your white-paper on the topic.

Just as a note, you can select a Frequentist engine if you prefer in under your general Organization settings.

what should I see if I were to pass the “not enough data” barrier (12 days remaining!). Is it a violin plot with a narrow distribution around 0%?

I don't know how narrow it would be, but yeah, on average it will be centered around 0. In fact, if you go to your metric definition and click "Edit" as shown in the screenshot below, you can change the "minimum sample size" to just a glimpse of what the results will look like.

You should also check your conversion window settings and whether you are including or excluding in-progress conversions.

That's a good suggestion, Scott, thanks. Another alternative is to use the Experiment Duration attribution model which will ignore conversion windows and roll up all metric values from first bucketing of a user through the end of the experiment (or run time, if the experiment is live).

great! thanks for your suggestions both! That’s really useful.

This looks very normal for an A/A test. The sample size is pretty small to rule out a problem but this is exactly what I would expect and hope to see from an A/A test.

Thanks Luke!

Awesome. Let us know if you have other questions.