https://www.growthbook.io/ logo
#announcements
Title
# announcements
s

square-leather-98692

05/15/2023, 12:01 PM
Hi Growthbook team! May I have a bit of advice? I wanted to start using experimentation in order to help make more data-informed decisions within my company. The company suffers from the classic “knee jerk” feature-build response to problems so, I wanted to introduce more rigour into what work is prioritised and subsequently built. Crappy ideas are cheap to demand, expensive to build! However, I believe the client’s sites just simply aren’t doing enough volume for quant methods to resolve and I have a stubborn “not enough data” lozenge within our test experiment results. Unbeknownst to me I had predominately been using “Frequentist” statistical approaches in the past at companies with a bit more scale in terms of ecommerce (Just Eat). I’m unfamiliar with Bayesian approaches (as in I haven’t hand-calc’ed any Bayesian approach to verify a set of results like I have with basic Chi-Squared or even simpler T-tests) but I just read your white-paper on the topic. So, my question; I got started with GrowthBook by doing my usual A/A test integration to ensure distribution and to correct our analysis SQL. What should I see if I were to pass the “not enough data” barrier (12 days remaining!). Is it a violin plot with a narrow distribution around 0%? The reason I ask is that I need to buy more time so that I can actually get onto the A/B without the whole idea being scrapped! More generally, what kind of volume does a site/app need in order to have these experiments resolve under 30 days. Is that even a fair question to ask of Bayesian engines?
h

helpful-application-7107

05/15/2023, 2:09 PM
Unbeknownst to me I had predominately been using “Frequentist” statistical approaches in the past at companies with a bit more scale in terms of ecommerce (Just Eat). I’m unfamiliar with Bayesian approaches (as in I haven’t hand-calc’ed any Bayesian approach to verify a set of results like I have with basic Chi-Squared or even simpler T-tests) but I just read your white-paper on the topic.
Just as a note, you can select a Frequentist engine if you prefer in under your general Organization settings.
what should I see if I were to pass the “not enough data” barrier (12 days remaining!). Is it a violin plot with a narrow distribution around 0%?
I don't know how narrow it would be, but yeah, on average it will be centered around 0. In fact, if you go to your metric definition and click "Edit" as shown in the screenshot below, you can change the "minimum sample size" to just a glimpse of what the results will look like.
More generally, what kind of volume does a site/app need in order to have these experiments resolve under 30 days. Is that even a fair question to ask of Bayesian engines?
It is a somewhat fair question to ask of Bayesian engines, but we don't have a strong power calculator built in to the app yet. For now, I would suggest using frequentist power calculators as a reasonable rule of thumb for the amount of traffic you need to get some statistical certainty (or do your own analysis, but that could get really complicated)
l

late-dentist-52023

05/15/2023, 2:18 PM
You should also check your conversion window settings and whether you are including or excluding in-progress conversions.
h

helpful-application-7107

05/15/2023, 2:19 PM
That's a good suggestion, Scott, thanks. Another alternative is to use the Experiment Duration attribution model which will ignore conversion windows and roll up all metric values from first bucketing of a user through the end of the experiment (or run time, if the experiment is live).
s

square-leather-98692

05/15/2023, 3:01 PM
great! thanks for your suggestions both! That’s really useful.
So, in the end after following the advice here the answer was that it came out slightly left of 0% but generally no “winner” which is what I was looking for from our A/A test integration. @helpful-application-7107 - may I safely assume that if my sample size was larger (and my integration sound) that violin plot would hunt around 0% and my Guardrail metric would tend toward 50% (a statistical 🤷 )?
h

helpful-application-7107

05/22/2023, 2:02 PM
This looks very normal for an A/A test. The sample size is pretty small to rule out a problem but this is exactly what I would expect and hope to see from an A/A test.
s

square-leather-98692

05/22/2023, 2:13 PM
Thanks Luke!
We have started an A/B as part of a 2nd phase. We’re having a great experience with GrowthBook right now. Congratulations on building such a good product!
h

helpful-application-7107

05/22/2023, 3:29 PM
Awesome. Let us know if you have other questions.
8 Views