square-leather-9869205/15/2023, 12:01 PM
helpful-application-710705/15/2023, 2:09 PM
Unbeknownst to me I had predominately been using “Frequentist” statistical approaches in the past at companies with a bit more scale in terms of ecommerce (Just Eat). I’m unfamiliar with Bayesian approaches (as in I haven’t hand-calc’ed any Bayesian approach to verify a set of results like I have with basic Chi-Squared or even simpler T-tests) but I just read your white-paper on the topic.Just as a note, you can select a Frequentist engine if you prefer in under your general Organization settings.
what should I see if I were to pass the “not enough data” barrier (12 days remaining!). Is it a violin plot with a narrow distribution around 0%?I don't know how narrow it would be, but yeah, on average it will be centered around 0. In fact, if you go to your metric definition and click "Edit" as shown in the screenshot below, you can change the "minimum sample size" to just a glimpse of what the results will look like.
More generally, what kind of volume does a site/app need in order to have these experiments resolve under 30 days. Is that even a fair question to ask of Bayesian engines?It is a somewhat fair question to ask of Bayesian engines, but we don't have a strong power calculator built in to the app yet. For now, I would suggest using frequentist power calculators as a reasonable rule of thumb for the amount of traffic you need to get some statistical certainty (or do your own analysis, but that could get really complicated)
late-dentist-5202305/15/2023, 2:18 PM
helpful-application-710705/15/2023, 2:19 PM
square-leather-9869205/15/2023, 3:01 PM
helpful-application-710705/22/2023, 2:02 PM
square-leather-9869205/22/2023, 2:13 PM
helpful-application-710705/22/2023, 3:29 PM