Hey GrowthBook team, thanks for your work on a ver...
# announcementsf
famous-potato-10766
05/10/2022, 5:52 PMHey GrowthBook team, thanks for your work on a very useful product!
A few questions:
1. Why is it necessary to archive experiment results when increasing percentage traffic? From the dashboard: “Changing the traffic percent or split will start a new phase of the experiment. All previously collected results data will be archived and it will start fresh from this point on.”
2. Could you advise on the “correct” time to interpret experiment results? We’ve had an experiment drop in and out of statistical significance, looking at “Chance to Beat Control” in the dashboard. I figure it’s not great to look every day and decide the experiment’s done on the day it shows >95% 🙂
f
fresh-football-47124
05/10/2022, 5:55 PM
Hi Kevin, thanks...
fresh-football-47124
05/10/2022, 5:59 PM
1. If you change variation assignment weights without making a new phase of the experiment, the data quality checks will start to throw warnings.
2. For data, we recommend at least 1 week of data, but for numerical limits for calling an experiment, it usually 'depends'. We give you both the risk and the chance to beat control - and one or the other (or both) can be used as your determining factors to end an experiment. Bayesian statistics is less susceptible to peaking problems, so its okay to look.
👍 1
f
famous-potato-10766
05/10/2022, 6:54 PMThanks. For #1: a new phase for a different split makes sense, but why is a new phase required when expanding the traffic percentage? E.g., why would increasing from 20% of traffic to 50% of traffic require archiving data collected to date?
f
fresh-football-47124
05/10/2022, 8:41 PM
you shouldn't have to change the phase for increasing the traffic percentage
f
famous-potato-10766
05/10/2022, 8:45 PMGot it, thanks! My confusion was on the phrasing in the UI here which mentions percent:
➕ 1
famous-potato-10766
05/10/2022, 8:45 PMThanks for your help!
f
fresh-football-47124
05/10/2022, 8:45 PM
ya, sorry about that - ya, we need to improve that language
fresh-football-47124
05/10/2022, 8:46 PM
it you adjust the traffic percentage with a feature flag, you don't need to change the experiment report at all
f
famous-potato-10766
05/10/2022, 8:47 PMI see. We've started with defining experiments inline, so we haven't defined feature flags in the dashboard yet.
27 Views