Hi team, would like to ask a few things wrt Growthbook A/B Test and Metric/Analysis setup. For context, our team have setup multiple environment within Growthbook Dashboard to match setup on our end for testing purposes (ie dev, QA, staging, production environments) Now comes to the question 1. Despite using the same experiment key across environment to run an experiment, is it safe to assume that Growthbook will maintain separate bucket for each environment to do randomization unit? Essentially, it will not randomize the identifier from different environment within the same bucket? 2. How Growthbook can accurately analyze A/B test result for specified environment? Do we need to fire environment information as part of the event tracking so we can exclude/include certain environment when setting up Growthbook metric? Thank you

Hi Brian, welcome 👋 1. The randomization function uses only the experiment key and the user attribute you've selected. This means that if you have the same user id (assuming that's what you're using in the experiment), and the same experiment on dev and production, you would be bucketed the same. Typically we see dev SDK implementations not actually fire the tracking event to any production event tracking/data warehouse. 2. Most organizations we see will have slightly different SDK implementations per environment, so that dev doesn't fire events to any production event destinations. Usually it is only necessary to see that GrowthBook is assigning users to the experiment on dev, and it is not needed to run any actual AB test there. You are welcome to pass any environment to the trackingCallback so it can be used in the SQL to segment traffic. 3. We require no 'environment' in either the assignment or metric query. There is nothing stopping you from adding that, and using that in the queries (you would only need to do it in the assignment query), and then having one assignment for prod, one for staging, etc. But again, it is usual to only track events from production.

hi @fresh-football-47124 thanks for the prompt response > The randomization function uses only the experiment key and the user attribute you've selected. > the first question is more toward, how would GB ensure the intended split toward the treatment groups within each environment (independent from the event firing) Lets just assume the following (exaggerating example): • there are 1000 user identifiers in dev environment and 30 user identifiers in production environment. And they are completely unique from each others • there are 2 groups in total (1 control 1 treatment) for dev env • there are 3 groups in total (1 control 2 treatment) for prod env How GB ensure the production version will have close to 10 users for each group knowing the randomization will account all 1030 unique user identifiers from all environment instead of randomizing it for each environment on 2nd thought, i think it will still randomize accurately per environment because each environment will have different rules, setup, and treatment > You are welcome to pass any environment to the trackingCallback so it can be used in the SQL to segment traffic. > how to pass/add environment information to the trackingCallback? isnt that something GB handles and pass to client, out of the box? Assuming environment can be passed into trackingCallback. how can it be used in the SQL to segment traffic? Understood that most other implementation does not necessarily fire event tracking in non production environment. But in the case it does like ours, how would you suggest us to setup the A/B Test, Assignment Query and Metric? This is what i have in mind, but happy to take feedbacks 1. Fire experiment exposure event and attach environment context 2. Fire intended/measured event and attach environment context 3. Given we intended to measure metric for production environment only, then set up Assignment SQL Query for the intended identifier and include prod event only 4. Setup Metric queries. From your explanation above, it does not seem like we need to point Metric query to production env given that should have been taken care and filtered out by Assignment Query done in step no 3 Assuming they are all valid, we may not need to do step no 2 also as long as the environment context is attached to experiment exposure event.

We don't automatically pass any info to the event tracker

thanks for confirming!

you don't need to attach the environment context to the conversion events you're measuring (step 2) actually

yep, i think so too