https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
g

gorgeous-needle-6147

01/10/2024, 6:00 PM
We have multiple ids that our product experiments are randomized by, e.g. browser cookie, user_id, page_id. The analysis metrics we set up in growthbook only make sense when they match the id that was used for randomizing experiment participants, e.g. we'd use a browser-cookie retention metric for a browser cookie randomized experiment, and a user_id retention metric for an experiment randomized by user_id. What's the best way to make sure that we use the right metrics for analyzing each experiment? Should we come up with a clear naming convention, or is there some sort of programmatic way we can prevent data scientists and product managers from accidentally adding the wrong metric to an experiment? thank you! cc @polite-pillow-33171 @worried-lizard-90583 @adventurous-diamond-953
f

fresh-football-47124

01/10/2024, 6:02 PM
Hi Chris - each metric is assigned to the randomization unit, but I'm not sure if there is a way currently to prevent them from being used. Let me check with the team on a solution here.
g

gorgeous-needle-6147

01/10/2024, 6:05 PM
thank you!
h

helpful-application-7107

01/10/2024, 6:34 PM
Hi Chris. > What's the best way to make sure that we use the right metrics for analyzing each experiment? • Ensure in your Data Source you have an
Identifier Type
and matching
Experiment Assignment Query
in for any id you randomize by. • Ensure your Metrics match to their corresponding
Identifier Type
. Metrics can have more than one
Identifier Type
if your data has multiple ID columns (e.g. you have both cookie_id and user_id in one column). • Make sure your analysts pick the right
Experiment Assignment Query
when analyzing their experiment. This makes sure we look for the right assignment events in your data source to see which `id`s are in the experiment. Ideally, we could tighten the connection between
Hash Attribute
ids and
Identifier Type
, but once you pick the right identifier type by picking the right Experiment Assignment Query/Table, the metrics will follow suite.
Argh! I just realized that you can still add metrics to the experiment even if it has (a) a different Identifier Type to your Metric and (b) there is no Identifier Join Query to map the two identifier types to each other. So I'm not sure I answered your question.
So I guess a naming convention would be best for now, but we should take it upon ourselves to only let you add Metrics if: (a) it has an Identifier Type that matches the one selected for the Experiment (b) or, it has an Identifier Type that has a Join Table defined between it and the Identifier Type of the experiment.
g

gorgeous-needle-6147

01/10/2024, 8:58 PM
thank you! that's helpful to know, and yes that feature would be great to have in the future
f

fresh-football-47124

01/10/2024, 10:35 PM
we created an issue for this, if you want to follow along: https://github.com/growthbook/growthbook/issues/2020
g

gorgeous-needle-6147

01/10/2024, 10:42 PM
will do, thanks!
4 Views