Hello GB team. I'm trying to setup an experiment a...
# ask-questionsb
brash-raincoat-88597
05/30/2022, 8:00 AMHello GB team. I'm trying to setup an experiment analysis using Growthbook and I have a question. One of the metric that we're tracking (customer's engagement score) does not seem to map to one of the metric types available (binomial, count, duration, revenue). This score is basically our own formulation of how engaged a customer is with our product, which takes into account multiple things (usage frequency/magnitude, social media engagement, support/feedback, etc) and it's a number between 0 and 1. We then run experiments to see the effectiveness of certain features in lifting that score and I'd like to see if I can analyse the experiments using Growthbook. Do you have any suggestion on how I should set this metric in GB?
f
fresh-football-47124
05/30/2022, 8:11 AM
Hi Andreas
fresh-football-47124
05/30/2022, 8:17 AM
You can use duration or revenue, even though it's not a perfect match. The only real difference between count, revenue, and duration metrics are how we format the values when we display them. For count, duration, and revenue metric if there are multiple rows for a user, we sum the values together. So if a user is put into an experiment and then makes two $10 orders, we would record $20 for a revenue metric. For your score, is it unique per user (or whatever randomization unit you're using)?
b
brash-raincoat-88597
05/30/2022, 8:30 AMHi Graham, thanks for the response. Yes that's a unique score per user, a decimal value between 0-1, with higher value the more desirable, and this is recalculated on daily basis. For this metric, each row in our data warehouse may look like this:
user_id | date | score
1 | 2022-05-30 | 0.5
1 | 2022-05-31 | 0.6
So now if I understand it correctly, I can set it as either duration or revenue, but I need to avoid returning multiple rows per user for each experiment, is that correct? If yes, how can I set it in GB such that it only query the metric for the date the experiment ends? (or, in some experiment, our data scientists actually want to see the score X days after the experiment ends)
f
fresh-football-47124
05/30/2022, 8:45 AM
We support some date placeholders in the queries
fresh-football-47124
05/30/2022, 8:47 AM
{{startDate}} or {{endDate}} - but I'm looking if endDate will do what you need - otherwise you could do something like coalesce() with endDate or NOW()... reading some code
fresh-football-47124
05/30/2022, 8:47 AM
yes, endDate is either the date you stopped the experiment, or the current date if its still running
👍 1
fresh-football-47124
05/30/2022, 8:59 AM
I'd be a bit concerned about the causal nature of the metric if you're only looking at the last day
b
brash-raincoat-88597
05/30/2022, 9:05 AMendDateI supposed that's referring to the template variables described here? https://docs.growthbook.io/app/metrics#sql-template-variables
brash-raincoat-88597
05/30/2022, 9:23 AMI'd be a bit concerned about the causal nature of the metric if you're only looking at the last dayHmm.... why is that? 🤔 Maybe there's something I'm missing. So in this specific experiment, we randomly split a segment of the users into 2 groups (control & variant) for 2 weeks. The variant is exposed to a new app feature which does not exist in control. At the end of the 2 weeks period, we would like to see if there is an uplift in the score distribution of the variant compared to the control. The score is re-calculated every day for every user in our population, so there will be one score per user on the last day. Assuming there's no other experiment/feature flag changes running in the 2 weeks period, shouldn't it be ok to only look at the score once the experiment ends?
f
fresh-football-47124
05/30/2022, 9:24 AM
ya, that's okay as you describe
b
brash-raincoat-88597
05/30/2022, 9:27 AM👍 thanks for the quick responses
34 Views