https://www.growthbook.io/
Join Slack
Hello GB team. I'm trying to setup an experiment analysis using Growthbook and I have a question. On...
# ask-questions
b
Hello GB team. I'm trying to setup an experiment analysis using Growthbook and I have a question. One of the metric that we're tracking (customer's engagement score) does not seem to map to one of the metric types available (binomial, count, duration, revenue). This score is basically our own formulation of how engaged a customer is with our product, which takes into account multiple things (usage frequency/magnitude, social media engagement, support/feedback, etc) and it's a number between 0 and 1. We then run experiments to see the effectiveness of certain features in lifting that score and I'd like to see if I can analyse the experiments using Growthbook. Do you have any suggestion on how I should set this metric in GB?
f
Hi Andreas
You can use duration or revenue, even though it's not a perfect match. The only real difference between count, revenue, and duration metrics are how we format the values when we display them. For count, duration, and revenue metric if there are multiple rows for a user, we sum the values together. So if a user is put into an experiment and then makes two $10 orders, we would record $20 for a revenue metric. For your score, is it unique per user (or whatever randomization unit you're using)?
b
Hi Graham, thanks for the response. Yes that's a unique score per user, a decimal value between 0-1, with higher value the more desirable, and this is recalculated on daily basis. For this metric, each row in our data warehouse may look like this: user_id | date | score 1 | 2022-05-30 | 0.5 1 | 2022-05-31 | 0.6 So now if I understand it correctly, I can set it as either duration or revenue, but I need to avoid returning multiple rows per user for each experiment, is that correct? If yes, how can I set it in GB such that it only query the metric for the date the experiment ends? (or, in some experiment, our data scientists actually want to see the score X days after the experiment ends)
f
We support some date placeholders in the queries
{{startDate}} or {{endDate}} - but I'm looking if endDate will do what you need - otherwise you could do something like coalesce() with endDate or NOW()... reading some code
yes, endDate is either the date you stopped the experiment, or the current date if its still running
👍 1
I'd be a bit concerned about the causal nature of the metric if you're only looking at the last day
b
endDate
I supposed that's referring to the template variables described here? https://docs.growthbook.io/app/metrics#sql-template-variables
I'd be a bit concerned about the causal nature of the metric if you're only looking at the last day
Hmm.... why is that? 🤔 Maybe there's something I'm missing. So in this specific experiment, we randomly split a segment of the users into 2 groups (control & variant) for 2 weeks. The variant is exposed to a new app feature which does not exist in control. At the end of the 2 weeks period, we would like to see if there is an uplift in the score distribution of the variant compared to the control. The score is re-calculated every day for every user in our population, so there will be one score per user on the last day. Assuming there's no other experiment/feature flag changes running in the 2 weeks period, shouldn't it be ok to only look at the score once the experiment ends?
f
ya, that's okay as you describe
b
👍 thanks for the quick responses
75 Views
Open in Slack
PreviousNext
Powered by

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.

Powered by