This message was deleted.
# announcementsw
worried-planet-2191
08/17/2022, 12:56 AMThis message was deleted.
f
future-teacher-7046
08/17/2022, 1:34 AM
The reason we do ratios like that is because variations may have different sample sizes. If you are running a 90/10 split for your experiment, the raw number of visitors won't be that useful on its own since the variation with 90% of traffic will always have a much higher number of returning visitors.
For all ratio metrics, we do show the raw numbers underneath if that helps.
r
red-fall-3528
08/17/2022, 8:44 AMthat helps a lot. thank you so much!
red-fall-3528
08/17/2022, 9:03 AMI have few questions to ask for your suggestion. They are related to metrics.
We plan to use GrowthBook for 2 types of experiments
• domain expertises -led: experiments suggested by products owner from their observations. We’’ll use specific metrics, for example conversion ratio, or CRT to measure the impact.
• business-led: experiments originate from business team. These experiments may come from business team’s intuition. The only focus is number of
retuning visitorsreturning visitorsretention ratesred-fall-3528
08/17/2022, 9:09 AMI would like to ask your suggestion on the business-led experiments. Have you run the experiments with the similar setup? Could you share any pitfalls with these experiments.
f
future-teacher-7046
08/17/2022, 11:57 AM
I would be a little careful with using click through rate as a primary metric. It's fairly easy to increase the click through rate while actually hurting the UX and long-term metrics like revenue (e.g. giant buttons, bright colors, "free kittens" CTA, etc.)
Usually companies will have a set of core business metrics that they use for every experiment, such as revenue, retention, engagement. Then, depending on the experiment you might add some secondary metrics like CTR that are more for debugging and exploration.
r
red-fall-3528
08/18/2022, 6:14 AMthat makes sense. We’re using DAU as a long-term metrics and ideally we want to add it into metrics on GrowthBook as well. I am not sure how can I add that into GrowthBook because all metrics have
All Experiment Usersf
future-teacher-7046
08/18/2022, 12:16 PM
Yeah, for DAU, we typically see companies make metrics like "day 2 retention" or "day 7 retention". Basically measuring what percent of experiment users come back after a certain amount of time.
To do that you would make a binomial metric, select an "app open" event (or similar) in your SQL, set a conversion delay (24 hours for day 2, 168 hours for day 7, etc), and finally set a conversion window time of 24 hours.
r
red-fall-3528
08/19/2022, 1:07 AMthanks so much. I think retention rates and the use of conversion delay.
and with conversion delay, that means I can use the same query and just change to conversion delay to get retention rates from different time frame, does it?
red-fall-3528
08/19/2022, 10:42 AMI have 2 questions:
• when I query metrics with different conversion delay, I still get the same values in the metrics preview. Is that normal? (the metrics showed in experiment are different, so I think it’s normal but just want to confirm)
• In our iOS implementation,
trackingCallbacktrackingCallbackred-fall-3528
08/19/2022, 10:44 AMcc: @bland-tiger-91293
b
bland-tiger-91293
08/19/2022, 10:54 AMI did following this: https://github.com/growthbook/growthbook-kotlin
f
future-teacher-7046
08/19/2022, 12:28 PM
Yes, conversion windows only have an effect inside an experiment, not on the metric preview page.
Can you share the list of feature rules that you expected to trigger the tracking callback? The experiment might not be set up properly there.
r
red-fall-3528
08/19/2022, 2:11 PMsure. we’re using these config rules: https://cdn.growthbook.io/api/features/key_stag_9cf44e2478194106
b
bland-tiger-91293
08/24/2022, 1:16 AMtrackingCallback is working fine now, Thanks Jeremy
238 Views