What does the "Converted users only" advanced sett...
a
What does the "Converted users only" advanced setting for count metrics do? It doesn't seem to exclude users who don't have the metric from the denominator in experiment analysis, like I expected
r
The "Converted users only" setting for count metrics changes how the metric is calculated—it only include users who have converted at least once. When you refresh the data with this option, you don't see any change? Are you able to share a screenshot of the config page?
a
A few things to note here: • Converted users only: Yes • We have a metric that tracks a user's ability in something that we're trying to improve. For any user, that metric can go up or down over time. The user's ability is measured as a score between 0 and 1. When the user's ability changes, we log it as an event in mixpanel, with a property which is the delta between the user's previous score and new score. So, a user might start at a score of 0.5, then given the deltas [-0.1, +0.2, +0.15] end at a score of 0.75. What we care about is that, during an experiment, the user's score thus went up a sum total of 0.25. • Since a user's ability can also go down, their sum total during an experiment could also be negative, e.g. -0.17. • Because growthbook doesn't seem to handle negative values well, the user value aggregation is sum(values) + 1, to give a number between 0 and 2 (instead of -1 and +1). • As you can see in the chart, the user average for this metric is 1 (0 before the aggregation function). • Most users in an experiment don't have any events logged for this metric.
Looking at the results for an experiment (after clicking Update), it appears that the denominator is the the total count for the group (21.5K and 21.7K are the same denominators that all the other metrics in this experiment have). Based on our knowledge of the shape of the data—the user average delta is 1—I would expect the denominators to be closer to the numerators 611 and 629)
r
Thanks for sharing all that! I've pinged our data science team and will let you know what they say.
a
Thank you!
r
It looks you stumbled on a bug 🐛. It looks like this functionality (converted uses only) isn't set up to work with Mixpanel, but we didn't hide the button in the UI. Part of the issue here is that support for Mixpanel has been deprecated—not by us directly but by Mixpanel themselves. They recommend using a separate data warehouse and have some guides on transitioning. Lmk if you have any additional questions here. Happy to help.
a
Ah, that's unfortunate. Do you have any workarounds, besides setting the experiment activation metric to require users to convert on that metric?
Also, do you or your team have any advice on tracking potentially negative metrics? When we tried it, GrowthBook showed that there wasn't enough data (since the minimum sample size is positive, and the sum total / average was negative).
r
Talked with our data team—they confirmed the pain points about working with negative values. At the moment, unfortunately, there isn't any real solution, but it's something we'd like to improve.
a
Thanks—is our workaround with adding 1 to the aggregation valid, or will it cause issues we haven't considered?
r
It seems like a good solution. Have seen an anomalies in your results?
a
None besides the issue with the denominator
👍 1
Thank you for your help on this one 🙂