What does the "Converted users only" advanced setting for count metrics do? It doesn't seem to exclude users who don't have the metric from the denominator in experiment analysis, like I expected

The "Converted users only" setting for count metrics changes how the metric is calculated—it only include users who have converted at least once. When you refresh the data with this option, you don't see any change? Are you able to share a screenshot of the config page?

A few things to note here: • Converted users only: Yes • We have a metric that tracks a user's ability in something that we're trying to improve. For any user, that metric can go up or down over time. The user's ability is measured as a score between 0 and 1. When the user's ability changes, we log it as an event in mixpanel, with a property which is the delta between the user's previous score and new score. So, a user might start at a score of 0.5, then given the deltas [-0.1, +0.2, +0.15] end at a score of 0.75. What we care about is that, during an experiment, the user's score thus went up a sum total of 0.25. • Since a user's ability can also go down, their sum total during an experiment could also be negative, e.g. -0.17. • Because growthbook doesn't seem to handle negative values well, the user value aggregation is sum(values) + 1, to give a number between 0 and 2 (instead of -1 and +1). • As you can see in the chart, the user average for this metric is 1 (0 before the aggregation function). • Most users in an experiment don't have any events logged for this metric.

Thanks for sharing all that! I've pinged our data science team and will let you know what they say.

Thank you!

It looks you stumbled on a bug 🐛. It looks like this functionality (converted uses only) isn't set up to work with Mixpanel, but we didn't hide the button in the UI. Part of the issue here is that support for Mixpanel has been deprecated—not by us directly but by Mixpanel themselves. They recommend using a separate data warehouse and have some guides on transitioning. Lmk if you have any additional questions here. Happy to help.

Ah, that's unfortunate. Do you have any workarounds, besides setting the experiment activation metric to require users to convert on that metric?

Talked with our data team—they confirmed the pain points about working with negative values. At the moment, unfortunately, there isn't any real solution, but it's something we'd like to improve.

Thanks—is our workaround with adding 1 to the aggregation valid, or will it cause issues we haven't considered?

It seems like a good solution. Have seen an anomalies in your results?

None besides the issue with the denominator