https://www.growthbook.io/ logo
#announcements
Title
# announcements
m

melodic-solstice-24408

07/26/2023, 8:42 AM
Hi team, I have a question on setting up a duration metric for experiment analysis. One of the options allows me to choose how to aggregate in case the same user has more than one measure of the metric (as per screenshot). However, the only enforced aggregation across users is
AVG
, which makes sense in general but for my specific case could lead to some noisy results given some annoying outliers in the metric distribution. Therefore, I would like to use
MEDIAN
to aggregate the metric across users, and as there’s no option to do so when creating the metric I was wondering if there’s a workaround that people use or if it will be possible in the future to also select the aggregation method across all users.
w

white-fireman-22476

07/26/2023, 8:47 AM
Hi Fabrizio. It sounds like you might be interested in winsorization which Growthbook does have the ability to do: https://docs.growthbook.io/app/metrics#capped-value
m

melodic-solstice-24408

07/26/2023, 8:54 AM
Thanks James! Is the
percentile capping
available from a specific version on?
w

white-fireman-22476

07/26/2023, 8:55 AM
Are you self hosting?
m

melodic-solstice-24408

07/26/2023, 8:56 AM
Yes we are
w

white-fireman-22476

07/26/2023, 8:57 AM
Ah yes it looks like it was added three weeks ago. So it might not have made it in the last release, but I do believe we are having a new release today.
🙌 1
👀 1
That said if you can't wait until later today, it is on the main branch already.
m

melodic-solstice-24408

07/26/2023, 9:03 AM
No it’s fine. I’ll ask the managers of our instance to update the release when it comes out - not urgent. Thank you so much for the help! 🙂
👍 1
h

helpful-application-7107

07/26/2023, 4:28 PM
However, the only enforced aggregation across users is
AVG
, which makes sense in general but for my specific case could lead to some noisy results given some annoying outliers in the metric distribution.
You're correct that the enforced aggregation across users is
AVG
however, the field you showing is the user-level aggregation. Normally you actually want to
SUM(value)
as per the default so that your user-level total is the sum of all of their values, and then in analysis we divide by the N users which gives you the average per user (so the
AVG
happens internally in our stats engine). If you choose
AVG(value)
as your custom aggregation, you're doing something very different, where each user will get the average of the rows that are accorded to them. Instead of looking at "duration per user" you're looking at "average duration per user" and comparing those across variations.
Therefore, I would like to use
MEDIAN
to aggregate the metric across users
This is a very different statistical test than averaging across users, which as I described above, is what we do above. This is not supported by growthbook, and the test you're describing is often called "quantile testing"
Hi Fabrizio. It sounds like you might be interested in winsorization which Growthbook does have the ability to do: https://docs.growthbook.io/app/metrics#capped-value
James is correct that to deal with your outliers you probably want to work with percentile winsorization. However, if you care about the "duration per user" and not "average duration per user" I suggest you user percentile winsorization and the
SUM(value)
default aggregation.
I should note that percentile winsorization will apply after the aggregation at the user level and before the averaging across users. So if you pick 95% capping, it will take the top 5% of
SUM(value)
per user and trim them to be the same as the 95th percentile.
w

white-fireman-22476

07/26/2023, 7:52 PM
The new release with percentile winsorization is live: https://github.com/growthbook/growthbook/releases/tag/v2.3.0
14 Views