https://www.growthbook.io/
Join Slack
Hi team, I have a question on setting up a duration metric for experiment analysis. One of the optio...
# announcements
m
Hi team, I have a question on setting up a duration metric for experiment analysis. One of the options allows me to choose how to aggregate in case the same user has more than one measure of the metric (as per screenshot). However, the only enforced aggregation across users is 
AVG
, which makes sense in general but for my specific case could lead to some noisy results given some annoying outliers in the metric distribution. Therefore, I would like to use 
MEDIAN
to aggregate the metric across users, and as there’s no option to do so when creating the metric I was wondering if there’s a workaround that people use or if it will be possible in the future to also select the aggregation method across all users.
w
Hi Fabrizio. It sounds like you might be interested in winsorization which Growthbook does have the ability to do: https://docs.growthbook.io/app/metrics#capped-value
m
Thanks James! Is the 
percentile capping
available from a specific version on?
w
Are you self hosting?
m
Yes we are
w
Ah yes it looks like it was added three weeks ago. So it might not have made it in the last release, but I do believe we are having a new release today.
🙌 1
👀 1
That said if you can't wait until later today, it is on the main branch already.
m
No it’s fine. I’ll ask the managers of our instance to update the release when it comes out - not urgent. Thank you so much for the help! 🙂
👍 1
h
However, the only enforced aggregation across users is 
AVG
, which makes sense in general but for my specific case could lead to some noisy results given some annoying outliers in the metric distribution.
You're correct that the enforced aggregation across users is 
AVG
however, the field you showing is the user-level aggregation. Normally you actually want to 
SUM(value)
as per the default so that your user-level total is the sum of all of their values, and then in analysis we divide by the N users which gives you the average per user (so the 
AVG
happens internally in our stats engine). If you choose 
AVG(value)
as your custom aggregation, you're doing something very different, where each user will get the average of the rows that are accorded to them. Instead of looking at "duration per user" you're looking at "average duration per user" and comparing those across variations.
Therefore, I would like to use 
MEDIAN
to aggregate the metric across users
This is a very different statistical test than averaging across users, which as I described above, is what we do above. This is not supported by growthbook, and the test you're describing is often called "quantile testing"
Hi Fabrizio. It sounds like you might be interested in winsorization which Growthbook does have the ability to do: https://docs.growthbook.io/app/metrics#capped-value
James is correct that to deal with your outliers you probably want to work with percentile winsorization. However, if you care about the "duration per user" and not "average duration per user" I suggest you user percentile winsorization and the 
SUM(value)
default aggregation.
I should note that percentile winsorization will apply after the aggregation at the user level and before the averaging across users. So if you pick 95% capping, it will take the top 5% of 
SUM(value)
per user and trim them to be the same as the 95th percentile.
w
The new release with percentile winsorization is live: https://github.com/growthbook/growthbook/releases/tag/v2.3.0
92 Views
Open in Slack
PreviousNext
Powered by

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.

Powered by