Hey! Loving that we now can create ratio metrics w...

# give-feedbackf

full-island-88199

10/21/2022, 9:13 AMHey! Loving that we now can create ratio metrics where the analysis unit <> randomization unit. We have a lot of use cases for this, so thanks! 🙌 🙌
I have created a couple of them, and they seem to calculate the rate ok, but the violin plot and Chance to beat control calculation is off (see picture). Is this the case for others too, or is it just with us? Both the nominator and denominator work well by itself.

✅ 1

f

future-teacher-7046

10/21/2022, 12:35 PM

Can you share the raw query results (users, count, mean, and stddev for each variation)? I can see if I can reproduce it.

f

full-island-88199

10/21/2022, 12:41 PMSure! See below. There are 3 groups, and it is the same problem for both “treatments”. The problematic one is a count metric divided by another count metric. (I also tried a revenue metric divided by a count metric, and that works fine).

f

future-teacher-7046

10/21/2022, 12:49 PM

Ahh, the standard deviation is zero. That would explain why the error bars are missing. Must be a bug in the query.

future-teacher-7046

10/21/2022, 12:51 PM

Ayre you to run the query without the final CTE? Basically want to see the raw stats before we do any variance correction.

f

full-island-88199

10/21/2022, 12:51 PMAh, yes. I should have caught that 😅 Yes, I’ll have a look

full-island-88199

10/21/2022, 12:55 PMfull-island-88199

10/21/2022, 1:02 PMThe variance is negative, so that’s why it is 0 in the last step. Calculated these too if they are useful

full-island-88199

10/21/2022, 1:11 PMBtw, we’re also using All attributions window if that matters 🙂

f

future-teacher-7046

10/21/2022, 1:55 PM

Thanks. That helps a lot and should be enough to reproduce. And is this on Redshift or another db engine?

🙌 1

f

full-island-88199

10/21/2022, 1:56 PMSnowflake ❄️

f

future-teacher-7046

10/21/2022, 2:32 PM

I'll have to dive into this further. We're using the Delta method for variance correction of ratio metrics, following this paper - https://arxiv.org/pdf/1803.06336.pdf
Passing your inputs into the formulas in the paper yields a negative variance, which is very strange.

🤔 1

h

helpful-application-7107

12/05/2022, 10:56 PMHi Hild, I was investigating this today and wondered if it's possible you have any NULL values in the metric source for the ratio metric numerator or denominator?

f

full-island-88199

12/06/2022, 8:29 AMHello! Thanks for looking into this! The metric is a share of a type of product sold out of a total, so if there is a number in the numerator, there should also be one in the denominator. But there could be a null in the numerator.
We found another way of doing this and our other ratio metrics seem to work fine, so don’t know what happened here. No need to dive to deep from our side anymore though 😊

23 Views