Hey! Loving that we now can create ratio metrics where the analysis unit <> randomization unit. We have a lot of use cases for this, so thanks! 🙌 🙌 I have created a couple of them, and they seem to calculate the rate ok, but the violin plot and Chance to beat control calculation is off (see picture). Is this the case for others too, or is it just with us? Both the nominator and denominator work well by itself.

Can you share the raw query results (users, count, mean, and stddev for each variation)? I can see if I can reproduce it.

Sure! See below. There are 3 groups, and it is the same problem for both “treatments”. The problematic one is a count metric divided by another count metric. (I also tried a revenue metric divided by a count metric, and that works fine).

Ahh, the standard deviation is zero. That would explain why the error bars are missing. Must be a bug in the query.

Ah, yes. I should have caught that 😅 Yes, I’ll have a look

Thanks. That helps a lot and should be enough to reproduce. And is this on Redshift or another db engine?

Snowflake ❄️

I'll have to dive into this further. We're using the Delta method for variance correction of ratio metrics, following this paper - https://arxiv.org/pdf/1803.06336.pdf Passing your inputs into the formulas in the paper yields a negative variance, which is very strange.

Hi Hild, I was investigating this today and wondered if it's possible you have any NULL values in the metric source for the ratio metric numerator or denominator?

Hello! Thanks for looking into this! The metric is a share of a type of product sold out of a total, so if there is a number in the numerator, there should also be one in the denominator. But there could be a null in the numerator. We found another way of doing this and our other ratio metrics seem to work fine, so don’t know what happened here. No need to dive to deep from our side anymore though 😊