@helpful-application-7107 values can't be null because of the condition in query, it can't even a zero number.

I would love to see some screen shots of your results and the intermediate results that are proving problematic.

For example, the screen shots that the user provided here would be helpful for your case and could help us debug the issue: https://growthbookusers.slack.com/archives/C01T6Q0TEG3/p1666343634581349

If I retrieve the query that the experiment is using (under view queries option), I can do the following:

...all previous steps... ,
__stats as (
    -- One row per variation/dimension with aggregations
    SELECT
      d.variation,
      d.dimension,
      SUM(IFF(m.value is null, 1, 0)) as num_nulls_numerator,
      SUM(IFF(d.value is null, 1, 0)) as num_nulls_denominator
    FROM
      __userDenominator d
      LEFT JOIN __userMetric m ON (d.user_id = m.user_id)
    GROUP BY
      d.variation,
      d.dimension
  )
  
  select *
  FROM __stats

To check if this final join gets any null (i.e. some users might be in the denominator and not in the numerator, therefore retrieving a null in

m.value

). That’s not the case, as this query retrieves 0 in both cases.

The funny thing is that we have another ratio metric, which uses the same denominator (i.e. the same count metric as denominator), and for this one the violin plot and the stats works well. I’ve checked and the final metrics calculated by the query: • variation • dimension • users • count • mean • stddev Are more or less in the same proportion in both cases. stddev is in both cases roughly 3 times the mean. The only difference I’m able to see might be that the not working metric might has a mean closer to 0 (around 0.07), but I’m not sure that would be the root of the issue.

Apart from that, I just wonder how this new ratio metrics works with the bayesian statistical framework. Are we still defining a gaussian posterior with

mean=query_mean

and std =

query_stddev / sqrt(num_users)

? Thanks a lot and let us know if we can help with anything else!

Just copy the cell below the ratio metric query along with the ratio metric query itself, and I'll be able to take a closer look.

@green-jordan-83609 shared some more details in a private DM. The issue in this case is that we use log approximations in our gaussian tests in

gbstats

and the values in your case are small (mean) and uncertain (variance), and the log approximation can suffer when the mean values are close to zero. We have a check built in

gbstats

here that returns no meaningful information for the posterior distribution when this check fails, since the log approximation could be inexact.

If the stddev of the ratio metric will decrease as you collect more data, this could go away. If not, then we could help you do a bespoke analysis in a jupyter notebook. In any case, we definitely need better error messaging and handling when cases like this occur. I'll open an issue for that.