Hello, I wonder if GrowthBook can analysis the 'accuracy' or 'precision' metric type. Say, now I have two variations, and each group, every user has a tn/fp/fp/fn(a term comes from classification task) attribute to indicate if he choose the right answer in some test. But I don't care much about the count of tn/fp/fp/fn, I just care the 'accuracy' or 'precision' which can be calculated from the count of tn/fp/fp/fn use a fixed formula. Does GrowthBook support such function?

Hi Jinglin, I am a little unclear or what it is you are trying to achieve. However Growthbook has a lot of flexibility of how you create your metric, and basically anything that can be calculated in sql as a number can be used. You can also change the aggregation per user from

SUM(value)

to something else if that is needed to get at the metric you want.

Thx James, I might not express what I want clearly, let me try to explain. The core is, the metric I want to get can't be calculated for every single user. That is, for only one user, there's no such formula to calculate the metric for him. The metric can only be calculated for a whole variation. For example, I have tow sets of questions, each user will enter one set to answer only one question. So, there's no such accuracy for a single user, I can only calculate the 'whole accuracy' for a set. Is that clearer?

I'm still not sure if I understand fully, but Growthbook does the right math for Bayesian or Frequentist analysis across all users.

Ok, here I'll show you with a specific example: The situation is, I divide user by their id into two group with different kind of questions to answer(like group A answer math questions and group B answer physic questions). But a user will only be ask one question, which means for one user, I can only get a metric about whether he gives the right answer. But then, what is my top concern is: what the percentage of "right user" in each group. ( for example 2 user give right answer while 3 wrong, the metric should be 40%). So, does GrowthBook support such metric type?

I think what you want here is a binomial metric https://docs.growthbook.io/app/metrics#binomial-metrics. Which says simply yes/no did that user convert - that is get a right answer. However you then might need to make a funnel metric: https://docs.growthbook.io/app/metrics#funnel-metrics to only include in the users that answered the question at all. In the end then you will get a result that shows the likelihood that people in that variation got it right given they answered the question.

Thx!! OK, I'll check it later to see if it works for my situation.

Sure thing. Binomial metrics with funnels are pretty common. Like sites trying to determine if a viewer saw the sign up page, and then later signed up.