https://www.growthbook.io/ logo
a

ancient-france-65319

09/14/2023, 7:36 AM
Hello, I wonder if GrowthBook can analysis the 'accuracy' or 'precision' metric type. Say, now I have two variations, and each group, every user has a tn/fp/fp/fn(a term comes from classification task) attribute to indicate if he choose the right answer in some test. But I don't care much about the count of tn/fp/fp/fn, I just care the 'accuracy' or 'precision' which can be calculated from the count of tn/fp/fp/fn use a fixed formula. Does GrowthBook support such function?
w

white-fireman-22476

09/14/2023, 8:00 AM
Hi Jinglin, I am a little unclear or what it is you are trying to achieve. However Growthbook has a lot of flexibility of how you create your metric, and basically anything that can be calculated in sql as a number can be used. You can also change the aggregation per user from
SUM(value)
to something else if that is needed to get at the metric you want.
a

ancient-france-65319

09/14/2023, 8:16 AM
Thx James, I might not express what I want clearly, let me try to explain. The core is, the metric I want to get can't be calculated for every single user. That is, for only one user, there's no such formula to calculate the metric for him. The metric can only be calculated for a whole variation. For example, I have tow sets of questions, each user will enter one set to answer only one question. So, there's no such accuracy for a single user, I can only calculate the 'whole accuracy' for a set. Is that clearer?
w

white-fireman-22476

09/14/2023, 8:26 AM
I'm still not sure if I understand fully, but Growthbook does the right math for Bayesian or Frequentist analysis across all users.
Or maybe you are trying to GROUP BY something like a test_id first to get the user's score on a test? Then get their average test score? Then see if people overall have a better test score with variation A vs B?
a

ancient-france-65319

09/14/2023, 8:52 AM
Ok, here I'll show you with a specific example: The situation is, I divide user by their id into two group with different kind of questions to answer(like group A answer math questions and group B answer physic questions). But a user will only be ask one question, which means for one user, I can only get a metric about whether he gives the right answer. But then, what is my top concern is: what the percentage of "right user" in each group. ( for example 2 user give right answer while 3 wrong, the metric should be 40%). So, does GrowthBook support such metric type?
The true situation is more complex, but the core is same: a metric can not be calculated for a single user.(and in my opinion I think this kind metric is not a typical A/B test)
w

white-fireman-22476

09/14/2023, 9:02 AM
I think what you want here is a binomial metric https://docs.growthbook.io/app/metrics#binomial-metrics. Which says simply yes/no did that user convert - that is get a right answer. However you then might need to make a funnel metric: https://docs.growthbook.io/app/metrics#funnel-metrics to only include in the users that answered the question at all. In the end then you will get a result that shows the likelihood that people in that variation got it right given they answered the question.
a

ancient-france-65319

09/14/2023, 9:04 AM
Thx!! OK, I'll check it later to see if it works for my situation.
w

white-fireman-22476

09/14/2023, 9:08 AM
Sure thing. Binomial metrics with funnels are pretty common. Like sites trying to determine if a viewer saw the sign up page, and then later signed up.