https://www.growthbook.io/ logo
Join Slack
Powered by
I am currently running an A/B test using Growthboo...
# ask-questions
g
I am currently running an A/B test using Growthbook to separate users, and I am tracking the experiment metrics through Mixpanel. Since there is no login system, I am using the device ID as the unique key for each user. I set the ratio of the control group to the experiment group as 5:5, but when I checked in Mixpanel, the actual ratio is approximately 9:1. I would like to know the reason for this discrepancy.
f
that's pretty wildly off
how large is the sample size?
g
sample size: control = 5,000 experiment : 500
f
hrmm, yes, that does not seem right
g
its like 10:1 raitio
f
did you change change the split percentage while it was running at all?
is this a URL redirect experiment by any chance?
g
not at all. we didn't make any change since we started the experiment
we don't use url redirection for this experiment.
f
huh
can you share the SDK implementation you're using?
g
@growthbook/growthbook-react": "^1.1.0
it looks like latest version...maybe
f
I mean, not just the version - but the code?
g
I am using Next.js, and since it’s a simple UI test, I am running it on the client side. The first image contains the logic in app.tsx, and the second image shows the logic on the page level where properties are being sent to Mixpanel. Just once each.
f
you should track exposure to the AB test via the trackingCallback()
g
Is the problem with the split percentage because developer is not using trackingCallback()?
As you can see from the attached image, tracking via trackingCallback() is working fine. Also, we confirmed that the value, whether ‘a’ or ‘b,’ is being passed from the Growthbook SDK. However, our issue is that the value is assigned as ‘a’ (the default value) much more frequently than the expected 5:5 ratio for ‘a:b.’ Does this have any connection to trackingCallback()?
f
What is happening is that Mixpanel might be a bit slow to load
so if the use effect fires first, setting the id attribute to null
so useFeatureValue will return the fallback
and then on the last line of the track call you have if it's not B, record A
it will be "fallback" but logged as "type A"
you can either generate your own ID to use as the assignment
or make things wait for the mixpanel ID
(generating your own is what we would recommend - there will be a 1:1 correlation between your id and the mixpanel id, but the assignment can happen instantly, and then tracking whenever that's ready)
g
We are running the Mixpanel instance’s init method before the Growthbook instance’s init method. After checking the library code, I found that the return type of the Mixpanel init method is not a promise. Looking into the internal logic of mixpanel.init(), the loading-related logic is executed synchronously. You can find the Mixpanel init code for the version we are using here. Based on my understanding, there’s no way for the Mixpanel distinct_id value to be null due to async loading delays during initialization. In fact, after testing multiple times in the browser console, I never encountered a case where the value was null. The attached image is related to the console output
f
a good way to test it is to change your ternary operator - check if that 'not typeA' case is actually 'typeB', and log that
or test it by logging the events in the trackingCallback() to mixpanel.
54 Views
Open in Slack
PreviousNext
Powered by