https://www.growthbook.io/ logo
Join Slack
Powered by
Hello! I've found that quite a few customers in an...
# ask-questions
p
Hello! I've found that quite a few customers in an experiment have been assigned multiple variants (which was helpfully flagged on the UI thankyou ), but in digging into it I've identified that customers are getting multiple variants if they are associated with multiple anonymous_ids. The experiment and feature are set to assign based on customer_id, so I'm quite confused as to why this is happening. Screenshots and an example in the thread.
Screenshot 2023-12-15 at 1.17.47 PM.png,Screenshot 2023-12-15 at 1.17.30 PM.png,Screenshot 2023-12-15 at 1.05.30 PM.png
f
where is the anonymous_id coming from?
p
From Rudderstack
It's an external ID that we use to stitch customers outside the product to authenticated users
f
and the customer id?
p
That's an internal ID that we generate (or call) when a user authenticates
f
I wonder if an adblock is causing the rudderstack cookie to not be as sticky as your internal cookies
what percentage of your users had this?
p
About 5% currently, but it is increasing overtime (likely due to more opportunities to be assigned another variant)
f
one thing you can check is the cookie length
I wonder if rudderstacks cookie is more short lived
p
But shouldn't the Growthbook assignment not even know/care about the anonymous_id when assigning, given we've said to assign based on customer_id?
f
the fact that you're getting this multiple assignment error means that you're not doing that
are you sure that youre not assigning by anon id and doing the report by customer id?
p
Given the feature in product says split users by customer_id, I assumed that would mean assigning by customer_id - is there another place we should check?
Screenshot 2023-12-15 at 1.27.55 PM.png
f
on the top of the experiment reports
you should see a bar, that has some info about the experiment data
to the right of "Analysis settings"
p
That also shows logged in visitors (meaning with customer_id)
I wonder if perhaps the Growthbook assignment is being called in a race condition at login, and perhaps the customer_id isn't always available when it's called - would it default to the alternative assignment ID if the customer_id is null?
f
if you call our sdk to get a value of a feature flag that includes an experiment on an attribute which is not defined, it will return the default value, but the trackingCallback is not going to be fired
p
hm strange so shouldn't be assigning based on the other attribute even in absence of customer_id
f
sometimes users can get this with sdk setups like id = some_id || another _id
that can cause this
p
I will confirm the implementation and keep you posted!
So it does not appear that we're ever sending the anonymous_id in place of or as a default or alternative for the customer_id... any other suggestions of where we might look to see why it seems to be using that alternative id?
Is it possible that because we've set both as identifiers in Growthbook (broadly, not for this experiment) it is concatenating them somewhere along the way? and then assigning based on both?
f
it might be, if you added a metric that is marked for anon_id, and the experiment is customer id, it will use the join query
p
We've run into this issue again, and it's becoming an increasingly large concern for us. If we aren't able to consistently show a customer the same variant within an experiment the whole experiment is a bit in jeopardy. None of the metrics used include the anonymous_id, the experiment is set to target customer_id at both the experiment level and in the feature. Is there any further support that you can provide?
r
Hi, hoping to chat this through
We're still unfortunately (on another experiment) having double ups despite it being a consistent customer­_id and the dev I'm working with isn't quite sure how to proceed.
f
Can you share the SDK implementation code?
Do you know sql enough to pull a list of records for the doubly exposed folks?
r
Yes have a list of the exposures, and can get the SDK implementation code one moment!
What data on the multiple exposures would be useful?
This is where we initialise Growthbook
And the specific trackingCallback set up
And for the specific experiment that's currently hitting this issue we do  ​`useFeatureValue(growthbookTopTenCarouselKey, 'control')`​ - how we're loading the feature. (growthbookTopTenCarouselKey is ​`explore-top-ten-carousel'`​)
h
The issue happens because the id that you are using as your Identifier Type in your analysis (this is determined by the Experiment Assignment Table you select for your experiment) is different from the one you're using to hash users. So if your hash attribute is the customer_id, you need to make sure that exact id is being tracked in rudderTrack and is the id your Experiment Assignment Query is pointed to on your Datasource page.
p
We have used customer_id across the metrics, and the analysis settings, as well as in the Feature set up so I don't believe this could be the cause
Screenshot 2024-03-25 at 9.14.02 AM.png
h
s.identity?.id
that you set as an attribute may not always be the same as what 
rudderTrack
is setting as the 
customer_id
. I wonder if there's a way to explicity add this 
s.identity?.id
to your ruddertrack to check if they every are different from the 
customer_id
. This is almost always the reason for multiple exposures.
23 Views
Open in Slack
PreviousNext
Powered by