https://www.growthbook.io/ logo
#announcements
Title
# announcements
b

big-article-38157

08/14/2023, 10:26 PM
Hey guys, bit of an odd one. We’ve got GrowthBook up and running and everything seems to be working well. This experiment calls the hook in React when a component is loaded and I have my tracking callback function set up to push an experiment_impression to GA4. 1% of 2k+ people in the test have multiple exposures and I can’t figure out why. BigQuery shows that the tracking callback pushed a crazy amount of experiment impressions with different variants at the exact same time, it’s so weird. I would only expect 1 experiment impression to be called as the test uses an anonymous ID we save in cookies. Any ideas why this could be happening? It seems odd that this would be something to do with our setup considering 99% of users don’t have this issue
h

helpful-application-7107

08/14/2023, 11:05 PM
Hmmm it seems your feature is being evaluated many times for some ID that is consistent in your warehouse but with different IDs on each evaluation. I don't know why it's being evaluated many times, that seems like something to do with where the feature eval is located in your code. Do the other parts of the code around the feature evaluation fire many times? As for the issue with multiple exposures, this means that the ID that is being passed to GrowthBook for hashing is changing for some unit that you're using for your ID in the experiment analysis. Is there some possibility that the ID that GA4 is logging as part of the tracking callback is different from the one set up in your feature flag as the assignment attribute? You could see what IDs exist in your data warehouse, or that the actual react hook has available to it
I remember something recently with GA4 and having two different IDs but I can't find the thread.
b

big-article-38157

08/14/2023, 11:14 PM
Thanks for the reply! So it is evaluated using a react hook that means it's called multiple times when loading and rehydrating and all that stuff but my understanding is that, once the user is out into the experiment (feature is loaded and variant is assigned), calling the hook doesn't trigger the trackingCallback again. We have a weird setup we're actually using a different ID to generate the variant and then using the pseudo id in GA4 to attribute the users in our GrowthBook queries. We need to do this because the GA pseudo id isn't loaded in time for tests we want to run on the server side so we generate our own id and save it in the user's cookies. The obvious answer I guess is that that part isn't working correctly (somehow the cookies are changing each time). I'm just finding it odd because it's for such a small amount of users
I'm wondering if the cookie setting is somehow failing so the user ends up being assigned a new variant each time the hook is called and it just ends in a mess
Part of me was hoping that maybe that wasn't the issue but now you've said it, it seems like the only valid one. Going to be a pain in the arse to debug since I need to figure out why the cookie setting is failing. Hmmm, was the thread you mention about this sort of issue?
s

silly-fall-70535

08/14/2023, 11:20 PM
Are you using nextjs
b

big-article-38157

08/14/2023, 11:22 PM
Yup
s

silly-fall-70535

08/14/2023, 11:28 PM
I have some random theories but I don't know growth book to well (having my own issues with getting things to work), but was going to say maybe some of your users are logging on to different browsers and the user ids aren't getting connected properly so they get exposed to different variations. I've seen that before for another testing framework
b

big-article-38157

08/14/2023, 11:30 PM
It's a good shout but this is happening within the same millisecond so is definitely not a user related action
h

helpful-application-7107

08/14/2023, 11:30 PM
once the user is out into the experiment (feature is loaded and variant is assigned), calling the hook doesn't trigger the trackingCallback again.
It shouldn't if the GrowthBook instance persists for that user. This could indicate that for some users they keep creating new growthbook instances, wherever that is in your app.
Hmmm, was the thread you mention about this sort of issue?
I found it, the problem stemmed from using GA and GA4 side-by-side, as they use slightly different client_id values, so seems unrelated to your issue.
I would try and figure out if you can log that hashing attribute along with your tracking callback to see what value is actually being used and why it's changing across runs. If the value is changing, then obviously we will fire the callback many times because we think it's a new user.
We have a weird setup we're actually using a different ID to generate the variant and then using the pseudo id in GA4 to attribute the users in our GrowthBook queries.
This is pretty much always the source of multiple exposure problems.
b

big-article-38157

08/14/2023, 11:36 PM
Yep, doesn't sound like it's a problem with GrowthBook which is great. What I'm loving about GrowthBook is a lot of the time things are so transparent and clear. Especially after coming from the murky world of Google Optimize. Thanks for the help! I'll do some digging
Do you have any thoughts on ways to get around generating our own ID?
Seems like the best option but would be super open to other ideas
h

helpful-application-7107

08/14/2023, 11:41 PM
Unfortunately I don't. What are the downsides to generating your own ID in your case?
b

big-article-38157

08/14/2023, 11:45 PM
Nothing aside from this issue we're having. Ideally I wouldn't like to save something in cookies on load and use two different "anonymous ids" when assigning a variant and attributing because it just adds noise and chance for weird bugs like the above. If it's the best/only option then so be it. It's definitely not the end of the world
I did try and have a think about something that's unique enough to anonymous users and consistent to work as an id but it sounds like a super unstable idea. When you start thinking about stuff like that it definitely makes generating an id seem the simplest haha
h

helpful-application-7107

08/14/2023, 11:59 PM
I see. So in sum it seems some user hits this hook and it is spawning a bunch of your custom IDs simultaneously but then they all get the same ga4 pseudo id.
b

big-article-38157

08/15/2023, 12:02 AM
Yup, so the problem sounds like for some reason the id we generate isn't saving to the cookies correctly and each time the hook is called, a new id is generated. Then GrowthBook rightly thinks the user saw multiple variants (because they did) in the experiment analysis
My first thought would be that the user has their cookies disabled but then GA and all that wouldn't work at all. I've checked and all users with this issue are on different devices and operating systems
h

helpful-application-7107

08/15/2023, 12:03 AM
But the timestamps are identical, so the hook is being called multiple times right away, no?
b

big-article-38157

08/15/2023, 12:04 AM
Yep. Under normal testing the hook gets called around 3 times. Once for a server side render, once on client while the features load and then once when the features load. At least that's my guess
So that's weird in itself that it happened a crazy amount of times in the same timestamp. Next port of call is to pass the generated id into ga so I can see if it's different. If the variant stays the same, the experiment impression would only be sent once regardless of the re-renders. Seems like the id is the first culprit to figure out
👍 1
h

helpful-application-7107

08/15/2023, 4:47 PM
Here's a snippet @fresh-football-47124 wrote to demonstrate an approach to generating a uuid for GA4: https://gist.github.com/Auz/0b8c79031b09bbee0e9542c4ec250f15 You could compare with your own approach to see if there are any differences
b

big-article-38157

08/15/2023, 5:44 PM
hahahaha, Graham sent it to me last night and there aren't really any differences tbh. Thank u tho!
h

helpful-application-7107

08/15/2023, 6:02 PM
got it
9 Views