Hey guys, bit of an odd one. We’ve got GrowthBook up and running and everything seems to be working well. This experiment calls the hook in React when a component is loaded and I have my tracking callback function set up to push an experiment_impression to GA4. 1% of 2k+ people in the test have multiple exposures and I can’t figure out why. BigQuery shows that the tracking callback pushed a crazy amount of experiment impressions with different variants at the exact same time, it’s so weird. I would only expect 1 experiment impression to be called as the test uses an anonymous ID we save in cookies. Any ideas why this could be happening? It seems odd that this would be something to do with our setup considering 99% of users don’t have this issue

Hmmm it seems your feature is being evaluated many times for some ID that is consistent in your warehouse but with different IDs on each evaluation. I don't know why it's being evaluated many times, that seems like something to do with where the feature eval is located in your code. Do the other parts of the code around the feature evaluation fire many times? As for the issue with multiple exposures, this means that the ID that is being passed to GrowthBook for hashing is changing for some unit that you're using for your ID in the experiment analysis. Is there some possibility that the ID that GA4 is logging as part of the tracking callback is different from the one set up in your feature flag as the assignment attribute? You could see what IDs exist in your data warehouse, or that the actual react hook has available to it

Thanks for the reply! So it is evaluated using a react hook that means it's called multiple times when loading and rehydrating and all that stuff but my understanding is that, once the user is out into the experiment (feature is loaded and variant is assigned), calling the hook doesn't trigger the trackingCallback again. We have a weird setup we're actually using a different ID to generate the variant and then using the pseudo id in GA4 to attribute the users in our GrowthBook queries. We need to do this because the GA pseudo id isn't loaded in time for tests we want to run on the server side so we generate our own id and save it in the user's cookies. The obvious answer I guess is that that part isn't working correctly (somehow the cookies are changing each time). I'm just finding it odd because it's for such a small amount of users

Are you using nextjs

Yup

I have some random theories but I don't know growth book to well (having my own issues with getting things to work), but was going to say maybe some of your users are logging on to different browsers and the user ids aren't getting connected properly so they get exposed to different variations. I've seen that before for another testing framework

It's a good shout but this is happening within the same millisecond so is definitely not a user related action

once the user is out into the experiment (feature is loaded and variant is assigned), calling the hook doesn't trigger the trackingCallback again.

It shouldn't if the GrowthBook instance persists for that user. This could indicate that for some users they keep creating new growthbook instances, wherever that is in your app.

Hmmm, was the thread you mention about this sort of issue?

I found it, the problem stemmed from using GA and GA4 side-by-side, as they use slightly different client_id values, so seems unrelated to your issue.

Yep, doesn't sound like it's a problem with GrowthBook which is great. What I'm loving about GrowthBook is a lot of the time things are so transparent and clear. Especially after coming from the murky world of Google Optimize. Thanks for the help! I'll do some digging

Unfortunately I don't. What are the downsides to generating your own ID in your case?

Nothing aside from this issue we're having. Ideally I wouldn't like to save something in cookies on load and use two different "anonymous ids" when assigning a variant and attributing because it just adds noise and chance for weird bugs like the above. If it's the best/only option then so be it. It's definitely not the end of the world

I see. So in sum it seems some user hits this hook and it is spawning a bunch of your custom IDs simultaneously but then they all get the same ga4 pseudo id.

Yup, so the problem sounds like for some reason the id we generate isn't saving to the cookies correctly and each time the hook is called, a new id is generated. Then GrowthBook rightly thinks the user saw multiple variants (because they did) in the experiment analysis

But the timestamps are identical, so the hook is being called multiple times right away, no?

Yep. Under normal testing the hook gets called around 3 times. Once for a server side render, once on client while the features load and then once when the features load. At least that's my guess

Here's a snippet @fresh-football-47124 wrote to demonstrate an approach to generating a uuid for GA4: https://gist.github.com/Auz/0b8c79031b09bbee0e9542c4ec250f15 You could compare with your own approach to see if there are any differences

hahahaha, Graham sent it to me last night and there aren't really any differences tbh. Thank u tho!

got it