Hi everyone, I have a question about variant rotation: is it normal to see such gap between variants...

adorable-solstice-28735

07/14/2023, 4:59 PM

Hi everyone, I have a question about variant rotation: is it normal to see such gap between variants? I know GA4 will take time to aggregate those events (it hasn't been ~48h yet), but I wonder if you guys see the variants rotating closely

adorable-solstice-28735

07/14/2023, 5:00 PM

my id attribute is a client_id parsed from Google cookie, just FYI

helpful-application-7107

07/14/2023, 5:14 PM

Are these users that have seen each event or just events total? If it is the latter, is it possible that clients in control hit the tracking callback more frequently?

adorable-solstice-28735

07/14/2023, 5:15 PM

those are users that have seen the experiment and one of the variants

adorable-solstice-28735

07/14/2023, 5:15 PM

ideally, the gap should be minimal, but maybe I just wait for more data to come in

helpful-application-7107

07/14/2023, 5:15 PM

so its 56 users (

client_id

), not 56 events total?

adorable-solstice-28735

07/14/2023, 5:15 PM

56 events, sorry

helpful-application-7107

07/14/2023, 5:16 PM

Yeah, so if control users are more likely to hit the tracking call back that could also explain this.

helpful-application-7107

07/14/2023, 5:17 PM

And that would be normal, and would just indicate that control users are, for whatever reason, hitting the tracking callback more. But you can test this by instead seeing unique

client_id

in each condition. You could also see this in the Growthbook experiment view if your datasource is set up.

adorable-solstice-28735

07/14/2023, 5:18 PM

I don't have a datasource set up (yet), sending those events to GA4 for testing. Let me process this, I think I am getting users vs events confused in this case

adorable-solstice-28735

07/14/2023, 5:20 PM

maybe it's better to set up the GA4 audiences instead of this event table

helpful-application-7107

07/14/2023, 5:21 PM

Each time user hits a feature evaluation, a tracking callback gets fired. So it's possible that a user can fire multiple events for the same experiment. That's just a key thing to start with.

adorable-solstice-28735

07/14/2023, 5:22 PM

correct, that's that what I see in this table. Let me try to segment it by client_id.. or creating the audiences instead. This table is not a valid presentation as I understand

helpful-application-7107

07/14/2023, 5:23 PM

Yes, this view can be informative, but it isn't necessarily indicative of a problem randomizing between variants.

adorable-solstice-28735

07/14/2023, 5:23 PM

thanks for explaining, it makes sense

adorable-solstice-28735

07/14/2023, 5:28 PM

I don't think I can achieve this with a simple table, I think the only way to correctly display the results in GA4 would be using the audiences. Or/and connecting the sources of course.

67 Views

Open in Slack

Previous Next

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.