https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
t

tall-branch-42668

01/10/2024, 6:53 PM
Question for the GrowthBook team: When a page loads on our site and the DOM is ready/fires, GrowthBook gets initialized fairly early in the processes that succeed that event. However, GTag loads asynchronously based on how the browser is loading the various elements we have set to populate per page. GB recommends loading/initializing GB after GTag loads, but we do not have GrowthBook set up in this way currently. The reason for that is our Paywalls and various pop-up modals are also early JS browser loads, and these are where our A/B testing focus currently is as a business. So, when we run an experiment on these JS elements, there’s currently a very slight time delay between the browser serving the base Control to the client, and then the GB trackingCallback determining if a Variant should be served to the client instead, which is unnoticeable to the user since this is currently a fraction of a second. However, if we loaded GrowthBook after the Gtag initializes (which could also be after some of these JS expressions), the Variant replacement time duration could extend and become noticeable to the user depending on how quickly the browser is initializing Gtags and other site elements. Our question is: do we need to wait for Gtag to initialize every time? Is there a different way to implement this so that the experiment-viewed cookie is applied to the user before or independently of Gtag loading? An example for an ongoing experiment: we’re running a paywall test on the site currently where the visual editor is adding an additional button/option to the paywall. We do not specifically have GrowthBook initializing after the Gtag initialization. We noticed in GA that some users were seeing the existing paywall (effectively the Control) but did not have the experiment-viewed cookie assigned in GA and therefore did not get served either variation of the experiment from the perspective of GrowthBook. We believe this is because a race condition is happening here, where most of the time Gtag is loading first and then GrowthBook is assigning the experiment-viewed cookie, serving a variation, and is then tracking results, but some of the time Gtag doesn’t load ahead of GrowthBook and so GrowthBook does not assign the experiment-viewed cookie or serve a variation at all. Please let us know your thoughts on how we can adjust for these excluded users @fresh-football-47124 @helpful-application-7107 cc @refined-addition-9216 @gorgeous-electrician-7828
f

fresh-football-47124

01/10/2024, 9:49 PM
Hi Matt - The reason we suggest waiting is so you can use the Google Client ID.
we have some example code to help you generate your own id, which you can use for assignment instead
the trackingCallback should or could be queued up to send when available
t

tall-branch-42668

01/10/2024, 10:35 PM
Thanks @fresh-football-47124. Relatedly, we're noticing that because of this issue, the trackingCallback wasn't always sending the
experiment-viewed
event to GA since in some instances the
gtag()
wasn't loaded yet, which we're adjusting. However, we can still see in the GB dash that there are more reported users in the experiment population cohorts and higher reported Metrics than we're able to see in GA. Is GB logging this anywhere we can access? And, if so, would we be able to retroactively fill GA for the users GB has accounted for but are missing in GA as a result of the
gtag()
not loading fast enough for these missing users?
f

fresh-football-47124

01/10/2024, 10:37 PM
the data is coming from BigQuery
via GA
t

tall-branch-42668

01/11/2024, 1:51 AM
In a perfect world these numbers would match. But I guess the question is as it relates to a specific example: GrowthBook dash reports 73.9k users were exposed to our Variation for an experiment. As an example, we have a trial_starts Metric attached to this experiment. GB dash reports 163 trial_starts for this Variant. When gathering data to compare from GA, the number of users exposed to this Variation is 69.9k, and trial_starts are 93. If all data is stored in Big Query, then GB and GA should have equal reporting. But, GB is serving the experiment to all users, and this is independent of the trackingCallback missing the
experiment-viewed
event send to GA from the
window.gtag
load, so what causes that large of a discrepancy? Does GrowthBook store experiment data in Google Big Query in more ways than via the
window.gtag
?
f

fresh-football-47124

01/11/2024, 2:35 AM
no, GrowthBook is modular, and the experiment reporting is entirely based off your data (BigQuery)
t

tall-branch-42668

01/11/2024, 2:38 AM
What's the mechanism by which BigQuery collects information like
variation_id
from GrowthBook?
f

fresh-football-47124

01/11/2024, 2:39 AM
from the trackingCallback method
Screenshot 2024-01-10 at 6.39.37 PM.png
t

tall-branch-42668

01/11/2024, 2:43 AM
Have you seen other customer instances of GB reporting higher numbers than a GA query for certain metrics?
f

fresh-football-47124

01/11/2024, 2:43 AM
its usually lower, as we do some deduping and data quality checks
t

tall-branch-42668

01/11/2024, 2:44 AM
That's what I would expect as well, having it report near double a GA query is puzzling
f

fresh-football-47124

01/11/2024, 2:44 AM
we give the raw query, you could run it yourself and debug it if you like
t

tall-branch-42668

01/11/2024, 2:47 AM
We also have our Metrics queries running a date range as
BETWEEN '{{startDateISO}}' AND '{{endDateISO}}'
In an instance where we set up a Phase 2 for an existing experiment, I assume the results of our Metrics in the dash automatically adjust for the date range we have set for the Phase selected?
Example, Phase 1: Dec 1st-14th Phase 2: Dec 15th-today If I select Phase 2 in the results tab and Update Data, the results for the Metrics tracked should be from Dec 15th-today, right? Want to confirm the date range in the Metric query
BETWEEN '{{startDateISO}}' AND '{{endDateISO}}'
would not count results from Dec 1st-today
h

helpful-application-7107

01/11/2024, 3:46 AM
If I select Phase 2 in the results tab and Update Data, the results for the Metrics tracked should be from Dec 15th-today, right?
Yes. I suggest you look at the queries we produce, by clicking the three dot menu on the results tab and clicking View Queries. There you'll see some of our date filtering logic and how our queries pull data from BQ to produce the numbers in the dash. I'm happy to help you understand them if you have questions.
👌 1
t

tall-branch-42668

01/11/2024, 3:50 AM
Perfect, thanks! Very helpful, exactly what I was looking for
4 Views