It does seem odd to me that these warnings would pop up now, after the Experiment has been running for many months. I'm wondering if the experiment duration has simply run long enough to "collect" lots of minor issues over time, which are now being surfaced. I'm going to ask our Data Scientist to chime in here during business hours on Friday. For the Sample Ratio Mismatch (SRM), it's possible that 1.3% is a statistically probable difference and nothing to worry about. Our Data Scientist can help you with this. As for the Multiple Exposures Warning, this generally is an issue with the implementation. Could you send me a screenshot or code snippet of each of the following? 1. The GrowthBook-related code, including the trackingCallback. 2. The Experiment Assignment Query, found in Left Navbar --> Metrics and Data --> Data Sources --> Click on the relevant one --> Experiment Assignment Query

Here is the experiment assignment query, removed our table names:

SELECT
  user_pseudo_id as anonymous_id,
  TIMESTAMP_MICROS(event_timestamp) as timestamp,
  experiment_id_param.value.string_value AS experiment_id,
  variation_id_param.value.int_value AS variation_id,
  geo.country as country,
  traffic_source.source as source,
  traffic_source.medium as medium,
  device.category as device,
  device.web_info.browser as browser,
  device.operating_system as os
FROM
  `xxxxxxx`.`xxxxxxx`.`events_*`,
  UNNEST(event_params) AS experiment_id_param,
  UNNEST(event_params) AS variation_id_param
WHERE
  (_TABLE_SUFFIX BETWEEN '{{date startDateISO "yyyyMMdd"}}' AND '{{date endDateISO "yyyyMMdd"}}' or 
  _TABLE_SUFFIX BETWEEN 'intraday_{{date startDateISO "yyyyMMdd"}}' AND 'intraday_{{date endDateISO "yyyyMMdd"}}')
  AND event_name = 'experiment_viewed'  
  AND experiment_id_param.key = 'experiment_id'
  AND variation_id_param.key = 'variation_id'
  AND user_pseudo_id is not null

Here is our `initGrowthBook`method that has the callback, I took out identifers. The server side implementation is for our cart, we do not use 3rd party JS on our checkout, so the callback sends to GA4 on the server side via an ajax call.

import { GrowthBook } from "@growthbook/growthbook";

export async function initGrowthBook(serverside = false) {
    let gbClientKey = "";
    let gbEnableDev = true;
    if (PRODUCTION) {
        gbClientKey = "";
        gbEnableDev = false;
    }

    // Custom generated ID for user
    let gbuuid = getUUID();

    const growthbook = new GrowthBook({
        apiHost: "",
        clientKey: gbClientKey,
        enableDevMode: gbEnableDev,
        // Targeting attributes
        attributes: {
            id: gbuuid,
        },
        trackingCallback: (experiment, result) => {
            if (serverside) {
                <http://DOI.Ajax.post|DOI.Ajax.post>(
                    '/checkout/experiment.json',
                    {
                        data: {
                            experiment_id: experiment.key || "",
                            variation_id: result.key || "",
                            gb_user_id: gbuuid || "",
                        },
                        dataType: 'json',
                        success: () => {
                        },
                        error: (json) => {
                            if (!PRODUCTION) {
                                console.log("GrowthBook error");
                                console.log(json);
                            }
                        },
                    }
                );
            } else {
                window.jxEventBus.push({
                    event: "experiment",
                    experiment_id: experiment.key,
                    variation_id: result.key,
                    gb_user_id: gbuuid,
                });
            }

            if (!PRODUCTION) {
                console.log("Experiment Viewed", {
                    experimentId: experiment.key,
                    variationId: result.key,
                    variationValue: result.value,
                });
            }
        },
    });
    await growthbook.loadFeatures({ autoRefresh: true });

    return growthbook;
}
// Attach to the window object
if ("undefined" !== typeof window) {
    window.initGrowthBook = initGrowthBook;
}

// This is taken from GrowthBook
// <https://docs.growthbook.io/guide/GA4-google-analytics#generating-your-own-id>
const getUUID = () => {
    const COOKIE_NAME = "gbuuid";
    const COOKIE_DAYS = 400; // 400 days is the max cookie duration for chrome

    // use the browsers crypto.randomUUID if set
    const genUUID = () => {
        if(window?.crypto?.randomUUID) return window.crypto.randomUUID();
        return ([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g, c =>
        (c ^ crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4).toString(16)
        );
    }
    const getCookie = (name) => {
        let value = `; ${document.cookie}`;
        let parts = value.split(`; ${name}=`);
        if (parts.length === 2) return parts.pop().split(';').shift();
    }
    const setCookie = (name, value) => {
        var d = new Date();
        d.setTime(d.getTime() + 24*60*60*1000*COOKIE_DAYS);
        document.cookie = name + "=" + value + ";path=/;expires=" + d.toGMTString();
    }

    // get the existing UUID from cookie if set, otherwise create one and store it in the cookie
    if(getCookie(COOKIE_NAME)) return getCookie(COOKIE_NAME);

    const uuid = genUUID();
    setCookie(COOKIE_NAME, uuid);
    return uuid;
}

I'll give you a recent experiment here:

Screenshot 2024-01-12 at 10.10.01 AM.png

This is "show/hide" a/b test. The content is either hidden or displayed.

relevant JS:

document.addEventListener("DOMContentLoaded", async () => {
    if ("function" !== typeof window.initGrowthBook) {
        return;
    }

    const growthbook = await window.initGrowthBook();
    const featureName = "items-to-consider";
    const feature = growthbook.isOn(featureName);
    
    if (feature) {
        const items = document.getElementById("jx-abt-items-to-consider");
        items.style.display = "block";
    }
});

My colleague on the Data Science team should be coming online in the next hour or two, and I've asked him to take a look

Appreciate it!

This one only has multiple exposures.

I'll get you the other thing, one sec

Can you show me the split on that one with multiple exposures only?

My guess is it will be similar, but probably just too small N to have stat sig SRM test

ok, one sec

Screenshot 2024-01-12 at 12.22.27 PM.png

I posted the assignment query above before you joined, I can share that again if you need me to.

Yeah as you can see that split is again 48.5/51.5

Our development has it's own GA4 and is not attached as a data source for Growthbook. So no date is shared, it's just for me to test to make sure everything fires correctly.

It's not about where the data goes; it's about whether users exposed in phase 1 cause carryover bias in phase 2 by being more likely to return if they were in the variation. But that's not a problem given your set up where phase 1 is just devs testing.

Example, had a bug and we used a new phase when the bug was fixed.

Based on what you just said, that could cause an issue?

using this: https://docs.growthbook.io/guide/GA4-google-analytics#generating-your-own-id

ok, so some of this was definitely not understanding everything from the get go.

Normally using

user_pseudo_id

is good enough because you always have it for everything from GA4, and it normally lines up well enough with a custom

gbuuid

. Many people get some multiple exposures (small percentage) when they mismatch, but it normally doesn't trigger SRM.

I had to add in the intraday tables to get data back from that query

Gotcha. This type of data analysis is extremely new to me and some is way over my head.

So for some reason that cookie isn't persisting for that user_pseudo_id.

So gotta figure out the root cause to why the cookie doesn't persist.

Yeah, the root issue is that gbuserid isn't persisting within a user_pseudo_id