This message was deleted.

@steep-grass-70833 has opened an issue Close Issue button

@green-kite-41783 @prehistoric-printer-86780 FYI

I can also add information about our settings: "assignment attribute": id (always the same for individual user) "attribute targeting" : {"isSpecificBrowser": true, "transactions": true, "country": {"$ne": "pl"}} default value: Variant 1

Hi Mohamed and Monika, thanks for reaching out! I've asked the team if there could be any other cause of the Multiple Exposures Warning. I'm also looking into why the split would be uneven (42/58). This might not matter, but just to cover it, are you self-hosting or using the cloud-hosted version of GrowthBook? Could you please send some screenshots of the configurations in your dashboard, as well as the GrowthBook-related code you're using?

there are typically a few reasons for SRMs, and the most common is adjusting the split mid experiment.

• We are using cloud version • We did not change the assignment or any settings mid experiment • Here are a few screenshots from the configurations

Transaction amount

metric query

SELECT
  user_id,
  timestamp,
  orderAmount_USD as value
FROM
  `desktop_growthbook.dt_cb_savings_1_0_transactions`

Transaction amount

metric query from experiment dashboard->view queries

-- Transaction amount (30-day window) (revenue)
WITH
  __rawExperiment AS (
    SELECT
      user_id,
      timestamp,
      experiment_id,
      variation_id,
      country
    FROM
      `osp-bu-desktop.desktop_growthbook.dt_cb_savings_1_0_assignments`
  ),
  __experimentExposures AS (
    -- Viewed Experiment
    SELECT
      e.user_id as user_id,
      cast(e.variation_id as string) as variation,
      CAST(e.timestamp as DATETIME) as timestamp
    FROM
      __rawExperiment e
    WHERE
      e.experiment_id = 'dt_cb_savings_1.0'
      AND e.timestamp >= '2023-10-30 12:50:00'
      AND e.timestamp <= '2023-11-07 07:15:04'
  ),
  __experimentUnits AS (
    -- One row per user
    SELECT
      e.user_id AS user_id,
      cast('All' as string) AS dimension,
      MIN(e.timestamp) AS first_exposure_timestamp,
      (
        CASE
          WHEN count(distinct e.variation) > 1 THEN '__multiple__'
          ELSE max(e.variation)
        END
      ) AS variation
    FROM
      __experimentExposures e
    GROUP BY
      e.user_id
  ),
  __distinctUsers AS (
    SELECT
      user_id,
      dimension,
      variation,
      first_exposure_timestamp AS timestamp,
      date_trunc(first_exposure_timestamp, DAY) AS first_exposure_date
    FROM
      __experimentUnits
  ),
  __metric as ( -- Metric (Transaction amount (30-day window))
    SELECT
      user_id as user_id,
      m.value as value,
      CAST(m.timestamp as DATETIME) as timestamp
    FROM
      (
        SELECT
          user_id,
          timestamp,
          orderAmount_USD as value
        FROM
          `desktop_growthbook.dt_cb_savings_1_0_transactions`
      ) m
    WHERE
      m.timestamp >= '2023-10-30 12:50:00'
      AND m.timestamp <= '2023-12-07 07:15:04'
  ),
  __userMetricJoin as (
    SELECT
      d.variation AS variation,
      d.dimension AS dimension,
      d.user_id AS user_id,
      (
        CASE
          WHEN m.timestamp >= d.timestamp
          AND m.timestamp <= DATETIME_ADD(d.timestamp, INTERVAL 720 HOUR) THEN m.value
          ELSE NULL
        END
      ) as value
    FROM
      __distinctUsers d
      LEFT JOIN __metric m ON (m.user_id = d.user_id)
  ),
  __userMetricAgg as (
    -- Add in the aggregate metric value for each user
    SELECT
      variation,
      dimension,
      user_id,
      SUM(COALESCE(value, 0)) as value
    FROM
      __userMetricJoin
    GROUP BY
      variation,
      dimension,
      user_id
  )
  -- One row per variation/dimension with aggregations
SELECT
  m.variation AS variation,
  m.dimension AS dimension,
  COUNT(*) AS users,
  'mean' as statistic_type,
  'count' as main_metric_type,
  SUM(COALESCE(m.value, 0)) AS main_sum,
  SUM(POWER(COALESCE(m.value, 0), 2)) AS main_sum_squares
FROM
  __userMetricAgg m
GROUP BY
  m.variation,
  m.dimension

Code:

const growthbook = new GrowthBook({
  apiHost: '<https://cdn.growthbook.io>',
  clientKey: 'sdk-fGO40cEeEBCXoNxF',
  enableDevMode: true,
  subscribeToChanges: true,
  trackingCallback: (experiment, result) => {
    if (experiment.key === EXPERIMENT_KEY.SAVINGS_GOALS_V1 && result.key) {
      StatsService.sendStatABTestEvent(
        result.key,
        LocalStorageService.getItemForUser('huuid'),
        experiment.key,
      );
    }
  },
});

export default function ABTestWrapper({ attributes, children }) {
  const setUserAttributes = async (attributes) => {
    const hashedUserId = await hash(LocalStorageService.getUserId());
    LocalStorageService.setItemForUser('huuid', hashedUserId);
    await growthbook.setAttributes({
      id: hashedUserId,
      ...attributes,
    });
  };

  useEffect(() => {
    growthbook.loadFeatures();
  }, []);

  useEffect(() => {
    if (attributes) {
      setUserAttributes(attributes);
    }
  }, [attributes]);

  return <GrowthBookProvider growthbook={growthbook}>{children}</GrowthBookProvider>;
}

const featureVariant = useFeatureValue(EXPERIMENT_KEY.SAVINGS_GOALS_V1);
{featureVariant === 'control' && (

@brief-honey-45610 @fresh-football-47124 any thoughts here?

Multiple exposures means that in the Experiment Assignment Query, for whatever Identifier Type you are using, we are finding multiple variation IDs for one Identifier ID, e.g:

user_id | experiment_id | variation_id
1234    | abc           | 1
1234    | abc           | 0

This normally happens when there is a problem in the code. Some other troubleshooting questions: • Why is there a variation 2 that has no traffic? • How are you tracking and setting IDs? Lastly, the 52/48 split is just random chance.

• Why is there a variation 2 that has no traffic?

This is intentional we are not sending any traffic to it.

• How are you tracking and setting IDs?

We use a hashed UUID as a user identifier for the experiment, it looks something like this:

d6092e3a09794b66caa6ac60ae2e9ead48d50a13b07d715c97277eebbfbb498c

With so many multiple exposures, it's likely that there's a difference in the ID being used for randomization to the one you're using in your

trackingCallback

. Is it possible that for some reason the growthbook attributes aren't being updated and therefore don't match the huuid in local storage?

is

const hashedUserId = await hash(LocalStorageService.getUserId());

always returning the same value for the same user? is there a possibility of a race condition?

@fresh-football-47124 thank you for your response! This is probably our case.