Hey everyone, we began using GB two weeks ago, but...
# ask-questions
e
Hey everyone, we began using GB two weeks ago, but we've encountered a strange issue. While the visual changes on the website are working, we've noticed that some critical metrics have disappeared. We're not sure what's causing this error
s
Hi AJ, it's possible the SQL behind the metrics is not correct. Have you tried running the SQL manually on your data source to confirm that they are working (returning non-zero)?
e
we just checked the metrics it looks like it has a return @swift-helmet-3648
s
That's strange. Is your assignment query for your experiment configured correctly to identify users by
user_id
or
anonymous_id
(the identifiers used by metrics)
t
We have configured the identifiers to user_pseudo_id - anonymous_id
So thats fine
h
Can you share the full metrics query that is generated to help us debug? Go to your Experiment, click the three dots on the top right side of the results tab, and click "View Queries".
There are a couple of things: 1. In your metric, it says you have
anonymous_id
and
user_id
as identifier types, but there is only one column in your metric return
anonymous_id
, so something is strange in your setup here 2. You have multiple exposures. This means that for whatever identifier type you are using for the experiment (this is defined by the Experiment Assignment Table you chose for your experiment), there are some
id
values that have multiple variants. This is another sign that something is wrong.
👀 1
t
Below is the full metrics query
Copy code
-- Count of begin_checkout (count)
WITH
  __rawExperiment AS (
    SELECT
      user_pseudo_id as anonymous_id,
      TIMESTAMP_MICROS(event_timestamp) as timestamp,
      experiment_id_param.value.string_value AS experiment_id,
      variation_id_param.value.int_value AS variation_id,
      geo.country as country,
      traffic_source.source as source,
      traffic_source.medium as medium,
      device.category as device,
      device.web_info.browser as browser,
      device.operating_system as os
    FROM
      `cedar-electronics-prismfly`.`analytics_253839784`.`events_*`,
      UNNEST (event_params) AS experiment_id_param,
      UNNEST (event_params) AS variation_id_param
    WHERE
      (
        (_TABLE_SUFFIX BETWEEN '20231012' AND '20231018')
        OR (
          _TABLE_SUFFIX BETWEEN 'intraday_20231012' AND 'intraday_20231018'
        )
      )
      AND event_name = 'experiment_viewed'
      AND experiment_id_param.key = 'experiment_id'
      AND variation_id_param.key = 'variation_id'
      AND user_pseudo_id is not null
  ),
  __experimentExposures AS (
    -- Viewed Experiment
    SELECT
      e.anonymous_id as anonymous_id,
      cast(e.variation_id as string) as variation,
      CAST(e.timestamp as DATETIME) as timestamp
    FROM
      __rawExperiment e
    WHERE
      e.experiment_id = 'delivery-timer-product-page'
      AND e.timestamp >= '2023-10-12 15:33:00'
      AND e.timestamp <= '2023-10-18 14:15:28'
  ),
  __experimentUnits AS (
    -- One row per user
    SELECT
      e.anonymous_id AS anonymous_id,
      cast('All' as string) AS dimension,
      MIN(e.timestamp) AS first_exposure_timestamp,
      (
        CASE
          WHEN count(distinct e.variation) > 1 THEN '__multiple__'
          ELSE max(e.variation)
        END
      ) AS variation
    FROM
      __experimentExposures e
    GROUP BY
      e.anonymous_id
  ),
  __distinctUsers AS (
    SELECT
      anonymous_id,
      dimension,
      variation,
      first_exposure_timestamp AS timestamp,
      date_trunc(first_exposure_timestamp, DAY) AS first_exposure_date
    FROM
      __experimentUnits
  ),
  __metric as ( -- Metric (Count of begin_checkout)
    SELECT
      anonymous_id as anonymous_id,
      m.value as value,
      CAST(m.timestamp as DATETIME) as timestamp
    FROM
      (
        SELECT
          user_pseudo_id as anonymous_id,
          TIMESTAMP_MICROS(event_timestamp) as timestamp,
          1 as value
        FROM
          `cedar-electronics-prismfly.analytics_253839784.events_*`
        WHERE
          (
            (_TABLE_SUFFIX BETWEEN '20231012' AND '20231018')
            OR (
              _TABLE_SUFFIX BETWEEN 'intraday_20231012' AND 'intraday_20231018'
            )
          )
          AND event_name = 'begin_checkout'
      ) m
    WHERE
      m.timestamp >= '2023-10-12 15:33:00'
      AND m.timestamp <= '2023-10-18 14:15:28'
  ),
  __userMetricJoin as (
    SELECT
      d.variation AS variation,
      d.dimension AS dimension,
      d.anonymous_id AS anonymous_id,
      (
        CASE
          WHEN m.timestamp >= d.timestamp THEN m.value
          ELSE NULL
        END
      ) as value
    FROM
      __distinctUsers d
      LEFT JOIN __metric m ON (m.anonymous_id = d.anonymous_id)
  ),
  __userMetricAgg as (
    -- Add in the aggregate metric value for each user
    SELECT
      variation,
      dimension,
      anonymous_id,
      SUM(COALESCE(value, 0)) as value
    FROM
      __userMetricJoin
    GROUP BY
      variation,
      dimension,
      anonymous_id
  )
  -- One row per variation/dimension with aggregations
SELECT
  m.variation AS variation,
  m.dimension AS dimension,
  COUNT(*) AS users,
  'mean' as statistic_type,
  'count' as main_metric_type,
  SUM(COALESCE(m.value, 0)) AS main_sum,
  SUM(POWER(COALESCE(m.value, 0), 2)) AS main_sum_squares
FROM
  __userMetricAgg m
GROUP BY
  m.variation,
  m.dimension
h
So I would guess the primary problem has to do with how you're setting
user_pseudo_id
in the tracking callback. What's happening is that there are multiple different
variation_id
values for the same
user_pseudo_id
.
t
Hey Luke the data is passing to the bigquery. We are using user_pseudo_id as 'identifiers'. Please have a look at the below screenshot.
image (76).png
Metrics are not showing up the value.
The experiments are throwing up the value only for user count and pageview.
@helpful-application-7107 can you throw some lights on this _"So I would guess the primary problem has to do with how you're setting
user_pseudo_id
in the tracking callback."_
h
I'm saying that you have multiple IDs in each variation for some reason.
Share the full query from the screenshot above and I can show you how to see that.
t
Here it is
SELECT variation_id_param.value.int_value AS
variation_id
, count(DISTINCT user_pseudo_id) as
Total_users
, count( DISTINCT CONCAT( user_pseudo_id, "-", ( SELECT value.int_value FROM unnest (event_params) WHERE key="ga_session_id" ) ) ) as
Sessions
, COUNTIF(event_name='add_to_cart') as
Add_to_cart
FROM
cedar-electronics-prismfly.analytics_253839784.events_*
, unnest(event_params) as
experiment_id_param
, unnest(event_params) as
variation_id_param
WHERE event_name='experiment_viewed' AND experiment_id_param.key = 'experiment_id' AND experiment_id_param.value.string_value = 'aa-test-cobra' AND variation_id_param.key = 'variation_id' Group by 1
But i am not able to understand why the metrics are not showing up the values in bigquery as well
h
But i am not able to understand why the metrics are not showing up the values in bigquery as well
Because you are filtering out event names in this query:
Copy code
SELECT
  variation_id_param.value.int_value AS `variation_id`,
  count(DISTINCT user_pseudo_id) as `Total_users`,
  count(
    DISTINCT CONCAT(
      user_pseudo_id,
      "-",
      (
        SELECT
          value.int_value
        FROM
          unnest (event_params)
        WHERE
          key="ga_session_id"
      )
    )
  ) as `Sessions`,
  ------- THIS LINE WILL DO NOTHING BECAUSE YOU ALREADY HAVE event_name='experiment_viewed' in the WHERE CLAUSE ------
  COUNTIF(event_name='add_to_cart') as `Add_to_cart`
FROM
  `cedar-electronics-prismfly.analytics_253839784.events_*`,
  unnest(event_params) as `experiment_id_param`,
  unnest(event_params) as `variation_id_param`
WHERE
  event_name='experiment_viewed'
    AND experiment_id_param.key = 'experiment_id'
    AND experiment_id_param.value.string_value = 'aa-test-cobra'
  AND variation_id_param.key = 'variation_id'
Group by 1
This query will show you that you have some
user_pseudo_id
that have multiple variations. Maybe you should consider restarting the test with a new
experiment
key, so that you can see if this happens again. Check. to ensure you are always setting the
user_psuedo_id
in the SDK before making any checks for feature values in the app.
Copy code
WITH variations AS (
SELECT
  COUNT(DISTINCT variation_id_param.value.int_value) as n_variations
  user_psuedo_id
FROM
  `cedar-electronics-prismfly.analytics_253839784.events_*`,
  unnest(event_params) as `experiment_id_param`,
  unnest(event_params) as `variation_id_param`
WHERE
  event_name='experiment_viewed'
    AND experiment_id_param.key = 'experiment_id'
    AND experiment_id_param.value.string_value = 'aa-test-cobra'
    AND variation_id_param.key = 'variation_id'
GROUP BY user_pseudo_id
)

SELECT
 n_variations,
 COUNT(*)
FROM variations
GROUP BY 1
t
I am getting this value
image.png
Does that mean only 8 user_pseudo_ids are able to see the variation 2 ?
Also @helpful-application-7107 please let me know if the below experiment query is right ?
SELECT user_pseudo_id as anonymous_id, TIMESTAMP_MICROS(event_timestamp) as timestamp, experiment_id_param.value.string_value AS experiment_id, variation_id_param.value.int_value AS variation_id, geo.country as country, traffic_source.source as source, traffic_source.medium as medium, device.category as device, device.web_info.browser as browser, device.operating_system as os FROM
cedar-electronics-prismfly
.
analytics_253839784
.
events_*
, UNNEST(event_params) AS experiment_id_param, UNNEST(event_params) AS variation_id_param WHERE ((_TABLE_SUFFIX BETWEEN '{{date startDateISO "yyyyMMdd"}}' AND '{{date endDateISO "yyyyMMdd"}}') OR (_TABLE_SUFFIX BETWEEN 'intraday_{{date startDateISO "yyyyMMdd"}}' AND 'intraday_{{date endDateISO "yyyyMMdd"}}')) AND event_name = 'experiment_viewed' AND experiment_id_param.key = 'experiment_id' AND variation_id_param.key = 'variation_id' AND user_pseudo_id is not null
We are using the same experiment query for all our clients which is working fine. We are finding issues only with one account.
Also to add on, the account we have a problem has two websites. We have added two projects under the account. One project is working fine, the issue is with the other project (which has a different website linked to it)
Can you please help us to figure it out whats wrong
h
Does that mean only 8 user_pseudo_ids are able to see the variation 2 ?
No, this means that there are 8 users that have seen 2 variations.
please let me know if the below experiment query is right ?
The query looks fine. If it's an issue with one project and the queries are the same, I would assume that the problem is how they are implementing the SDK in their code.
e
the code is pretty much the same with other store, @helpful-application-7107, sent you a DM for you to see the full code 🙏
h
Ok, so since the Multiple Exposures issue isn't resolved anymore, I think your setup is fine. The current issue is that when you do your analysis, no users show up as adding to cart. The main reason for this is: there are no
user_pseudo_id
values that both have (1) an experiment exposure and (2) an add_to_cart event. That isn't necessarily a problem, because maybe no users in your experiment are adding things to cart, but I don't think that's what's going on. I think it's more likely that you are not persisting the user_psuedo_id across pages, so that when users go to
add_to_cart
they're using a different
user_pseudo_id
. You are doing
getUUID
right in the
gb
instance, which means you aren't persisting it. Therefore, when you send the experiment exposure and the add_to_cart event to your BigQuery, they're happening with totally random user ids, so we cannot match them.
You need to fix your implementation to persist the UUID in your user's cookies or something. If you read the thread above, there's a gist shared at the end with an example of how to do that.
t
Hey @helpful-application-7107 we tried all the above methods, but the issue still exists. I am wondering if the issue is because two websites in connected under a single organization account? Is there any chance because of that? please advice.
h
Are they using the same data source to log their events and for analysis in GrowthBook?
t
No two different datasource
h
Then no, it shouldn't be a problem. How are you persisting UUIDs?
e
@helpful-application-7107 we are using this code I think Graham shared it to the other thread, this code good enough on saving UUIDs to the browser cookie. https://gist.github.com/Auz/0b8c79031b09bbee0e9542c4ec250f15 but we don't still have luck on getting the metrics we need
h
So you are now persisting the UUID in the cookie? Can you share the code in DM for the tracking callback as well as how you're tracking your metrics?
e
just to follow up @helpful-application-7107 sent you a DM 🙂