https://www.growthbook.io/ logo
Join Slack
Powered by
Hi, we're facing a problem with time travel (or la...
# experimentation
a
Hi, we're facing a problem with time travel (or lack off) in our experiments and metrics. We have some really slow moving metrics that are dependent on manual processes - meaning conversion can happen 14+ days after exposure. This works fine normally but sometimes we have multiple experiments lined up and want to end the current experiment before we have a significant result (because we will later on - we don't need more exposures, just conversions). When we do this, it seems like the results/metrics also are limited to the exposure period and we don't get these lagging effects represented correctly. When the experiment is over, all traffic for the experiment is routed to the baseline-option - also leading to Sample Ratio Mismatch for the baseline. More information in thread.
➕ 1
So this is first and foremost a implementation and usage problem. We're likely trying to test in an unintended way. However I do think the problem is solvable and probably best explained in an example. Say we have two periods: 1. Testing (week 1) 2. Measuring (week 2) So say we have customers exposed in week one. When I set the experiment phase to week 1, I would expect to follow the subset from that week and get the measures from everything in the future for these customers (no metric window). However as it is today, I would need to include week 2 also in the experiment phase in order to measure the conversions. But if we have had exposures in week 2 these are included as well.
In the Live SQL Preview for the metric it's stated (pseudo) like (and also that it's 'highly simplified' in it's current form):
Copy code
SELECT
  user_id AS user,
  -- Each matching user counts as 1 conversion
  1 AS value
FROM
  `metric_table`
WHERE
  (condition = 'conversion') AND
  -- Only after seeing the experiment
  timestamp >= exposure_timestamp
GROUP BY user
Which would lead to the right results (I believe). When examining the metric SQL for the exposure however, the CTE for metric is stated as (still pseudo):
Copy code
SELECT
      user_id as user,
      1 as value,
      timestamp
    FROM
      (
        SELECT
       <all columns>
         FROM
          `metric_table`
      ) m
      JOIN __identities_lookup_table i ON (i.user = m.user) -- a join table for different IDs
    WHERE
      (condition = 'conversion')
      AND m.timestamp >= <experiment start>
      AND m.timestamp <= <experiment end>
And it's this last line that is a problem for us. Is this a known problem? Would it be possible to activate something on either the experiment or the metric in order to not have a time limit on it?
s
Hi Thomas, I am a data scientist at GrowthBook, this is a great question. One solution is to employ conversion windows for your metrics. Then in your experiment settings (advanced), you can choose to exclude in progress conversions. One cost here - if you are using fact tables, you will lose fact table optimization. HTH, Luke
a
Thanks for your reply, Luke! This really helped! I thought the conversion window would only limit it even more but I see now that this is not the case. For us the best solution seems to be to just have a long (60 days) conversion window. Excluding in progress conversions seems to give no results - as there were no exposures 60 days ago:
Copy code
__userMetricJoin as (
    SELECT
      d.variation AS variation,
      d.user_id AS user_id,
      (
        CASE
          WHEN m.timestamp >= d.timestamp
          AND m.timestamp <= DATETIME_ADD(d.timestamp, INTERVAL 1440 HOUR) THEN m.value 
          ELSE NULL
        END
      ) as value
    FROM
      __distinctUsers d
      LEFT JOIN __metric m ON (m.user_id = d.user_id)
  )
Another weird thing is that when I have set the experiment start as 
2025-09-11
and end 
2025-09-25
, it's correctly represented in the exposure:
Copy code
AND e.timestamp >= '2025-09-11 00:00:00'
AND e.timestamp <= '2025-09-25 00:00:00'
But in the metric it's set as:
Copy code
AND m.timestamp >= '2025-08-28 00:00:00'
AND m.timestamp <= '2025-11-24 00:00:00'
2025-11-24
makes sense as that's 60 days (conversion window) after 
2025-09-25
but 
2025-08-25
doesn't make any sense to me. For us it doesn't cause any problems - just a curiosity
s
Hi Thomas,
as there were no exposures 60 days ago:
I am a bit confused - I think the sql is looking 60 days into the future, not the past, which seems correct (please see below): Luke WHEN m.timestamp >= d.timestamp AND m.timestamp <= DATETIME_ADD(d.timestamp, INTERVAL 1440 HOUR)
3 Views
Open in Slack
PreviousNext
Powered by