Hi all, we want to make sure we're tracking events/Metrics that happen within the duration of our experiments. Making a new Metric and editing the SQL shows some existing logic for where I think GrowthBook is pre-building in the logic to account for this. Myself and the data team are wondering if

_table_suffix

is a parameter of GB or is this something we need to set up in our own data warehouse for the tables we’re querying? cc @refined-addition-9216

Hi Matt -

_TABLE_SUFFIX

isn't a GrowthBook-specific parameter. This is most common with GA4/BigQuery Data Sources. Typically, GA4 stores events in Intraday tables, so most often, the

FROM

clause in the query is wildcarded (e.g. it queries

FROM events_*

) and the

_TABLE_SUFFIX

allows you to filter based on your metrics conversion window specifications.

@billions-xylophone-11752 and how do i interpret the latter half of the conditional text? is this supposed to be a templating injection? because we need to be able to see how to get the event within the duration of when the experiment is running…is the flow supposed to be essentially when the expeirment begins, we just make it greater than or equal to that date for that event?

Hey, Daniel! You are correct in those are template variables, and they're defined by the Experiment phase. So when the experiment analysis queries run, it will replace the template placeholder with the experiment's phase dates, to ensure that you're only looking at metrics within the experiment start/end date. If you need more control, you can configure the

Conversion Delay

for a particular metric. More details on this can be found here.

Are "experiment analysis queries" the same as Metric queries? Or do you mean 'Experiment Assignment Queries'?

how would you modify this for a query given that our date filter is in date type (so the equivalent of table_suffix) for filtering of the string literal ISO param

@tall-branch-42668 Sorry for the lack of clarity - when building an experiment analysis, we run a number of queries, including queries for the metrics and the experiment assignment queries. From the experiment analysis tab, you can click the "3 dot menu" and select "Show Queries" where you can see the actual queries we're running to provide the experiment analysis. And this is where you can see the template variables being replaced with the actual values.

We are using GBQ and GA4

@tall-branch-42668 I can't at the moment, unfortunately. If you outline what issue you're trying to solve, I can dig in a bit later today/first thing tomorrow. I'm just not exactly sure what the issue is.

We want to ensure the results of the experiment are only including Metrics built for events we track like

trial_created

or

purchase

that happen during the duration of the experiment we're running. Right now we're not confident that the queries we have in place are doing this. We think we're close to understanding this, however the

'{{date startDateISO "yyyyMMdd"}}'

and

'{{date endDateISO "yyyyMMdd"}}'

are unclear as to where

startDateISO

and

endDateISO

are getting defined. When we keep those in place without editing them, we get an error in the Rendered SQL (screenshots attached). The startDateISO seems to grab today, and endDateISO grabs a date 2 days from now - why is this happening? Additionally, per the above, we don't have

_TABLE_SUFFIX

in our GA4 setup, so we had assumed that potentially using

timestamp

in-lieu of this would work - which does not seem to be the case, but in-lieu of

_TABLE_SUFFIX

how can we set this up to only display results inside the duration of the test? @refined-addition-9216 Is this all simply because we don't have

timestamp

annotated as

"yyyyMMdd"

?

First things first:

we don't have

_TABLE_SUFFIX

in our GA4 setup

Just checking, is this because you have some custom export to BQ that has a different schema than the one we assume for most people with GA4?

That's a question for our Head of Data @refined-addition-9216, I'm unsure

I think you want to just have

timestamp BETWEEN '{{ startDate }}' AND '{{ endDate}}'

Should I include

ISO

in those?

If

timestamp

is

DATETIME

or

TIMESTAMP

then you don't want to use the formatting that we use for most people's GA4 Schemas where we need to filter the table_suffix down to ensure we aren't scanning too much data. You need formatting for the string literal that comes is more like

yyyy-MM-dd

and comes with time as well. https://docs.growthbook.io/app/metrics#sql-templates

Good! That's what we're after. An update: The edited query in the

trial_created

Metric I've set up is now fixed and seems happy with reducing the line to

timestamp BETWEEN '{{startDateISO}}' AND '{{endDateISO}}'

However, the results of the experiment we have this Metric set up on did not change upon updating the data. My next question is: do these start and end dates need to also exist in our Experiment Assignment Queries? We have some pretty basic queries set for the 'Experiment Assignment Queries'. For 'Anonymous Visitors', we have:

SELECT

user_pseudo_id,

TIMESTAMP_MICROS(event_timestamp) as timestamp,

experiment_name AS experiment_id,

experiment_variation AS variation_id,

geo.country AS country,

traffic_source.source AS source,

traffic_source.medium AS medium,

device.category AS device,

device.web_info.browser AS browser

device.operating_system AS os

FROM

``puck-data-platform`.

analytics

.`int_fct_ga_events``

WHERE

event_name = 'experiment-viewed'

AND user_pseudo_id is not null

For 'Logged-in Users', we have:

SELECT

user_pseudo_id,

TIMESTAMP_MICROS(event_timestamp) as timestamp,

experiment_name AS experiment_id,

experiment_variation AS variation_id,

geo.country AS country,

traffic_source.source AS source,

traffic_source.medium AS medium,

device.category AS device,

device.web_info.browser AS browser

device.operating_system AS os

FROM

``puck-data-platform`.

analytics

.`int_fct_ga_events``

WHERE

event_name = 'experiment-viewed'

AND user_id is not null

However, since we had set these up in GB first before diving as deep as we have, we had commented out the templated

(_TABLE_SUFFIX BETWEEN '{{date startDateISO "yyyyMMdd"}}' etc etc etc

So because this is a novel, question is here again: Do we need the

startDateISO

and

endDateISO

in our Assignment Queries as well?

You don't need them as we'll handle date filtering downstream of these queries. Furthermore, we even try to hit your partitions with an outer CTE, so adding them may do nothing. However, if you want to be safe and do your best to ensure BQ hits your partitions, you can also add them to your Assignment Queries.

When I go to set up a Visual Editor change for a new experiment I'm receiving this error:

'load-viz-changeset-failed'

- any ideas how to remedy? I had set a new Admin Secret Key for the Chrome extension to implement changes with the Visual Editor but now I'm running into this

Hi @tall-branch-42668 I can help w/ the visual editor. Can you confirm what API host you're using in the visual editor options dialog?

Yeah the host I entered is:

<https://cdn.growthbook.io>

From our SDK Config -> SDK Connections

Ah so that's the issue - It should be pointing to your API host and not the CDN ... Do know know what that is? I can help you find it if needed

Our API Host shows as CDN, should that change?

Oh yeah, that can be confusing. The SDKs use a different endpoint than our Visual Editor for caching reasons. That should probably be re-worded

Alrighty that worked, thank you!

Great no problem! One question I have - did you experience an error when trying to open the Visual Editor the first time that led you to entering the API credentials manually? It's intended to set those values for you automatically on first load.

Def re-word, for non-technical laymen like me it'd be helpful 🙃

I see, so the visual editor did not show up immediately when you loaded the page, is that right?

Tbh I can't totally recall, it may have popped up and prompted me with a button to enter those things, which opened a new tab for the extension

<chrome://extensions/?etcetc>

where I entered that info

Ok, no worries. Would you mind sharing the URL that you were trying to open the visual editor with? I can investigate on my side. Really appreciate your help

Got it, thanks!

Additionally, we're trying to implement a visual editor change on a specific JS expression on our site; our paywall (not tied to any URL), my dev says

I don't see a way to only run this test when the anon first paywall is visible. To do that, we need to be able to target the test to the result of a JS expression. I can only find ways to target based on URLs and attributes. If you see the same thing, I guess we need to check with GB but it doesn't seem possible from what I see so far

Is there another implementation route we can take for making this visual change for this test?

I think you'd need to set the result of the JS expression as an attribute to the GB SDK, visual experiment or not, in order to bucket the user into the right variation. Let me know if I misunderstood the scenario

Yep that's exactly what we did and it works

this is the JS SDK?

Yes it is (also forgot to attach screenshot of error from Experiment results, now attached)

It indeed looks like for some reason the

trackingCallback

is not sending the variation names as expected.

How do we resolve this?

I think we'd have to debug what your tracking callback looks like. Are you able to get the person who is in charge of that to share the information?

I haven't 😞 Sounds interesting though

Screenshot 2023-12-12 at 5.38.08 PM.png,Screenshot 2023-12-12 at 5.38.21 PM.png

Ah, you're returning

result.value

for the variation id, but that's going to return the actual value rather than the variation key. Like this example shows, you want to submit

result.key

to

gtag

, not

result.value

We had made the change from

key

to

value

a month ago based on a Zoom call we had with @fresh-football-47124.. we're sure that's the change?

No, it should be one implementation here that is the same across tests, but using

value

may have just "happened" to work before if the value for each feature was the same as the id for each feature. But I'm not 100% confident on what @fresh-football-47124 saw in that call.

Alright we're swapping back to the values we had originally, per your suggestion. Will let you know how that goes

I noticed a console.log statement in your tracking callback. Could you copy/paste the output of that from your browser console?

you may still have that error unless you change the experiment name or date limit the experiment so that just new events are considered - or just look at the most recent rows and make sure the variation id is being tracked correctly - it's not going to update past data.

Would adding a new Phase accomplish this?

Yes, I think in this case just creating a new phase should resolve this issue.

Do I need to reset all of my Visual Editor changes? When I click "Add New Phase" there's a warning that reads "Starting a new phase will immediately affect all linked Feature Flags and Visual Changes" - how will these be affected?

No you shouldn't need to reset your changes. It should be fine.

Setting a new Phase seems to have cleared the issue

Phases just rely on date to filter results. So we don't send phase information to the warehouse. The only way to reliably send this information is to change the experiment key altogether (often best done by creating an entirely new experiment, potentially linking the same features/visual changes).

Got it. How can I link the same features/visual changes from other experiments in GB? Or do you just mean manually replicating the same steps to set them up for the newly created experiment?

we don't offer that currently

Got it thank you

Yeah, it looks like some of the tracking callbacks from before your change are still sneaking through in the phase filter.

No problem, good to work out all of these bumps! I was also wondering how the Conversion Window Hours affect our experiments and results, particularly in this case. I have one of our Metrics set to 96hrs for this, does that mean that the trackingCallback would return the original `result.value`s from Phase 1 for exposed users for that time duration? As in, should Phase 2 start 96hrs later to make sure that we don't get the incorrect variation ID returns from Phase 1? Additionally, somewhat separately, if we're running an experiment and monitoring Metrics for something broad like trial_conversions or purchases, we don't always have customers convert on Day 0, sometimes they take 30 days to convert. If we ran a test for 30 days, how does GB monitor results of users who experienced the experiment on, say, Day 28 but who then converted on, say, Day 45 when the experiment is no longer running? Should we wait some time period beyond the experiment stop date to update the data for a more accurate result?

As in, should Phase 2 start 96hrs later to make sure that we don't get the incorrect variation ID returns from Phase 1?

No. The phases will filter out the experiment exposures, and your 96 hour conversion window will be from the first experiment exposure we can find in the phase 2 date range.