Hi there! I'm using the python SDK and just realized about an unexpected behavior that I'd like to share When the id attribute used for hashing and the splitting criteria is: • An empty string,

''

, then it is filtered out from the experiment, and the callback function doesn't run, and the feature value is the default one. Expected behavior. • A null value,

None

, then it is NOT filtered out from the experiment, the callback function runs, and the feature value is assigned based on the hashing function and splitting logic. I find it kind of unexpected behaviour, as I would expect the same behavior as if it was an empty string WDYT?

Any thoughts on why?

Nothing in the console

I have a navbar location I want to hide based on this flag

lead-pipeline

however even though the navbar is in the context of the

GrowthBookProvider

the

useFeature

hook returns an

unknownFeature

-- any suggestions for debugging?

hello, I've a question, I just looked at growthbook SDK and realized the tracking callback only happens in an experiment, not on a feature. If I've defined a feature in growthbook UI, and want to use it, should I be using experiment in my code instead of feature?

I hope this is my last question for today: Someone is expressing concern that:

if feature("something").On || feature("another-thing").On {
  // do something
}

And if the something is On, then we don't evaluate another-thing at all, I thought we generally don't do ORs in experiments, and only ANDs (like in case you have an experiment inside an experiment) And if it's really OR, then we'd have to evaluate those outside of the conditional and instead save them into a variable, But has anyone ever seen OR in an experiment

Do I need to add attributes manually through the GUI for Growthbook or is it enough to just do setAttributes and then it will be added to https://app.growthbook.io/settings/attributes ? Would be nice to have an API for adding new attributes if it doesn’t already exist 🙂

Hi, question; Can we track feature flag changes in real time on client, in a way the app can react to those changes instantly? Tried it with just basic configuration using

useFeature

hook with react, but changes wont kick in unless i refresh the app manually or close it.

Hi, I tried inputting multiple

includes

and

does not include

comparisons into a Force Rule Feature and it looks like I am only able to have 1 of each while extras are discarded. This is to perform an array object equivalency check. Is this intentional that only one of each of these is able to be used?

Can someone help me here, 10 days before, I was able to run localhost:3000 directly, now facing multiple issues, Need help in running the docker on local and opening the UI

@here one of my metric showing bit strange result, 110/109.

Hi, we're trying to use an override rule to force anyone who goes to a URL containing the word

hold

into the

on

version of a feature. Do we have to include URL as an attribute in the SDK for this to work, or does growthbook read from the current page URL by default (or something else?)

Hi,can anyone help me to understand where is the user id with it is cookies being stored in remote server or on the front end and how to change it to persistant session?

Hello, can someone please help me with connecting data from Matomo? Should I create new Data Source?

Hi, I'm having a problem when configurating the local version of Growthbook. I get an error when I enter localhost:3000 and I can't register or login. I tried with and without a

config.yml

. By the way, I'm on a mac

Hi! Quick question related to metrics and

conversion window

parameter: if I set up this

conversion window = 48 (hours)

, in a binary variable as an example, what would happen is: • All the distinct users assigned to the experiment in the

Experiment Assignment Query

will be counted in the denominator. • Just the users with a conversion, with a timestamp falling in the conversion_window (

experiment assignment timestamp

+ 48h) will be counted in the numerator. However, there will be some users added into the denominator, which still have time to conver (imagine they just joined the experiment, they still have 48h to convert). This means that the metric will not be matured until 48h after of finishing the experiment. Is this reasoning correct? Many thanks!

any plans for releasing a dotnet sdk? 🙂 I don't have enough expertise on this area to write one myself. If that won't work, I might have to wrap the golang SDK inside a docker container, and have that running as a sidecar which responds to my HTTP input 🙂

I recently created a growthbook proxy for work. I believe this would have helped everyone not just us. Having a proxy, which updates list of features based on webhook events sent by growthbook instead of things being polled from individual services would probably help growthbook keep it's CDN fees a little down. Basically my idea is to have a proxy kinda thing, which listens to webhook events AND has a polling duration. On top of that, it also accepts the growthbook API Key as environment variable and make sure it contains the latest set of features. This helps growthbook.io keep it's CDN fees a little down hopefully. And individual businesses wouldn't have to hardcode their API Keys in the clients or individual services every time. Is this something growthbook team would be interested in? 🙂 Since I've done this before, I should be able to contribute to this in almost 2 days, only thing new for me would be to have the webhook part (I wouldn't apply the change, rather when I receive the webhook, I'd ask for the latest features) 🙂 (only complication is that maybe CDN won't let me reach growthbook API just yet, unless you guys are invalidating that somehow) I understand it might be some undertaking, we can have a discussion first 🙂

We’re running an A/A experiment (random assignment to two variants, but user sees the same product). Typically there’s been small variation for metrics in Growthbook but most of the violin plots are centered at close to 0 now that ~50,000 users have been assigned. Our assignments should be random. But one of our core binomial metrics is showing a 2.3% percent drop in the “experiment” group. Growthbook computes a 3.68% chance that the variant will beat the control, when I’d expect ~50% chance, since the product is the same. This is a ratio metric, with the denominator set to one of the other states in our funnel. The denominator’s metric shows the expected 0% change. There’s a conversion window on the metric, but I tried copying the metric and 1) setting the denominator to all experiment users and 2) modifying the conversion window, and the percent size / probability distribution is roughly the same. This experiment contains has been running for 4 months. I created a new experiment phase and looked at just data from the last 1 month, and I wasn’t able to reproduce this. Seeing a 0% percent change for the new phase. I had a few questions, and I’m open to any advice people have on interpreting the results of A/A tests in Growthbook: • What could help explain this outcome? Are these false positives expected, even after such a long experiment length? • Could this suggest a flaw in our assignment logic or our metric definition in any way? Again, the difference in most other metrics is close to 0, and the second phase showed a 0% difference, so the first result (on 4x the users) is confusing. • Does this suggest that I should be skeptical of any experiment result that moves the metric less than ~2.3%? Is the A/A test helping me understand the underlying natural variance for the metric in any way?

Hey we had a general question around using the $elemMatch on multiple values is this possible through the UI or through the advanced query editor?

hello guys, how do i reactivate a test?

Hey growthbook members, I'm giving the stab for the growthbook proxy and have written a design document here: https://cyberhck.notion.site/Design-Document-Growthbook-Proxy-afd54d4e6d7c4794889337abacf1fc4c please have a look and give feedback/changes that you'd like 🙂 If no one has any objections, I'll try to start development this weekend if no one has any objections 🙂

Hi all, we’re looking to get a better understanding of how the “Metric Conversion Window” setting plays into the experiment analysis. We took an experiment and toggled the setting from “Include In-Progress Conversions” to “Exclude In-Progress Conversions” and, after running “Re-run All Queries” compared the queries generated via “View Queries”. Surprisingly, the queries were identical for both settings. Is this expected, or where should we expect to see the Metric Conversion Window logic applied?

Hi, I have a question on how to use a

Retuning Visitors

as metric in GrowthBook. I use Firebase Analytics and currently I get almost the metrics business want to see but

Retuning Visitors

. What I stuck right now is, I can get a user count which represents the

Retuning Visitors

but it is divided by

All Experiment Users

and I get ratio instead of actual count. Any help is aprreciated.

Hi again! I was just wondering if you have any tool / experience / advice in terms of minimum amount of data to reach significance in an experiment. I’m aware of the bayesian use of risk of loss for calling the test, but this is something useful once the experiment has already started and there is data being collected. However, in the experiment design phase, there are some timelines (at least to be used as minimum) we need to agree with the rest of the teams. Many thanks again.

Hi, team! We’re running an experiment where the Growthbook analysis does not find users allocated to

variant_id = 0

. Looking at the raw data from BigQuery, the variants (

0

and

1

) are equally distributed (see screenshot), but variant

0

users do not show up in the experiment results. I’ve also verified that the there are no `userId`s exposed to both variants. Here’s the query from the analysis:

-- Activation (binomial)
WITH __rawExperiment as (
  SELECT
    user_id,
    received_at as timestamp,
    experiment_id,
    variation_id
  FROM
    `rudderstack_data.experiment`
),
__experiment as (
  -- Viewed Experiment
  SELECT
    e.user_id as user_id,
    cast(e.variation_id as string) as variation,
    CAST(e.timestamp as DATETIME) as conversion_start,
    DATETIME_ADD(CAST(e.timestamp as DATETIME), INTERVAL 336 HOUR) as conversion_end
  FROM
    __rawExperiment e
  WHERE
    e.experiment_id = 'hello-sanity_template'
    AND CAST(e.timestamp as DATETIME) >= DATETIME("2022-08-15 19:00:00")
),
__metric as (
  -- Metric (Activation)
  SELECT
    user_id as user_id,
    1 as value,
    CAST(m.timestamp as DATETIME) as conversion_start,
    CAST(m.timestamp as DATETIME) as conversion_end
  FROM
    (
      SELECT
        user_id,
        CAST(activation_date AS TIMESTAMP) AS timestamp
      FROM
        `dbt_production.dim_user`
      WHERE
        is_activated = TRUE
    ) m
  WHERE
    CAST(m.timestamp as DATETIME) >= DATETIME("2022-08-15 19:00:00")
),
__distinctUsers as (
  -- One row per user/dimension
  SELECT
    e.user_id,
    cast('All' as string) as dimension,
    (
      CASE
      WHEN count(distinct e.variation) > 1 THEN '__multiple__'
      ELSE max(e.variation) END
    ) as variation,
    MIN(e.conversion_start) as conversion_start,
    MIN(e.conversion_end) as conversion_end
  FROM
    __experiment e
  GROUP BY
    e.user_id
),
__userMetric as (
  -- Add in the aggregate metric value for each user
  SELECT
    d.variation,
    d.dimension,
    1 as value
  FROM
    __distinctUsers d
    JOIN __metric m ON (m.user_id = d.user_id)
  WHERE
    m.conversion_start >= d.conversion_start
    AND m.conversion_start <= d.conversion_end
  GROUP BY
    variation,
    dimension,
    d.user_id
),
__overallUsers as (
  -- Number of users in each variation
  SELECT
    variation,
    dimension,
    COUNT(*) as users
  FROM
    __distinctUsers
  GROUP BY
    variation,
    dimension
),
__stats as (
  -- Sum all user metrics together to get a total per variation/dimension
  SELECT
    variation,
    dimension,
    COUNT(*) as count,
    AVG(value) as mean,
    STDDEV(value) as stddev
  FROM
    __userMetric
  GROUP BY
    variation,
    dimension
)
SELECT
  s.variation,
  s.dimension,
  s.count,
  s.mean,
  s.stddev,
  u.users
FROM
  __stats s
  JOIN __overallUsers u ON (
    s.variation = u.variation
    AND s.dimension = u.dimension
  )

Are you able to spot what’s wrong here?

Hey guys! We hosted Growthbook on Elastic Beanstalk and suddenly today it failed(No API response) with these logs. Could you point us in the right direction where we should debug, please? I will post the results here if I'm able to solve it, thanks!

Morning. Is there anything stateful in the core of the Growthbook service that would prevent a horizontally scalable deployment from working? For example:

Are SQL template variables available within Dimension definitions? We defined a dimension as:

SELECT 
  hid as user_id,
  CASE
    WHEN created_at >= '{{ startDate }}' THEN 'New'
    ELSE 'Existing'
  END as value
from owners

But in all queries where we try to slice by the dimension,

{{ startDate }}

isn’t substituted

Hi everyone, I was setting up growthbook cloud with mysql data source but keep getting access denied error even after allowing

52.70.79.40

. Anyone else facing the same?