Hello, is there a way or a best practice on how long a bayesian based test should run? We had several cases where results completely turned around within a week after the test was running for 4 weeks already. I attached 2 screenshots of an example, where we had 36% chance to beat control (metric: Suggested Jobs Page - Qualified Application) with -3.7% decrease and a week later with only 500 more users per group (2600 to 3100) the chance to beat control went up to 82% with a 10% increase in the same metric. This was just random that we left the test running for another week delivering completely different results and I don't understand how to tell this beforehand, i.e. being sure that the results won't change anymore after a specific runtime.

Another question: How could I add a metric to understand D1 and D7 retention for a specific metric?

Hello, everybody! I'm new in Growthbook and I'm doing some research to my organization and I wanted to understand how often can I refresh cache. I'm looking in docs, but I still haven't found something about this. Could you guys help me?

This message was deleted.

An experiment that changes the metric by less than this percent will be considered a draw (default 50)

Just curious why such large number was chosen for the default value, isn't 50% quite a big move for practically any metric?

hey guys, I'm working on installing growth book on my web app that's built in react. I went through the initial documentation steps in the docs however, I'm running into this issue when I try to load my app. Looking to get unblocked. Thanks!

Hey there, am I correct in thinking that the built in Date dimensional analysis includes only behavior that happened within a user's first conversion window on a given day? Any reason it can't include all of a user's behavior on a given day?

Hi, I am currently integrating growthbook to NextJS application. I can not make use of the SSR to load the GB features as the SSR response is being cached. Is there a work around to load the features quicker?

I used GrowthBook on client side using React/Next.js. But it doesn't seem to reflect in realtime the flag value when I toggle it in the GrowthBook dashboard. I have set

growthbook.loadFeatures({ autoRefresh: true })

in useEffect like in the docs at first render/initial webapp load. I'm also using GrowthBook's managed hosting. Anything I'm missing? Thank you in advance.

👋 Hi everyone!

Hello, folks. Is there a way to get results in Jupyter Notebook Query Runner in frequentist approach, i.e

p-values

,

confidence intervals

instead of

prob_beat_baseline

?

👋 hey all - in doing some prep work for a guest lecture I'm doing on building an experimentation platform I came across growthbook so thought I'd join in on the fun. My name is Paul Raff and I'm a Principal Data Scientist at Reddit, working on their experimentation platform among other things. I spent 7 years growing Microsoft's experimentation platform (ExP). I'll definitely be poking around and hope to contribute!

Cannot read property 'GrowthBook' of undefined For some of the user this error is thrown . My tracing callback function is not triggered at times for some users . Any possible reason why that’s happening @fresh-football-47124

Hi, I am using growthbook with nextJS. Currently I am following the given tutorials and loading features and setting attributes in

_app.tsx

. But I feel like its unnecessary for the pages that are not using the features. What would be the implications of setting attributes and fetching features in the same component that uses the feature?

Hey, I am using GrowthBook from Golang SDk. I wanted to know if I have to run the experiment for a particular user. How to provide the user information through code? For Example in the following image, the test-feature is like a killer switch. Suppose we want the killer switch to be ON for only User 123. what to do then?

What sort of caching do the javascript based SDKS do? There is a section called Built-in Fetching and Caching, but no information about how the caching actually works.

I have one more question, can we update the UI with Attributes from SDK? I don't see vice-versa.

Hello! I noticed there are missing rows for the percent change p values when I export the results from an experiment dashboard (see screenshot). I'm wondering if the missing values are a bug, or just not reported because the numbers are so small (which seems odd compared to some of the other pvalues listed (e.g., 1.08407061105709E-10). The issue only occurs when I export the results to a csv file- they appear just fine on the dashboard. @wide-jordan-14758 @ambitious-apartment-58735

Hi All!! Where can I find the architecture for Growthbook? I have tried to search throughout but cannot get. Also is there any benchmarking done ?

Hi folks, just wondering does means relative change calculation for ratio metrics involve some delta method too, as I see slightly different number compared to empirical relative change calculation i.e. (a-b)/b*100 ?

Nevermind, got it. The calculation is (a-b)/a *100.

Hi is it possible to send abtest results in other platforms like Google Analytics or ContentSquare?

Another SSR question (next.js or JVM based approches): the

experiment viewed

event for e.g. Snowplow requires us to send

experiment_viewed

with it's name, and

variation_id

where as the variation_id is the index of the variation, and not its name. F.e. if I set up Feature

button-color

with variants

A

and

B

the variation_ids will be 0 and 1. However, if my GrowthBook instance lives on the server, and I call

getFeatureValue

I only get

A

or

B

and not the variant_id required for the tracking call. Right now we tried to solve this in a very convoluted way: SERVER (nextjs):

const { status, features } = await getExperimentsFromCache();
  if (status !== 200) return { props: {} };

  // create a GrowthBook instance
  const growthBook = new GrowthBook({ features, attributes: userAttributes });

  // fetch all features and assign the user based on targeting Attributes
  Object.keys(growthBook.getFeatures()).map((feature) =>
    growthBook.getFeatureValue(feature, 'A')
  );

  // get all assignment results (we need it to grab the variation id)
  const allResults = growthBook.getAllResults();

  // construct our experiments object which contains keys, values and the GrowthBook variation id
  const experiments: Experiments = {};
  allResults.forEach((feature, name) => {
    if (feature?.result.inExperiment) {
      experiments[name] = {
        variant: feature.result.value,
        variationId: feature.result.variationId
      };
    }
  });

  return {
    props: { experiments }
  };

... and then the CLIENT (nextjs)

const trackViewedExperiment = (name: string, variationId: number) => {
  const eventObj = {
    event: 'growthbook_experiment_viewed',
    version: '1-0-0',
    owner: 'com.acme.experimentation',
    data: {
      experiment_viewed: name,
      variant_viewed: variationId
    }
  };
  // Send the viewed experiment to the data lake via snowplow (just logging here for clarity)
  console.log(`datalayer.push(${JSON.stringify(eventObj, null, 2)})`);
};

... is there any "easier" way to doing this?

Hey, I am trying to make some changes in the java sdk and test them. How have you created the jar from the code bcoz I seem to run into a lot of ClassNotFound errors for OkHttp and Gson transitive dependencies?

Hello, I wanted to ask a question about conversion windows and ratio metrics meaning data from outside of the experiment is including in calculations for the experiment. If I have a ratio metric where the denominator can occur substantially (in time) before the numerator, from what I can see I need to include a conversion window value. For my use case, this can be the length of the experiment, so might be high 10s of hours up to anything around 100 or 200. Setting a conversion window to 100 for example means that the endDate values in the generated query end up getting timeshifted by +100 hours, so if I’m doing retrospective analysis (or I have phases set up), I end up including data from outside of the time period of interest. I’m aware of the startDate and endDate jinja keywords that I can include in the definition of the metrics, but these end up getting affected by the conversion delay/window values too. The only workaround at the moment seems to be hardcoding date filters into the queries retrieving the data for the metrics which isn’t viable for anything other than playing around to work out how Growthbook functions. Am I missing something or am I doing something incorrect in my setup? Thanks in advance.

How could I add a metric/setup an experiment to understand D1 or Dx retention for a specific metric?

Hi All! How to connect PostgreSQL with my localhost. Also I don't have password for DB connection. Therefore how to connect to my local PostgreSQL?

Hi Growthbook Team! Thanks for your great efforts into this product; can’t wait to get it to production and start experimenting! I have a couple of questions, that I couldn’t find answers for: • Is a feature required to conduct an experiment, or can trackingCallback get triggered to send the tracking event based on the experiment alone? • How are users assigned to experiments? I didn’t find any related endpoint in the API response or SDK’s internal code (I read swift’s SDK), and I’m unsure if the assignment is done locally. Can you clarify this? I’m asking these questions because trackingCallback is not being triggered. Please let me know if you need any further details Looking forward to hearing from you, much appreciated 🙏

Hi Growthbook Team! How do I get the rule that is used from

onFeatureUsage

callback? For example, I have 3 rules as below, the first two are force rules, and the last one is for A/B experiments. I have tried several times but it seems

ruleId

is not working...

Hi team! After separating MongoDB to another service and doing dump & load of the existing data, is there anything else left to do in order to GrowthBook recognise the organisation and other data? Such as setting the mongo db database to use? Currently I can list admin, local, config and test databases with show collections command where test contains all the data.