For the auto-generated metrics chart, we find the past 3 months segmented by day to be limiting. Daily averages are highly variable and I imagine that's the case for many B2B/B2C business metrics based on the day of the week and a smaller sample size. More useful would be the previous 3 months grouped by week or month, or the previous year grouped by week or month. Even more useful might be two lines -- the above time period from the current year and one from the previous year (if available). A longer timeline, less aggressive grouping, and more context around those business metrics over time would make the information more stable and therefore useful at a glance for our purposes. Some options around this would be very nice, different metrics call for different groupings. But I realize that options are problematic to get into. It's a deep rabbit hole. 🙂

That's good feedback. We've been debating how many of the more general BI use cases we want to solve within Growth Book vs offloading to other apps like Superset or Mode.

Hey, a question about importing old experiments and filtering data for experiments in general. In the SQL code for importing of old experiments after configuring a data source, I've found these filters:

SELECT
  *
FROM
  __variations
WHERE
  -- Skip experiments with fewer than 200 users since they don't have enough data
  users > 200 -- Skip experiments that are 5 days or shorter (most likely means it was stopped early)
  AND datediff(day, start_date, end_date) > 5 -- Skip experiments that start of the very first day since we're likely missing data
  AND datediff(day, '2020-08-24 18:53:16', start_date) > 2
ORDER BY
  experiment_id ASC,
  variation_id ASC

while this filter

AND datediff(day, start_date, end_date) > 5

makes sense in general, it excludes some important edge cases, e.g. 1. Dates filter: imagine an email experiment where different versions of the email go on the same day to the users. The

datediff(day, start_date, end_date) > 5

will make this experiment impossible to analyze using Growthbook because in this case start_date = end_date ("experiment_view" event happens on the same day for everyone) 2. Imagine an experiment on an event-specific webpage (e.g. convention etc.) - same problem with all experiment exposures on the single day here Suggestion - can the values for these filters be made as optional parameters in the

config.yml

? Smth like

minimum_days = 6

by default

i'm not sure if the

0

and

0

is a good thing - and its just complaining i don't have a variation

1

yet

That's a new feature to detect when the experiment is missing data or configured incorrectly, but sounds like it has too many false positives. We can make that less sensitive

although the notification seems to suggest that it will start a fresh?

or is this user error on my part?

Hi, i'm trying to use the visual editor to change some text that appears dynamically on our calculator result but the component times out in the editor mode and means i can't change the text

Hi! Have you ever thought about adding a default dimension of new vs. returning visitors (relative to the experiment)? New as in first seen after the experiment has started, and returning as those who have some page views before the experiment was launched. Right now, I'm trying to define it using subqueries on the experiment query, but maybe this use case is general enough to always include by default?

Another question: we now have a self-hosted Growthbook instance running and a bunch of past experiments in the database. If I import any experiment, I can analyze its results, but the experiment is always marked as either "Stopped" (end date is before current date when importing) or "Did not finish" (end date is after current data when importing). In any case, I think the results would not update automatically, right? Is there a way to import a still running experiment? This is necessary if we want to use another source of experiment assignment other than Growthbook's SDK.. Or am I missing something?

hi! how about a Matomo Integration? I opened an issue for this. https://github.com/growthbook/growthbook/issues/86 is this something whcih was already considered or is in a pipeline. I think GrowthBook with Matomo is a killer combo, becuase it would be the first GDPR friendly/self hosted A/B Testing Framework I know which such a big scope!

Greetings <!here>, hope you all are doing well 🤓 I have few inquiries for potential GB improvement. Hope they'll make some sense to you guys The first one is related to adding the "sorting" feature to "Your Metrics" & "Data Sources" resume tables: As you can see the quantity of our metrics is growing, there are various datasources, tags, etc. It would be great to be able to fix their order for better orientation

The second thing is related to providing the confidence estimate for the given experiment when analysed at different dimension levels. I understand your concern about multiple-test problem, still, at least in Typeform, we need that number. IMHO, it is better to have it in the toolset and then letting people decide whether use it or not, than not having it and limiting final user at deciding what to do

I recently wiped my local mongo and after creating the first user, got booted out to the "Ask your admin to invite you to the organization." message. I noticed PR#92 was supposed to disable creating more than one organization on non-cloud deploys, but it seems creation is being disallowed even when there are no organizations at all. Poking around the db reveals the user was created anyway and that no organizations exist. Did I miss a change to how initial signup works for single-tenant setups between 0.5.0 and 0.6.0?

Small bit of feedback - it seems that the violin plot UI doesn’t handle really big winners very well. If I include other metrics which are closer to the 0% change range then it works fine, but just 1 metric with a big change doesn’t work. Sadly this is a rarity 😭

I have a feedback which might not be great from the statistical point of view (I am not sure). Is it possible to add the required sample rate given the current uplift and current base conversion rate? Example, I have these stats in my current experiment. It's been running for a while and we didn't have the prior conversion rates so we rely on ones generated by growthbook to calculate the required sample. We use statistical calculator to figure out the required sample size, but maybe it would make sense to have it in the interface. Again, I'm not sure it makes sense from the statistical point of view

hey all, I want to run A/B tests within my product on behalf of my users (other businesses) and display the results of the tests to them in our dashboard. This would require the ability to programmatically create and view experiments (likely via API). I don't believe Growthbook supports this at the moment though. Are there any immediate plans for this? Thanks!

Hi everyone! My name is Juan, I'm a Data Scientist and colleague of @miniature-france-97575 at multilayer.io @future-teacher-7046, I believe you both had a video call recently. First of all, we recently evaluated Growthbook and are very impressed! 🙂 I have a bit of feedback: • We would be very interested in seeing this enhancement go through, as with one of our main clients we need to connect through an SSH Bastion when working from outside their intranet. https://github.com/growthbook/growthbook/issues/46 I have a Python implementation example of connecting to an SSH Bastion to Redshift and Bigquery, if you guys are interested. • From experience with previous employers & clients, I would love to see a Metrics page that is a bit more configurable. What I mean is, at some places some stakeholders have asked to remove all "useless stats", and at other places they almost want links to the papers supporting the statistics. It all depends on the kind of stakeholders. Do not mean this as criticism though, excellent work! 🙂

Simple Deployment to Kubernetes using

nginx-ingress

and

cert-manager

- just sharing here

apiVersion: apps/v1
kind: Deployment
metadata:
  name: growthbook-web
  labels:
    app: growthbook-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: growthbook-web
  template:
    metadata:
      labels:
        app: growthbook-web
    spec:
      containers:
        - name: growthbook-web
          image: growthbook/growthbook:latest
          env:
            - name: APP_ORIGIN
              value: <https://growthbook>.<DOMAIN_NAME>
            - name: API_HOST
              value: <https://growthbook-api>.<DOMAIN_NAME>
            - name: MONGODB_URI
              value: <mongodb://root:password@growthbook-mongo.default:27017>
          ports:
            - containerPort: 3000
              name: web
            - containerPort: 3100
              name: api
---
apiVersion: v1
kind: Service
metadata:
  name: growthbook-web
spec:
  ports:
    - name: web
      port: 3000
      targetPort: 3000
    - name: api
      port: 3100
      targetPort: 3100
  selector:
    app: growthbook-web
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: growthbook-mongo
  labels:
    app: growthbook-mongo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: growthbook-mongo
  template:
    metadata:
      labels:
        app: growthbook-mongo
    spec:
      containers:
        - name: growthbook-mongo
          image: mongo:latest
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              value: root
            - name: MONGO_INITDB_ROOT_PASSWORD
              value: password
          ports:
            - containerPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: growthbook-mongo
spec:
  ports:
    - name: web
      port: 27017
      targetPort: 27017
  selector:
    app: growthbook-mongo
  type: ClusterIP
---
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt
    <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "300"
  labels:
    app: growthbook-web
  name: growthbook-web
spec:
  ingressClassName: nginx
  rules:
    - host: growthbook.<DOMAIN_NAME>
      http:
        paths:
          - backend:
              service:
                name: growthbook-web
                port:
                  name: web
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - growthbook.<DOMAIN_NAME>
      secretName: growthbook-letsencrypt
---
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt
    <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "300"
  labels:
    app: growthbook-api
  name: growthbook-api
spec:
  ingressClassName: nginx
  rules:
    - host: growthbook-api.<DOMAIN_NAME>
      http:
        paths:
          - backend:
              service:
                name: growthbook-web
                port:
                  name: api
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - growthbook-api.<DOMAIN_NAME>
      secretName: growthbook-api-letsencrypt

Hello 👋 Is there any plans on implementing ROPE on top of the "minimum threshold" in metrics to avoid early stopping of an experiment?

Greetings folks 👋 I have some feedback regarding experiments with more than 1 variant label. "The risk of choosing" gets calculated for only selected variant. What would be great is to see its value for every variant in their corresponding column (right now, for experiments with more than 1 variant, you show results in columnar manner). Does it make sense? Best regards

Hiya ! So we are test driving Growthbook rn with our Product team and one pain point we’ve noticed is setting up experiments. We are doing experiment assignment internally using our own system so we have to manually pull the experiment tracking ID, the segment IDs etc; is there any sort of automated way to create experiments along with the splits. I.e. can we feed that data in and then have the experiment made automatically so we don’t have to make them by hand each time

👋 Not sure if this is a bug. We tried changing the conversion window value, but it doesn't apply the change to the query. Any thoughts on why this is happening?

Hi! I'm testing out Growthbook as part of a Data Hackathon at Gusto. I am enjoying testing it out so far, as I think in principle it's a pretty good fit for our stack and use cases. One of the things that's different about B2B companies is that we can have more than one entity ID we randomize over for logged in users - we have some experiments/feature flags that operate at the user-level, and others that operate at the client-level (a Gusto customer can have many users, and, for example we probably want payroll admins at a given company to all have the same experience when running payroll). It would be great to be able to customize multiple different "user id columns" for use in experiments and metrics. From what I'm seeing so far, I can sort of fake it by choosing the right ID in the Experiments query (we already have a way of doing this), and then defining

user_id AS user_id

for user-level metrics and

company_id AS user_id

for client-level versions of metrics. But it feels hacky and I'm imagining could easily confuse non-technical users.

I would also love to make it straightforward to separate the metrics themselves from the rest of the

config.yml

file (esp. the data source settings) - I'm imagining we'd want to back the metrics themselves in a git repo (but without secrets like what's in the data sources), and eventually I'd imagine we'd want to integrate a more general metrics layer like dbt is building.

Is there anywhere I can get a dump of summary statistics for a given experiment?

Hello! Loving GrowthBook and thank you so much for all your efforts! 🙌 I had a couple of questions/feedback/ideas. First, I was wondering if there are any plans to add outlier detection on the metrics? Second, it would be awesome to be able to adjust the time before and after the results when drilling down on date (picture) - would be so useful to ensure that the groups were “identical” prior to the experiment, and also if the experiment has a long term effect / how long it takes before the group align to same effect (if that makes sense…?)

it would be great to have a way to apply a group of metrics to an experiment with one click (e.g. our growth team is going to regularly look at a few metrics in every experiment, so it would be nice to just indicate “growth experiment”) similarly it would be nice to be able to generally organize all the metrics (tags, folders, etc) since there will be a number of teams with a variety of metrics that apply to their work.

Hey guys, appreciate the work on the sidebar. I will say it's a bit jarring to have one accordion close as another opens. Slack keeps sections open toggle others open/closed for example

One thing I've found myself doing is creating multiple forced override rules for small groups of users. There are many times where we want to roll out a feature to a limited number of users in prod (Our internal team, Super users, Beta testers) and I find myself copying forced rules a bunch. Any thoughts on templated rules/ a project forced rule library?