growthbookusers #give-feedback

Hello! Me again 😆 And again thanks for a great tool! We have had some internal discussions on the conversion window(CV), and that for many teams it does not make sense to actually have a limit. We have solved this by setting CV = 9999999 and adding the experiment’s start and end date to the metrics, but it is not bullet proof. Better might be to add more possibilities to choose another type of CV. I.e. when should, in theory, a conversion window actually start and end? Some of the answers are: 1. From the first

experiment_viewed

to first

experiment_viewed + window_length

2. From the last

experiment_viewed

to last

experiment_viewed + window_length

3. From the first

experiment_viewed

to last

experiment_viewed + window_length

4. From any

experiment_viewed

event to

experiment_viewed + window_length

(multiple conversion windows) In GB it is currently 1, but in many use cases we would prefer 4 or at least 3. Is this something you have considered, or have thoughts about?

prehistoric-summer-28496

06/08/2022, 1:14 PM

Hi! Is there a parameter for the conversion window similar to

'{{ startDate }}'

and

'{{ endDate }}'

? I am asking because for now, in use cases where conversion window should be infinity, the

'{{ endDate }}'

parameter gets filled by (

real end date taken from experiment configuration + conversion window

) and this leads to changing results after an experiment is stopped. Thus: 1. Having

'{{ conversionWindow }}'

parameter would allow to nicely edit metrics. 2. Also, it is not obvious that

'{{ endDate }}'

is actually experiment end date + conversion window 🙂 It gets clear after looking an rendered queries only.

💡 1

➕ 1

prehistoric-summer-28496

06/09/2022, 7:45 AM

Hi again 🙂 After some processing of the facts that the

'{{ endDate }}'

parameter (1) does not actually correspond the end date of the experiment in stopped experiments and current date in running experiments as one could logically assume and (2) corresponds to

experiment_end_date + conversion window

and

current_date + 2 days

in stopped and running experiments respectively, I would like to suggest to introduce

'{{conversionWindow}}')

and

'{{buffer}}')

parameters, so these two can be simply added to

{{ endDate }}'

which would correspond to the real end date of the experiment (or current date for running experiments).

chilly-umbrella-90459

06/13/2022, 2:53 PM

That’s probably has been discussed, but I would really love to see more use to environments where we could’ve connect environment to dataset and have All Metrics and Experiments to exist in all environments. For me that would be dev and prod environment — database structure and queries are the same, but now I need to create two sets of Metrics and set up two experiments to test and publish everything. Would be great to do it only once and put experiment from dev to prod mode with a button 🙂

chilly-umbrella-90459

06/13/2022, 2:57 PM

🚧 Attributes list here https://app.growthbook.io/settings/attributes constantly changes to default one + I don’t really understand why and when do we need attributes set (I saw them only in experiment creation but didn’t really understood what they do)

chilly-umbrella-90459

06/13/2022, 8:58 PM

One more: data isn’t updated in Result tab even after clicking Update data. Although label changes to “Updated less than a minute ago”, but when I look at queries results from menu, I see there old data. If I run these queries, I got different results in Athena web interface. I guess it reuses the same file with results maybe?

chilly-umbrella-90459

06/14/2022, 9:09 PM

First of all, great tool, honestly, so far enjoy using it and thanks for a fast fix with athena queries Just wanted to +1 on idea to have sample size calculation in the UI, as right now there is no clear indication on how reliable are results for given sample/base rate. Would be nice to have!

💡 1

chilly-umbrella-90459

06/15/2022, 8:08 AM

Got this error trying to introduce count metric that basically sums all likes and dislikes and went to -1

polite-pillow-33171

06/16/2022, 6:55 PM

Question about metrics (going of of this thread) - is there a roadmap for metric types development (i.e. which other metric types are in the works and timeline)?

lively-belgium-36533

06/17/2022, 9:47 AM

Hi, when I set

forceVariation

, and then i use it

useFeature

, i found it returns me the default value, not the forced variation value, then i check the source code. i found in line 353: here the

res.inExperiment

is false. is this a bug? or how can i get the forced variation value?

bulky-oxygen-41438

06/23/2022, 12:42 PM

We're having problems viewing all the projects when clicking the Projects button in the left side nav. If we go into the other menu items, sometimes we get to see more of the list - so it looks like it's got some overflow

hidden

on the space of the items in the nav, instead of the whole nav. The list in the picture should show more than 10 items, but only showing two

fresh-football-47124

06/26/2022, 9:07 PM

Hey everyone: we are working on a new UI for adding experiments to features, to hopefully make it easier to understand. Let us know if you have any thoughts.

👍 5

🙌 1

ambitious-activity-2609

06/28/2022, 3:12 PM

The word "observed" is spelled wrong in the SRM Warning:

polite-pillow-33171

07/06/2022, 6:21 PM

Can analysis for experiments that have 3 variations be more precise (~33.33% instead of 34/33/33%)? More generally, can experiments with odd-numbers of variations be more evenly split with greater decimal precision? For a 3-way experiment, we currently split traffic equally at ~33.33% (if not to a greater decimal precision) using an in-house enrollment system. However, when we use GrowthBook for analysis, GrowthBook does not accept values for traffic splits beyond the first two decimals (see image), so in a way, “even split” in GrowthBook is not really even. As a result, when we analyze our experiments, we often get a SRM Warning, esp since our tests have large sample sizes and p-values are a function of the sample size. While the SRM Warning is statistically true, we find the warning is to be conceptually misleading because the actual traffic split (of ~33.33%) is more even than GrowthBook’s expected “even split” of 34/33/33. We’re concerned that this warning will limit our Data Science team’s willingness to trust GrowthBook for analyses of large experiments and for tests that have odd-numbers of variations. Would it be possible for greater decimal precision on traffic splits? Thanks for all of your hard work on developing a great product!

adamant-exabyte-53836

07/07/2022, 12:50 PM

JFYI: there are many high & critical CVEs in prod found in image growthbook/growthbook:git-3ba8bda ID: sha256:5fa561444e125e3c605d782b229d5317e532b2abdadb8e449cda4bd760c0af2b OS Distribution Debian GNU/Linux 11 (bullseye) Critical (4) CVE-2022-2068 | CVE-2021-44906 | CVE-2021-3918 | CVE-2021-42740 High (27) CVE-2021-32723 | CVE-2022-23647 | CVE-2022-24785 | CVE-2022-31129 | CVE-2021-23341 | CVE-2021-32723 | CVE-2022-23647 | CVE-2020-15138 | CVE-2022-0654 | CVE-2020-7753 | CVE-2022-21680 | CVE-2022-21681.. most of which are caused by using node.js v14.x , it seems •

WORKDIR /usr/local/src/app

•

RUN apt-get update && apt-get install -y wget gnupg2 && echo "deb <https://deb.nodesource.com/node_14.x> buster main" > /etc/apt/sources.list.d/nodesource.list && wget -qO- <https://deb.nodesource.com/gpgkey/nodesource.gpg.key> | apt-key add - && echo "deb <https://dl.yarnpkg.com/debian/> stable main" > /etc/apt/sources.list.d/yarn.list && wget -qO- <https://dl.yarnpkg.com/debian/pubkey.gpg> | apt-key add - && apt-get update && apt-get install -yqq nodejs=$(apt-cache show nodejs|grep Version|grep nodesource|cut -c 10-) yarn && apt-get clean && rm -rf /var/lib/apt/lists/*

full-island-88199

07/07/2022, 1:10 PM

Hi! Just details, but would it be possible to sort the projects in alphabetical order? Also to sort by name and role under Team ? 🙏

gray-crayon-34592

07/08/2022, 8:57 AM

Hi! This is my first feedback comment so let me just share first how awesome it is to work with GrowthBook!! 🚀 Now, my little feature request... would it be possible when browsing "All Projects" and viewing a table of Features or Experiments or Metrics to add an extra column to the left of the tags showing the Project this element belongs to?

gray-crayon-34592

07/08/2022, 9:01 AM

We also want to add some governance on how tags are named and used but we find it problematic having to explain "the rules" outside of GrowthBook. Maybe you could enable an advanced feature that allows admins to add an info icon ℹ️ with customisable text next to a section in a form (or just a little customisable description below the title of the section). This way we could add a couple of lines on how to name Feature Keys or which tags to add for a Feature.

busy-horse-73824

07/08/2022, 9:10 AM

^ we would also be keen to add some kind of rules/conditions around what can be set up by default, or perhaps e.g. some templates for how features should be set up

busy-horse-73824

07/08/2022, 9:52 AM

Btw unrelated to the above, but it'd be good if metrics could have additional conditions/variations added inline e.g. we have a simple

course_page_viewed

event, which has

course_key

chapter_key

and

page_id

fields. If we're testing whether a content change helps people progress further, we might want to see whether more people make it to the end of the course, or to the next chapter, or whatever. Currently we have to make an individual metric for every single thing we might want to look at. It'd be better if we could have a base

course_page_viewed

metric, maybe with some minimal initial filtering, then optionally filter on extra fields Or as a rough equivalent, defining a metric with specific parameters which have to be filled in when selecting it (not as powerful, but maybe more compatible with e.g. SQL-backed metrics)

future-candle-10380

07/08/2022, 6:27 PM

Hi! I think I found a bug on the cloud version of growthbook or at least it behaves differently than local version. In the attached picture

Screenshot - cloud

and

Screenshot - local

you can see the data is exactly the same. In fact it’s taking from the same mixpanel production data store but the results are different. In cloud Chance to beat control is 50% and there is no Percent change graph. In local version it’s 36% with nice graph. In files

Screenshot - query cloud

and

Screenshot - query local

you can see that the query result data is identical. In files

Screenshot - metric cloud

and

Screenshot - metric local

you can see that the metric configuration is identical In files

Screenshot - experiment cloud

and

Screenshot - experiment local

you can see that the experiment configuration is identical When I export the transformed data that are going into the gbstats calculation it’s the same as in the query:

Copy code

{
  "var_id_map": {
    "0": 0,
    "1": 1
  },
  "var_names": [
    "Small",
    "Big"
  ],
  "weights": [
    0.5,
    0.5
  ],
  "type": "count",
  "ignore_nulls": false,
  "inverse": false,
  "max_dimensions": 20,
  "rows": [
    {
      "users": 16,
      "count": 13,
      "mean": 5.769230769230769,
      "stddev": 7.148467787993196,
      "dimension": "",
      "variation": "0"
    },
    {
      "users": 10,
      "count": 8,
      "mean": 5,
      "stddev": 2.7386127875258306,
      "dimension": "",
      "variation": "1"
    }
  ]
}

And when I run it through jupiter notebook I get exactly the same results (chanceToWin: 0.36) as in local version (see `Screenshot - notebook`:

Copy code

{
  "unknownVariations": [],
  "dimensions": [
    {
      "dimension": "",
      "srm": 0.23931654122149193,
      "variations": [
        {
          "cr": 4.6875,
          "value": 75,
          "users": 16,
          "stats": {
            "users": 16,
            "count": 13,
            "stddev": 6.803611336557586,
            "mean": 4.6875
          }
        },
        {
          "cr": 4,
          "value": 40,
          "users": 10,
          "expected": -0.1466666666666666,
          "chanceToWin": 0.3600440835131542,
          "uplift": {
            "dist": "lognormal",
            "mean": -0.15860503017663863,
            "stddev": 0.4426092654556529
          },
          "ci": [
            -0.6415991584084633,
            1.031741260830155
          ],
          "risk": [
            0.4933414277382564,
            1.1808414277382564
          ],
          "stats": {
            "users": 10,
            "count": 8,
            "stddev": 3.2058973436118907,
            "mean": 4
          }
        }
      ]
    }
  ]
}

I can’t debug the cloud version but my suspicion is that the calculation is for some reason different than on local version and I am trying to figure out why. Local commit:

16303dfba972b805bc3fcc5e6e6514cbf21a2209

busy-horse-73824

07/11/2022, 12:29 PM

Currently growthbook doesn't seem to work with custom Mixpanel events - is it possible to support those/some way I haven't seen before?

ancient-receptionist-63597

07/13/2022, 6:40 PM

It looks like the optional "Name" field in A/B Experiment Features for variations is not persisted through the GBExperiment or GBExperimentResult objects in Kotlin. Are there any thoughts on adding this as a property within the Kotlin SDK and/or in other SDKs to assist with internal analysis? Thanks!

helpful-hydrogen-62495

07/22/2022, 3:55 PM

Hey guys, an idea we had that would be helpful to see made a bit easier to do in GB. We have our internal team testing group of like 8 ppl who we end up adding to each feature flag manually prior to turning on A/B tests or before general availability of a feature. Every time now I have to add a new forced rule to every new feature flag and copy over that list of emails from a previous feature to a new feature, which makes it difficult to keep track of who is in the list. This is all in Prod btw, so it’s not about the dev/prod env. Perhaps there can be a way to designate a group of emails or something that we can more easily add to a feature flag for this kind of use case?

eager-rainbow-87882

07/22/2022, 10:26 PM

In short: Add support for Mongo
ISODate
on feature conditions. Long history: We are going to store some ISO date strings as part of our attributes and we need to use force conditions based on date ranges, like, for example forced enable a feature for users created before an

ISODate('2021-01-01')

. But if you already support specifying rules based on date ranges in some different way, please point me out the documentation about it. But the solution need to be dynamic, not fixed in our code (so doing the check in our code to assign the user to a specific boolean attribute is not an option for our use case).

few-memory-16563

07/28/2022, 4:26 PM

I would love to be able to see dates instead of relative time on the "stopped" experiments tab:

🙌 1

➕ 1

busy-horse-73824

08/03/2022, 3:05 PM

We've hit a couple of bugs

busy-horse-73824

08/03/2022, 3:06 PM

Error in Mixpanel query

busy-horse-73824

08/03/2022, 3:05 PM

Error when creating a feature - works if we switch the behaviour to "simple"