Thinking about an experiment we’re running right now but take this idea with a grain of salt. It would be extremely interesting to see a histogram of how many times each user generates a metric. We have a “number of exercises completed” metric where the

metric_count / user_count

is something like 5000 / 500. Would love to see a graph when I hover over that metric that shows me if the distribution is normal, skewed or just has a few outliers skewing the count (say 1 user is responsible for 1000 activities pulling up the average for everyone else). This is definitely possible in our data warehouse (mixpanel) but would be interesting to have it in the platform.

Would be good to have an api to create experiments instead of importing for analyzing results.

When you're switching from one data source to another, naturally you'd want to look at the same experiments from both data sources...

Something weird happening here, possibly a bug with the query being generated(?)

We have an interesting situation which we're not quite sure how to address The data we're working with is a combination of our ETL'd Mixpanel events, and internal backend transactional databases A lot of the data we work with is "special category" - health/sexuality/similar - so in order to protect it without locking down our data warehouse, we have a pseudonymization process. Most data (Mixpanel events and some backend data) is held against a "pseudonym ID" rather than the regular user ID. Some data (e.g. Stripe records, basic user registration info, etc) does just use the normal user ID though. In order to make this processable in tandem with the pseudonymized data in our warehouse, we make a copy all the user ID-oriented records so they can be used with the pseudonym ID as well, hashing all the IDs etc, removing any identifying information. Critically, to prevent reidentification via ordering/matching of high-entropy fields, we truncate all timestamps in this process, usually to the day

10/10 on the SQL support, we've just been working with it today and found it far easier than we thought we would. We were expecting to have to make a single table combining every source of events into a single consistent format, and to need to make our own record of experiment triggers, but it worked with all our DB structures with very minimal modification 💯 🙌

Hi Is anybody aware of reason why i cant delete my secret api key - is this a bug or not? I am using Build: 207c383 (2022-10-19). I click the delete, it pops up confirmation box , i click the button and the confirmation box reappears as below image shows. The secret api key remains in list and hasn't been deleted.

Hi there. I’ve read the docs on the usage Growthbook PHP library and it states that the library uses mongoDb-like syntax for matching experiment conditions. However, it’s not exactly clear for me what condition I use for matching a regular expression. The documentation on MongoDb states that the $eq operator implicitly evaluates a regular expression if one is found in the value fields. But will it work the same in Growthbook or should I explicitly specify the $regex operator?

Hi, I am sure there is a bug with the invite user in the latest release (Build: be1a246 (2022-11-01)). I don't have a smtp server set up and i get a manual link for the user to sign up. When they click on the link the user is able to start the register process but after creating password and clicking submit button the screen shows the following: This was working the other day. This is on self-hosted instance of growthbook.

Hi! We hit a blocker implementing the visual editor - link to our issue here. Looking to see if this can be addressed or if there is anything we're missing.

Hi team! I would like to suggest one feature, in case you think it would add value. We’ve finished our first experiment, and we have a clear winner for one of our metrics. We wanted to report the relative uplift detected (what it’s called Percent Change in the UI), but rather than just the mean of this distribution, also some CI (95% in our case). To get that, we had to pull the data from the metrics query, replicate the posteriors and simulate to get this distribution. I think it would be easier, if GrowthBook somehow show some CIs directly in the UI for this uplift distribution, or otherwise provides a way of interacting with this uplift distribution (which could be a jupyter notebook, etc). Thanks for the all your support!

Hi there! Is there any suitable operator in GrowthBook that would be suitable for condition “contains text” and “doesn’t contain text” ? I figure I could solve the “contains” cases with $regex, but still might need to cover “doesn’t contain” situation. Maybe there’s some inversion of $regex?

Hi everyone. Got confused a little bit when adding an idea to Growthbook. I really dig I have a chance to pre-calculate the sample size for an experiment, but here’s where I got confused. I chose a metric and user segment to calculate the sample size, but in the metric settings I have a conversion delay of 24 hours and conversion window of 48 hours, so I expected it to be reflected in the underlying query, but it doesn’t look it is. I expected the “conversion start” and “conversion end” to add 24 and 48 hours respectively.

__metric as ( -- Metric (Retention [1st Day][GA])
    SELECT
      user_id as user_id,
      1 as value,
      CAST(m.timestamp as DATETIME) as timestamp,
      CAST(m.timestamp as DATETIME) as conversion_start,
      CAST(m.timestamp as DATETIME) as conversion_end
    FROM
      (
        SELECT
          CAST(user_id AS INT64) AS user_id,
          1 AS value,
          created_at AS timestamp
        FROM
          `normalized.events`
        WHERE
          event_name = "page_view"
          AND DATE(created_at) >= '2022-09-01'
      )

Webhooks page is shown with a broken layout (Picture 1) when we have huge URL endpoints. Could you guys update that page's CSS to have something like this?

.gbtable td:nth-child(2) {
  word-break: break-all;
  width: 30%;
}

This way, the table can be shown right inside the container size without breaking it (Picture 2).

Hey GrowthBook, one feature suggestion which would help us out a lot: Right now as far as we know we can only track metrics on a per user basis. However, for us it would be also interesting relate the tracked metrics to other metrics than number of users. One example: we currently can only measure jobs viewed per user not jobs viewed per jobs sent (which is the more relevant metric for us). Our work around right now is that we added jobs sent as a guardrail metric and are kind of manually checking that we don't misinterpret the data Happy to go into more detail 😊

Hello! We have a very peculiar thing happen to us in a new revenue metric. When we run the query in GrowthBook I get these results (negative means and smaller stddev), while if I copy the exact same query and run it in Snowflake, I get these results (positive means and larger stddev)… 🤷‍♀️ The graph also looks very strange… Any ideas?

I wish there was an easier way to discard a feature revision. Currently you have to open the review and publish modal and

discard

is dangerously close to

publish

. Maybe a trash can icon on the right combined with a popover confirm?

I’ve finally gotten around exporting experiment analysis to Jupyter Notebooks, but there’s something I don’t get. When getting SQL query results and importing them to a data frame, data looks just about right, But when running Data Quality Check/Preparation and displaying the results, mean and standard deviation always show 1.0 and 0.0 respectively. As a result, the final table always shows probability to beat the control around 0.5

Ratio metric acting weird Hello! We are having some problems with ratio metrics (it is not the same as the negative variance problem). In the screenshot, you see the nominator number on top, then the denominator as second, then finally the ratio. But as you can see, the numbers for the nominator do not match. All have the same conversion window. When I take out the query, I cannot reproduce the numbers for the ratio metric. Could you help? Cc: @gray-crayon-34592

Hi, we're encountering a need for a lot of horizontal scrolling in order to compare variants to each other in the current Experiment Analysis UI. Is it possible to put in a feature request to be able to change the layout so within each metric row there are sub rows for each variant, so all the variants are vertically stacked within each metric? This way there is no horizontal scrolling required. Let me know if this makes sense/is possible 🙂 (cc @ambitious-apartment-58735)

Hey GrowthBook team 🙂 I'm a bit confused by the new on/off toggle for feature variations... First of all, why is variation 0 by default toggled off? 🤔 Second, what happens if we include 100% of traffic but only one of 2 variations is toggled on with a traffic split of 50%? Would that mean that 100% of the traffic is seeing that variation or 50%? 🤔

Hey Growthbook team, need some help clarifying what might be an edge case bug. In one of our configurations we left an experiment rule with a 0% inclusion and a

0/100% ON

Split. The default value is serving OFF, and this is the only rule. See image Our expectation: The flag should evaluate to

false

to all users because the experiment has 0% user inclusion and the default value is OFF. Actual result: The flag evaluated to this:

{
  "value": true,
  "on": true,
  "off": false,
  "source": "experiment",
  "ruleId": "",
  "experiment": {
    "variations": [
      false,
      true
    ],
    "key": "flag",
    "coverage": 0,
    "weights": [
      0,
      1
    ],
    "hashAttribute": "id"
  },
  "experimentResult": {
    "inExperiment": true,
    "variationId": 1,
    "value": true,
    "hashAttribute": "id",
    "hashValue": "inPNbLwltv0"
  }
}

Hi Growthbook team! We are using Growthbook for React Native and would like to add a targeting attribute based on the user's app version, but it doesn't seem like the existing attribute types support comparing semantic versions. Is this something that has come up before, and do you have any recommendations for how we could implement something like this? Ideally, we would like to be able to add an override rule that would allow us to do something like

currentVersion >= '4.3.2'

for example.

Hey team GrowthBook, it would be great if we could have more than 7 metrics to track with each experiment. We are running more and more "complex" experiments where we want to control for effects across the product, so 7 metrics are a bit of a limitation for us.

Minor bug found: If you copy a count metric with “converted users only” = Yes, and make it binomial, then the “converted only” property is copied without the ability to remove it. It should probably automatically be removed when changing type to binomial.

Negative Metric Values giving weird "Chance to Beat Control" & "Percent Change" Results Hey! I'm using a metric (see screenshot) that gives a negative value due to subtracting some costs. As a result of the negative value (I assume), the "chance to beat control" returns 50% and "percent change" returns 0% even though there is clearly a difference between the two results (i.e. treatment performing less negative which is good). Is this a bug or is it something on my side?

Hey guys, some quick usability feedback: When I use the search function and go to a result (e.g. on the metrics, experiments, or feature page) and I go back, the search term is removed and the page is shown in the unfiltered state again. Would be super cool to keep the search term there because sometimes I want to look at a few similar experiments and then I always have to search again

Hi - we often share screenshots of experiment results and it would be handy to include an experiment's name somewhere near the results section in the UI so that we could capture the name of the experiment in the same screenshot with the results

Hi @future-teacher-7046 and @fresh-football-47124, we have a question regarding the metrics in GrowthBook. We use metrics to track conversions, as is described in the tool. We also visualise the metric under the path “/metric/{metric id}“. We found that the way of calculating average was a bit confusing for us, when you compare the numbers here to the numbers in experiments. On the metric page the average is calculated with regard to all users that are in the conversion table, if you do not actively write a query that actively includes non-converting users. In an experiment using the same metric under the path “/experiment/{experiment id}“, however, it is calculated it with regard to all users taking part in the experiment. This explicitly includes users that never convert. Hence, the experiment’s metric is lower than in the metric view. Is it a conscious decision to not use the set of all experimenters from all experiments to calculate the graphs in the metrics view? Our five cents are that this view would be very helpful if the metric averages can be calculated for all users automatically. Especially since the metric is defined for a data source containing all users, converting or not. This would make for much simpler metric queries. It would be the same out-of-the-box usefulness GrowthBook provides at other places when it comes to programmatically combining queries in a correct and transparent way - definitely a killer feature imho 😉 FYI @adventurous-dream-15065

Hi #feedback, we are currently looking for tools to support with A/B testing at my company. Growthbook seems like a good option as it's feature flagging and analysis platform can be used separately. To begin with, I imagine we would just use the analysis platform and the feature flagging would be handled by our own devs and we would plug in our experiment data to the analysis platform. Does anyone here have experience using Growthbook in this way and would they be happy sharing their experiences with me? Thank you!