Hi GrowthBook team, I'm able to see that my feature is returning "true" in the chrome extension but returns false for -- growthbook.isOn()

Is there an API endpoint for adding a new idea in the Ideas section? People fill out a Google form at our company when they want to propose a new experiment. We ask a number of specific questions relevant to our product. Form submissions get sent to Slack, but I’d love to record them as an Idea and then convert the idea to an experiment after I’ve reviewed the experiment design. The structure of Google forms makes it easier to collect what we need (e.g. we can require fields), but if y’all supported us creating / requiring our own fields on input in the Idea UI too, that would work great.

not yet, it's on our list

both the API and the custom form fields for ideas/experiments

Do y’all generate webhook events when an experiment reaches significance for a specific metric?

we send an email

but no webhook for that atm

should be easy to add

though we find that often calling an experiment isn't exactly black and white

but if you can set up the conditions by which you want the test turned on/off, we could support that

awesome, yes I would definitely be willing and want to define those conditions

Hi, is there any way to set single source of data (only BQ) and provide BQ JSON key file and ProjectId in backend env.

I am trying to use the chrome extension to override the variation but I am not getting the updated value and cannot figure why the exact reason, anyone had a similar issue?

About using mixpanel as a data source: • in there we have different projects for our environments, so in this case I need to add several data sources (one per project)? • does the user name needs to be like my own or is it an user name for growthbook? • the secreat I imagine that it is the api secret, right? The user id in mixpanel, currently so we are able to track the same user in our server events we start using the mixpanel distinct id but as soon as he signs up we identify the user in mixpanel with our own id and we will have basically the same approach for tests. Then how would I set this up in growthbook? I could not find a reference on how to do this

It looks like the extension override only applies the render method in the moment I change something, but if I have already changed and reload the page the value is there but the override is not triggered, is that correct?

Hi for using bigquery using env variable do we need to give OAUTH permission to the service account?

Do Read-Only team members count against the user limit?

Hello again 🙂 I am now trying to know what would be the correct approach to do testing for the backend part, what I want to do is to change the variation of an user so I assume I need to add an override rule. I saw that there is 3 types, but I do not know for sure which one I should use between force value and ab experiment. I think both would work, just not sure how to setup exactly. Is there an example I could use?

I am having an error trying to retrieve data from mixpanel for the experiment, is this a bug?

Uncaught exception TypeError: reduce(), groupBy(), and groupByUser() accept at most 8 reducers.
  .groupBy(["value.variation"], [
   ^

Stack trace:
TypeError: reduce(), groupBy(), and groupByUser() accept at most 8 reducers.
    at main (<anonymous>:178:4)

Question about growthbiook extension. I see after installing. the feature data and attributes are exposed .?Would it happen on production that data seen by anyone who installs the growthbook extension

I’m loving Growthbook so far. We have our own framework for assigning users to experiments, so the ability to define the SQL tied to those assignments is great. We were looking for a flexible experiment reporting tool so that we didn’t have to keep running the stats manually, so Growthbook is exactly what we needed!

Now that we’ve setup our first experiment and metrics in Growthbook, I had some basic questions: some specific to Growthbook, others related to the design and reporting of experiments. Since we’re new to experimentation, we’re looking for best practices, so I’m hoping the community has some advice

At Pipedream, one core metric we want to monitor is the number of workflows (automations) that users create. I want to make sure I’m correctly defining this metric. Since we have a record of all workflows created, I’m selecting the

user_id

, the

timestamp

of creation, and a value of

1

, since creating a workflow should increment the measure:

SELECT
  user_id,
  created_at AS timestamp,
  1 AS value
FROM workflows

Is that correct? Separately we’re tracking a binomial metric on the users that reach the state of “created at least one workflow”, but since the goal of some experiments is to move the total count of workflows, and the goal of others is to move the rate of users that reach the state of creating at least one workflow, we thought both metrics were appropriate.

I read the docs on setting capped values, but I’d love to know how people approach setting a cap. Are you reviewing the distribution of values per user and setting based on the 95/99% value, for example? Many of our metrics are skewed, with most values low and a long power law tail to the right.

I’m running a drip campaign experiment now: 50% of users get a simple welcome email, 50% of users get the welcome email + a new drip campaign that sends them 4 additional emails spaced out over a couple of weeks. Our hypothesis is that users in the experiment group will see a higher rate of success on a few metrics related to our funnel. I’m having trouble defining the conversion delay and window here. Since users in the exposed group get 4 emails over many days, they could convert after the first email, or the fourth. My intuition tells me I should set a conversion delay of 2 days after exposure (when the first email in the drip campaign is sent), and a window of 14 days (a few days after the last email is sent). If the user converts from any email, the experiment is a success. We’d then modify specific emails in future experiments to tease out the effects of one email over the other. Does that logic sound right, or would you approach it differently? The challenge I’m having is that I want to define this delay / window only on the metrics for this experiment. The delay / window for other experiments would be different. I can only see a way to define this logic in the metric itself, and not for a (metric, experiment). How do people handle this?

I have a lot of questions around when the right time to call an experiment is in the Bayesian world. We’ve been assigning users to an A/A test for a few months now, 50/50 at random. I’ve connected the experiment in Growthbook and am reviewing all my metrics. It’s great to see the small differences in effect size here, and my intuition is to use this knowledge of natural variance in future experiments. e.g. if the A/A test shows a difference of 1% for some core metric, 1% change in future experiments is probably just noise, correct? I’m just trying to assess whether the data I get from the A/A experiment is useful in that manner

Beyond that, what heuristics are people using to determine when to call experiments? The measures of confidence and risk are useful and intuitive, but I’m foreseeing a lot of experiments that end up in the “no effect” bucket, e.g. with 40-60% Chance to Beat Control a few weeks out. In these cases, is the right move to wait for more data, or should I call the experiment and trust that there’s probably no effect? I know it’s probably hard to enumerate rules here, but I’m curious on people’s general approach!

Final questions that I had today: I’m seeing a lot of discussion of Type I rate in the docs, e.g. when I define dimensions. 1. Is the long-term Type I error rate in a Bayesian world fixed like in a frequentist world? If so, what is the rate in Growthbook? 2. Does Growthbook correct for the Type I error rate when I split data by dimensions?

Quick question, lets say I am running an growthbook instance in the server, and I go and change the traffic split for the test. How would this instance know that the traffic split changed? Does the library gets updated automatically for this cases?

👋 Hi everyone!