https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
w

white-architect-17617

11/08/2023, 7:16 AM
Hello there, I'm experiencing a discrepancy issue with an experiment in my Nuxt application and could use some guidance. Experiment Setup: • Type: Feature flag • Data Source: Mixpanel • Flow: We have a multi-step application form; the experiment begins at the final step and concludes with a submit event. • Assignment Attribute:
application_uuid
(generated at the 2nd step, prior to the experiment start) • Implementation Details: The
application_uuid
is created on the 2nd step, and I invoke the
setAttributes
function immediately to incorporate this UUID. The code snippet used is as follows:
Copy code
await this.$growthbook.loadFeatures();
this.$growthbook.setAttributes({
  ...this.$growthbook.getAttributes(),
  application_uuid: data.data.uuid,
});
For users who revisit,
setAttributes
is triggered on the
mounted
lifecycle hook. Issue: When I filter for "Experiment Started" events in Mixpanel, which are fired in the
trackingCallback
, it shows the actual number of users participating in the experiment. However, on the GrowthBook experiments page, the number of users is significantly lower. Could there be a step I'm missing that would explain this discrepancy? Has anyone else encountered a similar issue or can point me in the direction of what might be causing this? Thank you in advance for your assistance!
r

rhythmic-agent-34208

11/08/2023, 7:16 AM
@white-architect-17617 has opened an issue Close Issue button
Our official support hours are 6:30am - 5pm Pacific Time, Monday through Friday. You may occasionally hear from us outside of these hours. Your support request has been logged in our system. Our support team will get back to you very soon!
f

fresh-football-47124

11/08/2023, 9:06 AM
Hi - there are typically two reasons for this
one is that the date range of the experiment is not correct
two is that the query does a unique on the users
w

white-architect-17617

11/08/2023, 10:02 AM
Thank you for the quick response! Can you please provide more details about the date range and query? Could you tell me if you’re referring to the conversion window? I didn’t understand what you meant by “query”.
r

rhythmic-agent-34208

11/09/2023, 1:32 AM
Hi Akin, thanks for providing so much detail in your message earlier! I'll double check with the team tomorrow about your most recent questions and let you know what I find out.
Hi Akin, The date range in the SQL query could be incorrect or not accounting for the conversion window, and what we mean by query is how you're telling GrowthBook to pull the data in order to create the metrics: https://docs.growthbook.io/app/metrics#query-settings If you can send us a screenshot of the query or copy-paste it that would be helpful. To access the query that is running, go to Metrics --> select the desired metric --> scroll to the bottom of the page --> click on the red "View Queries" button
w

white-architect-17617

11/16/2023, 1:01 PM
I appreciate your response. Our goal metrics will be measured through an event in Mixpanel triggered by a call-to-action (CTA) click in our email. The CTA click may not happen within the default conversion window of 72 hours, as our email campaign timeline is 7 days. Do you think this could be the reason for any discrepancies in the data? The query of the experiment :
Copy code
function main() {
  // Helper aggregation functions
  const sum = (arr) => arr.reduce((sum, x) => sum + x, 0);
  const count = (arr) => arr.length;
  const countDistinct = (arr) => new Set(arr).size;
  const min = (arr) => Math.min(...arr);
  const max = (arr) => Math.max(...arr);
  const avg = (arr) => count(arr)>0?(sum(arr)/count(arr)):0;
  const percentile = (arr, p) => {
    const s = [...arr].sort((a,b)=>a-b);
    if(!s.length) return 0;
    if(p<=0) return s[0];
    if(p>=100) return s[arr.length-1];
    const r = (s.length-1)*p/100;
    const rf = Math.ceil(r) - r;
    return s[Math.floor(r)]*rf + s[Math.ceil(r)]*(1-rf);
  };
  const median = (arr) => percentile(arr, 50);
  // Experiment exposure event
  function isExposureEvent(event) {
    return event.name === "Experiment Started" && event.properties["Experiment Id"] === "technical_assessment_first" && event.time >= 1698173940000 && event.time <= 1700120128322;
  }
  // Application Submitted as FT
  function isActivationMetric(event) {
    return event.name === "Submit application" && event.properties["bookMeeting"]+'' == "true" && event.properties["bookCallOnly"]+'' == "false";
  }
  // Fast-track Flow Completed
  function isMetric0(event) {
    return event.name === "Application Step Completed" && event.properties["step_name"]+'' == "Fast-track Flow Completed";
  }
  return Events({
    "from_date": "2023-10-24",
    "to_date": "2023-11-16",
    "event_selectors": [
      {
        "event": "Application Step Completed"
      },
      {
        "event": "Submit application"
      },
      {
        "event": "Experiment Started"
      }
    ]
  })
  .filter(function(event) {
    // Experiment exposure event
    if(isExposureEvent(event)) return true;
    // Application Submitted as FT
    if(isActivationMetric(event)) return true;
    // Fast-track Flow Completed
    if(isMetric0(event)) return true;
    // Otherwise, ignore the event
    return false;
  })
  // Array of metric values for each user
  .groupByUser(function(state, events) {
    state = state || {
      inExperiment: false,
      multipleVariants: false,
      activated: false,
      start: null,
      variation: null,
      m0: [], 
    };
    for(var i=0; i<events.length; i++) {
      const event = events[i];
      // User is put into the experiment
      if(isExposureEvent(event)) {
        if(!state.inExperiment) {
          state.inExperiment = true;
          state.variation = event.properties["Variant Id"];
          continue;
        }
        else if(state.variation !== event.properties["Variant Id"]) {
          state.multipleVariants = true;
          continue;
        }
        else {
          continue;
        }
      }
      // Not in the experiment yet
      if(!state.inExperiment) {
        continue;
      }
      // Saw multiple variants so ignore
      if(state.multipleVariants) {
        continue;
      }
      // Not activated yet
      if(!state.activated) {
        // Does this event activate it? (Metric - Application Submitted as FT)
        if(isActivationMetric(event)) {
          state.activated = true;
          state.start = event.time;
        }
        else {
          continue;
        }
      }
      // Metric - Fast-track Flow Completed
      if(isMetric0(event) && event.time - state.start < 259200000) {
        state.m0.push(1);
      }
    }
    return state;
  })
  // Remove users that are not in the experiment
  .filter(function(ev) {
    if(!ev.value.inExperiment) return false;
    if(ev.value.multipleVariants) return false;
    if(ev.value.variation === null || ev.value.variation === undefined) return false;
    if(!ev.value.activated) return false;
    return true;
  })
  // Aggregate the metric value arrays for each user
  .map(function(user) {
    // Metric - Fast-track Flow Completed
    user.value.m0 = user.value.m0.length ? 1 : 0;
    return user;
  })
  // One group per experiment variation with summary data
  .groupBy(["value.variation"], [
    // Total users in the group
    mixpanel.reducer.count(),
    // Metric - Fast-track Flow Completed
    mixpanel.reducer.numeric_summary('value.m0'),
  ])
  // Convert to an object that's easier to work with
  .map(row => {
    const ret = {
      variation: row.key[0],
      dimension: '',
      users: row.value[0],
      metrics: [],
    };
    const metricIds = [
      // Fast-track Flow Completed
      "met_405opf1mlncw1a93"
    ];
    const metricTypes = [
      "binomial"
    ];
    for(let i=1; i<row.value.length; i++) {
      ret.metrics.push({
        id: metricIds[i-1],
        metric_type: metricTypes[i-1],
        count: row.value[i].count,
        main_sum: row.value[i].sum,
        main_sum_squares: row.value[i].sum_squares,
      });
    }
    return ret;
  });
}
r

rhythmic-agent-34208

11/16/2023, 8:32 PM
It looks like you're using an activation metric in growthbook, which will filter out users to only those who have triggered the activation metric.
w

white-architect-17617

11/17/2023, 4:49 AM
Yes, we use it to indicate users who are only related to the experiment.
r

rhythmic-agent-34208

11/20/2023, 5:40 PM
Right, so if you remove that, do the numbers line up with Mixpanel?
18 Views