Hello there, I'm experiencing a discrepancy issue with an experiment in my Nuxt application and could use some guidance. Experiment Setup: • Type: Feature flag • Data Source: Mixpanel • Flow: We have a multi-step application form; the experiment begins at the final step and concludes with a submit event. • Assignment Attribute:

application_uuid

(generated at the 2nd step, prior to the experiment start) • Implementation Details: The

application_uuid

is created on the 2nd step, and I invoke the

setAttributes

function immediately to incorporate this UUID. The code snippet used is as follows:

await this.$growthbook.loadFeatures();
this.$growthbook.setAttributes({
  ...this.$growthbook.getAttributes(),
  application_uuid: data.data.uuid,
});

For users who revisit,

setAttributes

is triggered on the

mounted

lifecycle hook. Issue: When I filter for "Experiment Started" events in Mixpanel, which are fired in the

trackingCallback

, it shows the actual number of users participating in the experiment. However, on the GrowthBook experiments page, the number of users is significantly lower. Could there be a step I'm missing that would explain this discrepancy? Has anyone else encountered a similar issue or can point me in the direction of what might be causing this? Thank you in advance for your assistance!

@white-architect-17617 has opened an issue Close Issue button

Hi - there are typically two reasons for this

Thank you for the quick response! Can you please provide more details about the date range and query? Could you tell me if you’re referring to the conversion window? I didn’t understand what you meant by “query”.

Hi Akin, thanks for providing so much detail in your message earlier! I'll double check with the team tomorrow about your most recent questions and let you know what I find out.

I appreciate your response. Our goal metrics will be measured through an event in Mixpanel triggered by a call-to-action (CTA) click in our email. The CTA click may not happen within the default conversion window of 72 hours, as our email campaign timeline is 7 days. Do you think this could be the reason for any discrepancies in the data? The query of the experiment :

function main() {
  // Helper aggregation functions
  const sum = (arr) => arr.reduce((sum, x) => sum + x, 0);
  const count = (arr) => arr.length;
  const countDistinct = (arr) => new Set(arr).size;
  const min = (arr) => Math.min(...arr);
  const max = (arr) => Math.max(...arr);
  const avg = (arr) => count(arr)>0?(sum(arr)/count(arr)):0;
  const percentile = (arr, p) => {
    const s = [...arr].sort((a,b)=>a-b);
    if(!s.length) return 0;
    if(p<=0) return s[0];
    if(p>=100) return s[arr.length-1];
    const r = (s.length-1)*p/100;
    const rf = Math.ceil(r) - r;
    return s[Math.floor(r)]*rf + s[Math.ceil(r)]*(1-rf);
  };
  const median = (arr) => percentile(arr, 50);
  // Experiment exposure event
  function isExposureEvent(event) {
    return event.name === "Experiment Started" && event.properties["Experiment Id"] === "technical_assessment_first" && event.time >= 1698173940000 && event.time <= 1700120128322;
  }
  // Application Submitted as FT
  function isActivationMetric(event) {
    return event.name === "Submit application" && event.properties["bookMeeting"]+'' == "true" && event.properties["bookCallOnly"]+'' == "false";
  }
  // Fast-track Flow Completed
  function isMetric0(event) {
    return event.name === "Application Step Completed" && event.properties["step_name"]+'' == "Fast-track Flow Completed";
  }
  return Events({
    "from_date": "2023-10-24",
    "to_date": "2023-11-16",
    "event_selectors": [
      {
        "event": "Application Step Completed"
      },
      {
        "event": "Submit application"
      },
      {
        "event": "Experiment Started"
      }
    ]
  })
  .filter(function(event) {
    // Experiment exposure event
    if(isExposureEvent(event)) return true;
    // Application Submitted as FT
    if(isActivationMetric(event)) return true;
    // Fast-track Flow Completed
    if(isMetric0(event)) return true;
    // Otherwise, ignore the event
    return false;
  })
  // Array of metric values for each user
  .groupByUser(function(state, events) {
    state = state || {
      inExperiment: false,
      multipleVariants: false,
      activated: false,
      start: null,
      variation: null,
      m0: [], 
    };
    for(var i=0; i<events.length; i++) {
      const event = events[i];
      // User is put into the experiment
      if(isExposureEvent(event)) {
        if(!state.inExperiment) {
          state.inExperiment = true;
          state.variation = event.properties["Variant Id"];
          continue;
        }
        else if(state.variation !== event.properties["Variant Id"]) {
          state.multipleVariants = true;
          continue;
        }
        else {
          continue;
        }
      }
      // Not in the experiment yet
      if(!state.inExperiment) {
        continue;
      }
      // Saw multiple variants so ignore
      if(state.multipleVariants) {
        continue;
      }
      // Not activated yet
      if(!state.activated) {
        // Does this event activate it? (Metric - Application Submitted as FT)
        if(isActivationMetric(event)) {
          state.activated = true;
          state.start = event.time;
        }
        else {
          continue;
        }
      }
      // Metric - Fast-track Flow Completed
      if(isMetric0(event) && event.time - state.start < 259200000) {
        state.m0.push(1);
      }
    }
    return state;
  })
  // Remove users that are not in the experiment
  .filter(function(ev) {
    if(!ev.value.inExperiment) return false;
    if(ev.value.multipleVariants) return false;
    if(ev.value.variation === null || ev.value.variation === undefined) return false;
    if(!ev.value.activated) return false;
    return true;
  })
  // Aggregate the metric value arrays for each user
  .map(function(user) {
    // Metric - Fast-track Flow Completed
    user.value.m0 = user.value.m0.length ? 1 : 0;
    return user;
  })
  // One group per experiment variation with summary data
  .groupBy(["value.variation"], [
    // Total users in the group
    mixpanel.reducer.count(),
    // Metric - Fast-track Flow Completed
    mixpanel.reducer.numeric_summary('value.m0'),
  ])
  // Convert to an object that's easier to work with
  .map(row => {
    const ret = {
      variation: row.key[0],
      dimension: '',
      users: row.value[0],
      metrics: [],
    };
    const metricIds = [
      // Fast-track Flow Completed
      "met_405opf1mlncw1a93"
    ];
    const metricTypes = [
      "binomial"
    ];
    for(let i=1; i<row.value.length; i++) {
      ret.metrics.push({
        id: metricIds[i-1],
        metric_type: metricTypes[i-1],
        count: row.value[i].count,
        main_sum: row.value[i].sum,
        main_sum_squares: row.value[i].sum_squares,
      });
    }
    return ret;
  });
}

It looks like you're using an activation metric in growthbook, which will filter out users to only those who have triggered the activation metric.

Yes, we use it to indicate users who are only related to the experiment.

Right, so if you remove that, do the numbers line up with Mixpanel?