Hello! Why fbadid does have an impact on variation assign? We found out users coming from fb ads are assigned to variation 1 in experiment and SRM was detected (Sample Ratio Mismatch (SRM) detected. We expected a

50.0/50.0

split, but observed a

42.7/57.3

split)

⭐ Our official support hours are 6:30am - 5pm Pacific Time, Monday through Friday. You may occasionally hear from us outside of these hours. Your support request has been logged in our system. Our support team will get back to you very soon!

Hi @bored-vase-12851 that's interesting. Are you setting your attributes using the fbaid?

no, we're setting attributes using mixpanel distinct id

Hi Valentin, I just met with one of our Sr. Engineers about your support request. The not-so-good news is that this is right on the edge of what we're able to provide support for. There's a lot happening inside of this very specialized use case, like using Shopify, Mixpanel, FB Ads, etc. The sort-of-good news is that we can at least suggest some things to look into. Firstly, parsing out a cookie can be problematic: • The user's client may not allow cookies • The user's device might not support cookies How are you determining with confidence that the cookie is being picked up? Do you have fallback logic in place for if the cookie is not picked up or not present or has a

Null

value? If for some reason the cookie is surfacing as a

Null

value, all those users would be rolled into the user_id of

Null

and you would need to filter out those users so you don't show them the experiment. You would need to add something like what is shown in the

// NEW . .

code below:

mixpanel.init('134dc40801e82a050d72d3f1871328c4', {
      debug: true,
      ignore_dnt: true,
      loaded: function() {
        growthbook.setAttributes({
          ...growthbook.getAttributes(),      
          id: getShopifyCookieValue('_shopify_y')
        })
        console.log("shopify id: " + getShopifyCookieValue('_shopify_y'));
        // console.log(growthbook)

        // NEW......
        if (!getShopifyCookieValue('_shopify_y')) {
          console.error('Invalid cookie!');
          return;
        }

. . .

1. What does it mean when you say "the results with the distinct_id from Mixpanel were relatively close"? I mean that the issue with fbadid users being assigned variation 1 in experiment/reports were same with both cookie and mixpanel distinct id as attribute In the attached screenshot we filter by URL (clean) contains "fbadid" that's the only area with unbalanced reports. If we filter down does not contain parameter mentioned, reports are showing balanced traffic with smaller, better performance on rebrand variant (as expected) however we're hesitant to trust 100% the data in this scenario

Hi Valentin, did you see my comments earlier in this thread about the problems with using cookies? Could you please address that since it's potentially causing problems.

Let me text here what we discussed. There are two hypotheses why the old version of the site gets more traffic: Cookies with IDs that are transmitted to GB are not available in some cases. Solution add Sentry logging to confirm the hypotheses generate our own id test saving the id in local storage Mixpanel Distinct id changes during a session (except cases when we assign out internal client id - purchase) Solution add Sentry logging o confirm the hypotheses ✅ We'll run it for more days and I'll keep you posted

Sounds like a good plan, thanks Valentin 🙂