https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
b

bored-vase-12851

11/29/2023, 10:26 AM
Hello! Why fbadid does have an impact on variation assign? We found out users coming from fb ads are assigned to variation 1 in experiment and SRM was detected (Sample Ratio Mismatch (SRM) detected. We expected a
50.0/50.0
split, but observed a
42.7/57.3
split)
r

rhythmic-agent-34208

11/29/2023, 10:26 AM
ā­ Our official support hours are 6:30am - 5pm Pacific Time, Monday through Friday. You may occasionally hear from us outside of these hours. Your support request has been logged in our system. Our support team will get back to you very soon!
s

swift-helmet-3648

11/29/2023, 2:26 PM
Hi @bored-vase-12851 that's interesting. Are you setting your attributes using the fbaid?
b

bored-vase-12851

11/29/2023, 2:27 PM
no, we're setting attributes using mixpanel distinct id
it's a shopify integration (npm in a custom build with Vite) w/ Mixpanel. we've checked most of our data and the only issue is with fbadid parameter in URL
b

brief-honey-45610

11/29/2023, 9:06 PM
Hi Valentin, I just met with one of our Sr. Engineers about your support request. The not-so-good news is that this is right on the edge of what we're able to provide support for. There's a lot happening inside of this very specialized use case, like using Shopify, Mixpanel, FB Ads, etc. The sort-of-good news is that we can at least suggest some things to look into. Firstly, parsing out a cookie can be problematic: ā€¢ The user's client may not allow cookies ā€¢ The user's device might not support cookies How are you determining with confidence that the cookie is being picked up? Do you have fallback logic in place for if the cookie is not picked up or not present or has a
Null
value? If for some reason the cookie is surfacing as a
Null
value, all those users would be rolled into the user_id of
Null
and you would need to filter out those users so you don't show them the experiment. You would need to add something like what is shown in the
// NEW . .
code below:
Copy code
mixpanel.init('134dc40801e82a050d72d3f1871328c4', {
      debug: true,
      ignore_dnt: true,
      loaded: function() {
        growthbook.setAttributes({
          ...growthbook.getAttributes(),      
          id: getShopifyCookieValue('_shopify_y')
        })
        console.log("shopify id: " + getShopifyCookieValue('_shopify_y'));
        // console.log(growthbook)

        // NEW......
        if (!getShopifyCookieValue('_shopify_y')) {
          console.error('Invalid cookie!');
          return;
        }

. . .
A couple of other questions that came up while we were troubleshooting were: 1. What does it mean when you say "the results with the distinct_id from Mixpanel were relatively close"? 2. Was the success of the A/A test expected or unexpected in light of your use case?
b

bored-vase-12851

11/30/2023, 6:38 AM
1. What does it mean when you say "the results with the distinct_id from Mixpanel were relatively close"? I mean that the issue with fbadid users being assigned variation 1 in experiment/reports were same with both cookie and mixpanel distinct id as attribute In the attached screenshot we filter by URL (clean) contains "fbadid" that's the only area with unbalanced reports. If we filter down does not contain parameter mentioned, reports are showing balanced traffic with smaller, better performance on rebrand variant (as expected) however we're hesitant to trust 100% the data in this scenario
As for the AA question, yes we hoped to be successful šŸ˜Š we have balanced traffic 20k users in each variation after running the AA experiment for about 7 days
The thing is, on variation 1 we do a redirect, could that be the cause of the issue? Distinct id and cookie does not change on redirect... And conversion windows are correctly set
r

rhythmic-agent-34208

11/30/2023, 3:20 PM
Hi Valentin, did you see my comments earlier in this thread about the problems with using cookies? Could you please address that since it's potentially causing problems.
As for this part:
The thing is, on variation 1 we do a redirect, could that be the cause of the issue? Distinct id and cookie does not change on redirect... And conversion windows are correctly set
I will check with the team and get back to you soon.
b

bored-vase-12851

11/30/2023, 3:25 PM
Let me text here what we discussed. There are two hypotheses why the old version of the site gets more traffic: Cookies with IDs that are transmitted to GB are not available in some cases. Solution add Sentry logging to confirm the hypotheses generate our own id test saving the id in local storage Mixpanel Distinct id changes during a session (except cases when we assign out internal client id - purchase) Solution add Sentry logging o confirm the hypotheses āœ… We'll run it for more days and I'll keep you posted
r

rhythmic-agent-34208

11/30/2023, 3:59 PM
Sounds like a good plan, thanks Valentin šŸ™‚
5 Views