https://www.growthbook.io/ logo
Powered by
b

busy-horse-73824

09/09/2022, 4:10 PM
👋 We've been getting SRM errors, but really struggling to understand why or what to do about it
message has been deleted
They're pretty simple tests, and set up in a fairly simple way
Test runs on: https://mojo.so/landing/default If the user is in the test group, it redirects them immediately to https://mojo.so/tests/onboarding/right-left
With some minor simplification, this is the entirety of the relevant JS running on the page (it's also pretty easy to read in the <head> of the above two links)
Untitled.html
One thing which comes to mind is - potentially it's redirecting so quickly that the 
experiment_triggered
event somehow is getting lost But it's all within the same hostname, and Mixpanel normally handles that fine (and if it didn't, frankly I'd expect a much more skewed result than e.g. 47%/53%)
f

fresh-football-47124

09/09/2022, 4:21 PM
is the redirected group the lower trafficed test?
b

busy-horse-73824

09/09/2022, 4:57 PM
Yeah, typically around 43-48% of the traffic, where it should be around 50%
Typically we'd be talking about 3000+ users overall
Any thoughts on this one? It's a little hard to know what we should do about it
f

fresh-football-47124

09/13/2022, 5:54 PM
@future-teacher-7046?
❤️ 1
b

busy-horse-73824

09/13/2022, 5:54 PM
or even how we should think about investigating/debugging the issue tbh
f

future-teacher-7046

09/13/2022, 5:57 PM
SRM usually happens when the id you use to split traffic is different from the id you use to analyze results. For analysis, if you are using Mixpanel, we use their built-in 
distinct_id
for analysis. What is 
window.MOJO_SESSION_CONTEXT.device.vendor_id
set to? Is it something custom that you created?
For example, if you split traffic by company id, you might end up with an even split of 100 companies for each variations. But if one of those companies had a huge number of employees (distinct_ids), you would see a sample ratio mismatch during analysis
f

fresh-football-47124

09/13/2022, 6:25 PM
Happy to meet and help you debug it if you like
b

busy-horse-73824

09/14/2022, 10:19 AM
hmm, the 
vendor_id
is very basic - it's just a 16-byte random hex string (from 
crypto.getRandomValues
), which is stored as a domain-wide cookie - so it should be pretty perfectly distributed All our other tests & non-redirect tests seem to be doing fine, it's just a particular redirect one we're struggling with
code for the session context, in case it's useful
Untitled.txt
I guess there could be some issue with the fact that the page redirect will happen pretty quickly? But then the feature should still be evaluated again on the destination page
May book in a time to look at it if we can't figure it out and there's no obvious areas you guys can point to
Powered by