Hi, I have a Sample Ratio Mismatch I’m having trou...
# ask-questionsq
quiet-pharmacist-40710
11/15/2023, 11:48 AMHi, I have a Sample Ratio Mismatch I’m having trouble making sense of. We changed the experiment after a week to include android (as well as iOS) users and since then there have been fewer iOS users going into the baseline group, it’s been particularly pronounced on high volume days. For the Android users, we have consistently fewer going into Variant 1 than the other two variants. We’ve run other experiments using the same user id to split users with no problems. Ideally I’d like to know what the problem is so I can try and salvage the data we have already collected, rather than just starting again.
r
rhythmic-agent-34208
11/15/2023, 11:48 AM⭐ Our official support hours are 6:30am - 5pm Pacific Time, Monday through Friday. You may occasionally hear from us outside of these hours.
Your support request has been logged in our system. Our support team will get back to you very soon!
b
brief-honey-45610
11/15/2023, 3:16 PMHi Graeme, once our west coast (US) team comes online today I can ask them about this and get back to you with an update.
👍 1
q
quiet-pharmacist-40710
11/16/2023, 10:15 AMHi @brief-honey-45610 any luck?
r
rhythmic-agent-34208
11/16/2023, 6:31 PMHi Graeme, I've asked our Data Scientist to take a look. Could you please send any relevant screenshots to help us troubleshoot?
q
quiet-pharmacist-40710
11/16/2023, 6:47 PMimage.png
quiet-pharmacist-40710
11/16/2023, 6:48 PMimage.png
quiet-pharmacist-40710
11/16/2023, 6:50 PMIt does not make sense for either variant to reduce conversion, both reduce the amount of background processing. This was the initial phase:
quiet-pharmacist-40710
11/16/2023, 6:50 PMimage.png
quiet-pharmacist-40710
11/16/2023, 6:52 PMdate cohorts:
quiet-pharmacist-40710
11/16/2023, 6:52 PMimage.png
quiet-pharmacist-40710
11/16/2023, 7:01 PMI have found another experiment (android only) which also has a mismatch:
quiet-pharmacist-40710
11/16/2023, 7:07 PMOur other tests for iOS are not showing the warning, but do have a slight bias toward assigning to the control group (e.g. 49.8/51.2 over ~40k samples).
r
rhythmic-agent-34208
11/16/2023, 8:27 PMIt's very likely these results are suffering from carry-over bias, I'm happy to explain more because your example is almost text book if I'm right. One questions:
When you added the Android targeting, did you re-randomize for phase 2? You should have seen some prompt like the following:
rhythmic-agent-34208
11/16/2023, 8:39 PMIn short, it looks like you didn't re-randomize. Adding more users to your experiment is actually ok to do mid-experiment, if a bit weird. If my guess is right, you should be able to change your phase to extend all the way back to the beginning of the experiment (beginning of phase 1) and at least your unexpected iOS results and iOS SRM should go away.
q
quiet-pharmacist-40710
11/17/2023, 10:17 AMthanks @helpful-application-7107 so this can be explained by lower retention in the baseline group (for iOS) and users from phase 1 not being present in phase two? Any ideas for my android issues? I think we should do an A/A test if there’s nothing obvious.
h
helpful-application-7107
11/18/2023, 12:27 AMso this can be explained by lower retention in the baseline group (for iOS) and users from phase 1 not being present in phase two?Yeah, that's my hypothesis if you didn't re-randomize. Carry over bias would mean that you have more people retaining in treatment from phase 1, so if you don't re-randomize, you'll end up with more users from that group in your phase 2 and your treatment will be over represented. Not sure about android issues. An A/A test or two might be useful.
👍 1
25 Views