Hi, I have a Sample Ratio Mismatch I’m having trouble making sense of. We changed the experiment after a week to include android (as well as iOS) users and since then there have been fewer iOS users going into the baseline group, it’s been particularly pronounced on high volume days. For the Android users, we have consistently fewer going into Variant 1 than the other two variants. We’ve run other experiments using the same user id to split users with no problems. Ideally I’d like to know what the problem is so I can try and salvage the data we have already collected, rather than just starting again.

⭐ Our official support hours are 6:30am - 5pm Pacific Time, Monday through Friday. You may occasionally hear from us outside of these hours. Your support request has been logged in our system. Our support team will get back to you very soon!

Hi Graeme, once our west coast (US) team comes online today I can ask them about this and get back to you with an update.

Hi @brief-honey-45610 any luck?

Hi Graeme, I've asked our Data Scientist to take a look. Could you please send any relevant screenshots to help us troubleshoot?

image.png

It's very likely these results are suffering from carry-over bias, I'm happy to explain more because your example is almost text book if I'm right. One questions: When you added the Android targeting, did you re-randomize for phase 2? You should have seen some prompt like the following:

thanks @helpful-application-7107 so this can be explained by lower retention in the baseline group (for iOS) and users from phase 1 not being present in phase two? Any ideas for my android issues? I think we should do an A/A test if there’s nothing obvious.

so this can be explained by lower retention in the baseline group (for iOS) and users from phase 1 not being present in phase two?

Yeah, that's my hypothesis if you didn't re-randomize. Carry over bias would mean that you have more people retaining in treatment from phase 1, so if you don't re-randomize, you'll end up with more users from that group in your phase 2 and your treatment will be over represented. Not sure about android issues. An A/A test or two might be useful.