Hi. I`ve got a problem with the test:

Sample Ratio Mismatch (SRM) detected. P-value below 0.001

Sounds like a bug. I went through the dimensions, and it seems the most crucial point here is traffic with referral = t.co from the UK Previously, there were no issues, and now I have no idea what I can do and how to fix this problem. Can I exclude somehow this anomaly from the existing test?

Your dimensions can't always reveal where the problem is coming from and if some dimensions are too small they won't show the issue at all. Normally the issue comes up in your SDK implementation where for some reason you aren't tracking experiment exposures right after looking up feature values so some users in one test variant are being bucketed more than others.

So, should I check SDK installation on the website in terms of bucketing?

Yes, that's what I'm suggesting. Sometimes SRM warnings are false positives, though, and if this issue has never happened on other experiments with similar set ups, and the difference in the buckets isn't that much from what you expect, then maybe it is a false positive. You can exclude a group from the test using the Custom SQL filter in the analysis settings for your experiment, but I don't think that will solve the problem if there truly is one.

Ok, we`ll check it. How to exclude them from the test?

Hi Volodymyr, are you specifically wanting to exclude users ​*from the experiment *​who are in the UK and who were also referred by t.co? If so, you can use a Targeting Condition to exclude them from the experiment. If you want to exclude those users ​*from your analysis*​, Luke's suggestion to use the Custom SQL filter will do that for you.

Got it. Appreciate your help!

You're very welcome! Let us know if you have any other questions. 🙂