https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
b

bumpy-finland-5739

02/16/2024, 5:53 PM
Hello, I seem to have a sample ratio mismatch in one of our recent tests. We have run multiple tests both using the JS SDK (visual editor) and making codebased changes using PHP SDK. Previous tests have run as expected with an even split and even current tests are running as expected apart from this one. The test has been running for nearly a month and has never evened out. Is there a way for us to identify why this particular test has an issue? We also have no multiple exposure warnings just to rule that out. Thank you
f

fresh-football-47124

02/16/2024, 6:37 PM
SRM errors can be caused by a few things: 1. Did you change the split at all during the running of the experiment? 2. Do you have any caching? Sometimes caching can cause an increase in one variant
b

bumpy-finland-5739

02/16/2024, 7:24 PM
Thanks Graham, we haven’t changed the split at any point. We do have multiple caching layers so it could be this. For this test and future tests is there anything we can do to point more towards the variant with significantly less without having to change the split?
f

fresh-football-47124

02/16/2024, 7:27 PM
no, you shouldn't need to (nor should you) adjust the split to make the traffic 'more equal'
the assignment is random, and if the split percentage is off, then there is likely a bug- hence the SRM warning
b

bumpy-finland-5739

02/16/2024, 7:32 PM
Ok thanks for clarifying. What would be the best way of trying to debug the issue? Like I say we have run multiple tests through growthbook and this is the first of its kind we’ve experienced. Worth mentioning this is a redirect test.
f

fresh-football-47124

02/16/2024, 7:38 PM
you can try to find a user who was assigned A but should have been assigned B
that might take some scripting
b

bumpy-finland-5739

02/16/2024, 7:41 PM
Ok and how would I know which variant the user was meant to be assigned if it’s a random allocation?
f

fresh-football-47124

02/16/2024, 7:41 PM
we used deterministic hashing
so if you enter the same experiment name, assignment attribute (user id typically), you'll get the same result
b

bumpy-finland-5739

02/16/2024, 7:53 PM
Ok thank you we will investigate and see if we can find any evidence. Thank you
12 Views