https://www.growthbook.io/ logo
w

wide-jordan-14758

03/28/2023, 10:39 PM
Hi all, we did not have an SRM warning show up on our experiment dashboard when there was a detectable mismatch in our data:
Copy code
VARIATION_ID	USERS
    cell1	        439,863
    cell2	        442,411
    cell3	        442,951
    cell5	        442,880
Any idea why the SRM wasn’t caught?
For context, it’s being flagged now with a warning, but when the SRM first showed up last week, there was no warning for quite a few days
cc @ambitious-apartment-58735
h

helpful-application-7107

03/28/2023, 10:49 PM
When is the data in your first message from?
When the error appeared, or from the period when there was no warning?
We have a pretty conservative threshold for SRM, which may explain what's going on here: https://docs.growthbook.io/app/experiments#sample-ratio-mismatch-srm
w

wide-jordan-14758

03/28/2023, 11:14 PM
when there was no warning
h

helpful-application-7107

03/28/2023, 11:15 PM
yeah if I'm doing my chi-square tests right the SRM p-value there is 0.002, and our threshold is 0.001 😕
w

wide-jordan-14758

03/28/2023, 11:15 PM
ah yep i’m getting the same thing
h

helpful-application-7107

03/28/2023, 11:15 PM
we can definitely, at some point, let orgs set their own threshold to match their own risk tolerance
w

wide-jordan-14758

03/28/2023, 11:15 PM
we typically use .01 for SRMs so this was a flag for us, but good to know why it wasn’t showing up in GrowthBook
h

helpful-application-7107

03/28/2023, 11:16 PM
we've optimized for a more conservative threshold for now, but eventually can consider letting it be set by customers + sequential methods
👍 1