Hi all, we did not have an SRM warning show up on ...
# announcements
Hi all, we did not have an SRM warning show up on our experiment dashboard when there was a detectable mismatch in our data:
Copy code
    cell1	        439,863
    cell2	        442,411
    cell3	        442,951
    cell5	        442,880
Any idea why the SRM wasn’t caught?
For context, it’s being flagged now with a warning, but when the SRM first showed up last week, there was no warning for quite a few days
cc @ambitious-apartment-58735
When is the data in your first message from?
When the error appeared, or from the period when there was no warning?
We have a pretty conservative threshold for SRM, which may explain what's going on here: https://docs.growthbook.io/app/experiments#sample-ratio-mismatch-srm
when there was no warning
yeah if I'm doing my chi-square tests right the SRM p-value there is 0.002, and our threshold is 0.001 😕
ah yep i’m getting the same thing
we can definitely, at some point, let orgs set their own threshold to match their own risk tolerance
we typically use .01 for SRMs so this was a flag for us, but good to know why it wasn’t showing up in GrowthBook
we've optimized for a more conservative threshold for now, but eventually can consider letting it be set by customers + sequential methods
👍 1