https://www.growthbook.io/
Join Slack
Hi all, we did not have an SRM warning show up on our experiment dashboard when there was a detectab...
# announcements
w
Hi all, we did not have an SRM warning show up on our experiment dashboard when there was a detectable mismatch in our data:
Copy code
VARIATION_ID	USERS
    cell1	        439,863
    cell2	        442,411
    cell3	        442,951
    cell5	        442,880
Any idea why the SRM wasn’t caught?
For context, it’s being flagged now with a warning, but when the SRM first showed up last week, there was no warning for quite a few days
cc @ambitious-apartment-58735
h
When is the data in your first message from?
When the error appeared, or from the period when there was no warning?
We have a pretty conservative threshold for SRM, which may explain what's going on here: https://docs.growthbook.io/app/experiments#sample-ratio-mismatch-srm
w
when there was no warning
h
yeah if I'm doing my chi-square tests right the SRM p-value there is 0.002, and our threshold is 0.001 😕
w
ah yep i’m getting the same thing
h
we can definitely, at some point, let orgs set their own threshold to match their own risk tolerance
w
we typically use .01 for SRMs so this was a flag for us, but good to know why it wasn’t showing up in GrowthBook
h
we've optimized for a more conservative threshold for now, but eventually can consider letting it be set by customers + sequential methods
👍 1
75 Views
Open in Slack
PreviousNext
Powered by

GrowthBook Users

Open source platform for stress free deployments, measured impact, and smarter decisions.

Powered by