Hi all, we did not have an SRM warning show up on ...
# announcementsw
wide-jordan-14758
03/28/2023, 10:39 PMHi all, we did not have an SRM warning show up on our experiment dashboard when there was a detectable mismatch in our data:
Copy code
VARIATION_ID USERS
cell1 439,863
cell2 442,411
cell3 442,951
cell5 442,880wide-jordan-14758
03/28/2023, 10:39 PMFor context, it’s being flagged now with a warning, but when the SRM first showed up last week, there was no warning for quite a few days
wide-jordan-14758
03/28/2023, 10:39 PMcc @ambitious-apartment-58735
h
helpful-application-7107
03/28/2023, 10:49 PMWhen is the data in your first message from?
helpful-application-7107
03/28/2023, 10:49 PMWhen the error appeared, or from the period when there was no warning?
helpful-application-7107
03/28/2023, 10:51 PMWe have a pretty conservative threshold for SRM, which may explain what's going on here: https://docs.growthbook.io/app/experiments#sample-ratio-mismatch-srm
w
wide-jordan-14758
03/28/2023, 11:14 PMwhen there was no warning
h
helpful-application-7107
03/28/2023, 11:15 PMyeah if I'm doing my chi-square tests right the SRM p-value there is 0.002, and our threshold is 0.001 😕
w
wide-jordan-14758
03/28/2023, 11:15 PMah yep i’m getting the same thing
h
helpful-application-7107
03/28/2023, 11:15 PMwe can definitely, at some point, let orgs set their own threshold to match their own risk tolerance
w
wide-jordan-14758
03/28/2023, 11:15 PMwe typically use .01 for SRMs so this was a flag for us, but good to know why it wasn’t showing up in GrowthBook
h
helpful-application-7107
03/28/2023, 11:16 PMwe've optimized for a more conservative threshold for now, but eventually can consider letting it be set by customers + sequential methods
👍 1
27 Views