https://www.growthbook.io/ logo
#announcements
Title
# announcements
w

wide-jordan-14758

12/07/2022, 9:36 PM
Hi, I'm hoping to get more information on how Sample Ratio Mismatches are detected. We have experiments that are flagged as having an SRM but the displayed splits are rounded to a 10th of a percent whereas the true splits should be irrational numbers (6 variants with equal allocation of the full audience) so I'm wondering if the SRM flag has something to do with rounding that is happening before the proportions test. Any info is much appreciated! (cc @ambitious-apartment-58735)
f

future-teacher-7046

12/07/2022, 10:38 PM
yeah, that's likely the cause. We take the total test traffic and multiply by the expected split to use in our comparison. So it's expecting
.167*total
instead of
.166666*total
. With enough traffic, that could cause a false positive for the SRM test
w

wide-jordan-14758

12/07/2022, 10:51 PM
Thank you! @ambitious-apartment-58735 Thoughts on if we can still use this feature for QA?
a

ambitious-apartment-58735

12/08/2022, 1:55 AM
I think we’d end up running the check independently — or we can insist on numbers of cells that lead to rational splits 😆. @future-teacher-7046 would it be easy for you to increase precision there? Is our data (sometimes millions of users/cell) atypically large for growthbook users?
f

future-teacher-7046

12/08/2022, 2:12 AM
Yeah, we can increase precision pretty easily. Millions of users is not atypical for GrowthBook, but 6-way splits are less common.
a

ambitious-apartment-58735

12/08/2022, 2:35 PM
Ah, makes sense. Thanks!
12 Views