This message was deleted.

yeah, that's likely the cause. We take the total test traffic and multiply by the expected split to use in our comparison. So it's expecting

.167*total

instead of

.166666*total

. With enough traffic, that could cause a false positive for the SRM test

Thank you! @ambitious-apartment-58735 Thoughts on if we can still use this feature for QA?

I think we’d end up running the check independently — or we can insist on numbers of cells that lead to rational splits 😆. @future-teacher-7046 would it be easy for you to increase precision there? Is our data (sometimes millions of users/cell) atypically large for growthbook users?

Yeah, we can increase precision pretty easily. Millions of users is not atypical for GrowthBook, but 6-way splits are less common.

Ah, makes sense. Thanks!