https://www.growthbook.io/ logo
f

fancy-zoo-90893

04/09/2022, 6:33 AM
Quick question: SRM with p-val = 0 means there's no SRM, right? So this message shouldn't be displayed?
More context: I'd set the dates of the experiment incorrectly so there was initially a mismatch with a higher p-val (0.00003...) but when I fixed it, I can see that the numbers are indeed close to each other (7.4% vs. 9.57%) so there shouldn't be a mismatch anymore.
My guess is that GrowthBook just remembers that there was a mismatch and continues to show this message? Or am I missing something and there really is a mismatch still?
Thanks! 😄
f

fresh-football-47124

04/09/2022, 6:35 AM
hi Anand
are you using the feature flags to do the variation assignments?
f

fancy-zoo-90893

04/09/2022, 6:39 AM
Yep!
f

fresh-football-47124

04/09/2022, 6:41 AM
was it 50:50 split in the flag?
f

fancy-zoo-90893

04/09/2022, 6:42 AM
Also yes 😄
f

fresh-football-47124

04/09/2022, 6:43 AM
huh, ya let me look into that - you do have the experiment also set to 50:50?
f

fresh-football-47124

04/09/2022, 7:07 AM
since you're numbers are low, I'd ignore it for now - I'll ping jeremy with the details
the p value is like .2 x 10 ^ -31 and we're rounding to 8 decimal places - so it shows as 0
in the code we check if that p-value < 0.001 to show the warning.
f

future-teacher-7046

04/09/2022, 1:55 PM
Looks like the data just updated and the warning is gone. Something weird was definitely going on with that calculation though. 135 vs 115 users should result in a p-value of like 0.2, which is nowhere near significant. I'll take a look and try and reproduce the issue.
🙌 1
f

fancy-zoo-90893

04/10/2022, 9:34 AM
Thanks!