Quick question: SRM with p-val = 0 means there's n...
# announcementsf
fancy-zoo-90893
04/09/2022, 6:33 AMQuick question: SRM with p-val = 0 means there's no SRM, right? So this message shouldn't be displayed?
fancy-zoo-90893
04/09/2022, 6:34 AMMore context: I'd set the dates of the experiment incorrectly so there was initially a mismatch with a higher p-val (0.00003...) but when I fixed it, I can see that the numbers are indeed close to each other (7.4% vs. 9.57%) so there shouldn't be a mismatch anymore.
fancy-zoo-90893
04/09/2022, 6:35 AMMy guess is that GrowthBook just remembers that there was a mismatch and continues to show this message? Or am I missing something and there really is a mismatch still?
fancy-zoo-90893
04/09/2022, 6:35 AMThanks! 😄
f
fresh-football-47124
04/09/2022, 6:35 AM
hi Anand
fresh-football-47124
04/09/2022, 6:35 AM
are you using the feature flags to do the variation assignments?
f
fancy-zoo-90893
04/09/2022, 6:39 AMYep!
f
fresh-football-47124
04/09/2022, 6:41 AM
was it 50:50 split in the flag?
f
fancy-zoo-90893
04/09/2022, 6:42 AMAlso yes 😄
f
fresh-football-47124
04/09/2022, 6:43 AM
huh, ya let me look into that - you do have the experiment also set to 50:50?
f
fancy-zoo-90893
04/09/2022, 6:50 AMf
fresh-football-47124
04/09/2022, 7:07 AM
since you're numbers are low, I'd ignore it for now - I'll ping jeremy with the details
fresh-football-47124
04/09/2022, 7:08 AM
the p value is like .2 x 10 ^ -31 and we're rounding to 8 decimal places - so it shows as 0
fresh-football-47124
04/09/2022, 7:09 AM
in the code we check if that p-value < 0.001 to show the warning.
f
future-teacher-7046
04/09/2022, 1:55 PM
Looks like the data just updated and the warning is gone. Something weird was definitely going on with that calculation though. 135 vs 115 users should result in a p-value of like 0.2, which is nowhere near significant. I'll take a look and try and reproduce the issue.
🙌 1
f
fancy-zoo-90893
04/10/2022, 9:34 AMThanks!
67 Views