https://www.growthbook.io/ logo
Join Slack
Powered by
I have a test experiment that I'm running with see...
# announcements
c
I have a test experiment that I'm running with seed data. However, I'm getting an error of 
A Sample Ratio Mismatch (SRM) was detected with p-value of 0.
I inserted 2000 experiment_viewed records, split 50:50 across the 2 variations. From the 1000 users that saw variation 0, 300 went onto convert. Of the 1000 users that saw variation 1, 700 went onto convert. 1000 users did not convert at all. I'm trying to understand why this is giving a SRM.
The help doc on this says "SRM happens when the actual traffic split is different from what you expect.", but in our case the experiment_viewed is exactly 50:50
The metric is 30:70, which is what I would expect, but this is what GB is complaining about
I think the reason for this is because of this
It seems like any user that doesn't convert to a subscription is excluded
Which seems strange, and I'm probably missing something obvious here. Why would you totally exclude users that don't convert to a subscription?
f
Activation Metric is used to completely filter people out of the analysis. For example, if you are assigning variations on page load, but the experiment doesn't actually happen until a modal pops up, you could use a "Viewed modal" event as the activation metric.
Most of the time you don't need to set an activation metric. It's only for cases where the experiment query returns extra users that didn't actually see the experiment and you need to filter those out to get cleaner data.
c
Oh, got it! Thank you @future-teacher-7046 - I removed the activation metric, and it looks much better!
79 Views
Open in Slack
PreviousNext
Powered by