Hi team GB it s my first time encountered an SRM ```SRM happ GrowthBook Users #announcements

Hi team GB, it's my first time encountered an SRM ...

loud-hydrogen-50191

08/14/2023, 4:59 AM

Hi team GB, it's my first time encountered an SRM ..

Copy code

SRM happens when the actual traffic split is different from what you expect.
For this test, the p-value of the SRM check is 6.8e-7.That means there's only a 1 in 1470588 chance the observed traffic split happened by random chance.

May i know how GB determines this SRM for a single experiment, like is there an sql available for me to run ? Is it affected by the

Data Sources

Join Tables

or perhaps

Experiment Assignment Queries

fresh-football-47124

08/14/2023, 5:37 AM

it could be caused by the assignment attribute of the experiment being different than the assignment in the experiment report

fresh-football-47124

08/14/2023, 5:37 AM

but usually they are caused by changing of the split mid experiment

loud-hydrogen-50191

08/14/2023, 5:46 AM

hi @fresh-football-47124 thanks for the quick reply. I have checked that 1. the hashed attribute did not change, 2. the experiment start time is above the last feature change i can see in the audit log. 3. split was not modified

fresh-football-47124

08/14/2023, 7:01 AM

what split do you have?

loud-hydrogen-50191

08/14/2023, 8:38 AM

hi Graham, sorry for the late reply, it's 50/50

Copy code

{
  "defaultValue": false,
  "rules": [
    {
      "condition": {
        "deviceId": "test-forced-on"
      },
      "force": true
    },
    {
      "condition": {
        "loginStatus": 0
      },
      "variations": [
        false,
        true
      ],
      "coverage": 1,
      "weights": [
        0.5,
        0.5
      ],
      "key": "MyExptKey",
      "hashAttribute": "deviceId"
    }
  ]
}

loud-hydrogen-50191

08/14/2023, 8:45 AM

Copy code

Warning: Sample Ratio Mismatch (SRM) detected. We expected a 50/50 split, but observed a 50.4/49.6 split (p-value = 9.2e-7). There is likely a bug in the implementation.

1. Sorry, i misunderstood your question. It's 50.4/49.6 split. 2. And i assume GB is able to calculate the split number is because of all of the queries (i currently have 10 of them) for this particular experiments 3. And i'm wondering which field is used to calculate the split number ? I can see

count, main_sum, main_sum_square

68 Views

Open in Slack

Previous Next