https://www.growthbook.io/ logo
Join Slack
Powered by
This message was deleted.
# ask-questions
w
This message was deleted.
f
Hi Albert, correct - it’s random, but should be very close to 50/50
we do have automatic SRM checks
l
Hi again Graham .. thanks for the quick response.. May i conclude that with pure isOn(randomUuid), there's still a possibility to get SRM, eventhough the chance is pretty low. And when it happens, it's better to restart the experiment in hope of getting better bucketing that does not trigger SRM ?
f
its very rare to trigger the SRM with random assignment - I’ve not seen it - did you run your checks through the SRM calculator?
l
Oh, sorry i did not run it through the SRM calculator. Do we have an online calculator, or do we need to check the GB code to run it manually ? I was concluding this based on my previous experience with SRM where i can see the message 
*Warning: Sample Ratio Mismatch (SRM) detected*. We expected a 50/50 split, but observed a 49.6/50.4 split (p-value = 0.0000353). There is likely a bug in the implementation.
with 
49.6
on the error message, and 
49.569
i was getting with one of my 200k isOn() .. i naively conclude that 49.56 is < 49.6, and thus should trigger SRM.
f
yes, I guess that’s correct
@helpful-application-7107 or @future-teacher-7046 thoughts?
l
Thank you for clarifying ! And also .. is it possible that different implementations of UUID might trigger better allocation than other implementations ?
f
hrmm, are you using a new version of GrowthBook?
it shouldn’t but we added extra protection for this case with the v2 hashing algorithm
l
The run i made was with the latest single bundle inclusion with <script> tag
Copy code
<script src='<https://cdn.jsdelivr.net/npm/@growthbook/growthbook/dist/bundles/index.js>'></script>
f
try it with hashVersion: 2
l
The relevant code
Copy code
var gb = new window.growthbook.GrowthBook({
  enableDevMode: true
})
gb.setAttributes({ deviceId: v4() })
let { variationId } = gb.run({
  key: "my-experiment",
  variations: [false, true],
  coverage: 1,
  weights: [0.5, 0.5],
  hashAttribute: "deviceId"
})
ah .. that's probably it .. let me retry ..
but sorry, from my features json created by the GB UI, here's an excerpt
Copy code
{
  "defaultValue": false,
  "rules": [
    {
      "variations": [
        false,
        true
      ],
      "coverage": 1,
      "weights": [
        0.5,
        0.5
      ],
      "key": "my-experiment",
      "hashAttribute": "deviceId"
    }
  ]
}
there's no 
hashVersion: 2
in it, which means 
isOn('my-experiment')
will default to hashVersion 1 ? I have checked the GB UI, and i do not see any options to specify hashVersion in both the feature page and the settings page.
f
right, its a bit hidden, as you shouldn’t change it
🙏 1
l
Ah .. with that, i assume that as long as the default hashVersion is not 2 yet, 
isOn
will make use of v1 that might trigger SRM from the variation allocations ?
f
If you edit the experiment rule
you’ll see it
it defaults to v2 if your SDK supports it
l
Great news .. super thanks @fresh-football-47124.. I will check further.
Hi again @fresh-football-47124, Is there a way for me to easily determine based on the bucketing result alone whether it is an SRM case or not ? I mean what i have been doing is just loop 200k isOn() and then get the ratio. Is this enough to determine it's an SRM ? Because it seems like SRM has something to do with p-value, or probably random sampling from both variations ? Sorry i'm quite novice in this.
f
We use a Chi-squared test
yes, 200k is more than enough samples
🙏 1
l
I will study the metrics sql further and the statistics involved to really understand and debug SRM .. thank you so much for your guidances @fresh-football-47124 🙏
139 Views
Open in Slack
PreviousNext
Powered by