https://www.growthbook.io/ logo
Join Slack
Powered by
Hello dear GB team, - If i were to create a new g...
# ask-questions
l
Hello dear GB team, • If i were to create a new gb instance with js sdk, hardcoding via gb.setFeature() with 50/50 allocation for variation true / false • and then do a loop 200k times • and within the loop i will create a new UUID for the hashing attribute • gb.setAttribute(deviceUuid) • call isOn() for the featureId i have set to gb instance • will the result be 100k on variation A and 100k on variation B ? 1. I have tested this with different loop counts: 100k, 200k, 400k .. 2. And it seems to me that the bigger the loop count, the bigger the chance for the variations to reach 50/50 3. Once i did 200k run, and the lowest variation percentage is 49.569 on one side 4. With this percentage, seems like this will trigger SRM .. please correct me if i'm wrong ?
f
Hi Albert, correct - it’s random, but should be very close to 50/50
we do have automatic SRM checks
l
Hi again Graham .. thanks for the quick response.. May i conclude that with pure isOn(randomUuid), there's still a possibility to get SRM, eventhough the chance is pretty low. And when it happens, it's better to restart the experiment in hope of getting better bucketing that does not trigger SRM ?
f
its very rare to trigger the SRM with random assignment - I’ve not seen it - did you run your checks through the SRM calculator?
l
Oh, sorry i did not run it through the SRM calculator. Do we have an online calculator, or do we need to check the GB code to run it manually ? I was concluding this based on my previous experience with SRM where i can see the message 
*Warning: Sample Ratio Mismatch (SRM) detected*. We expected a 50/50 split, but observed a 49.6/50.4 split (p-value = 0.0000353). There is likely a bug in the implementation.
with 
49.6
on the error message, and 
49.569
i was getting with one of my 200k isOn() .. i naively conclude that 49.56 is < 49.6, and thus should trigger SRM.
f
yes, I guess that’s correct
@helpful-application-7107 or @future-teacher-7046 thoughts?
l
Thank you for clarifying ! And also .. is it possible that different implementations of UUID might trigger better allocation than other implementations ?
f
hrmm, are you using a new version of GrowthBook?
it shouldn’t but we added extra protection for this case with the v2 hashing algorithm
l
The run i made was with the latest single bundle inclusion with <script> tag
Copy code
<script src='<https://cdn.jsdelivr.net/npm/@growthbook/growthbook/dist/bundles/index.js>'></script>
f
try it with hashVersion: 2
l
The relevant code
Copy code
var gb = new window.growthbook.GrowthBook({
  enableDevMode: true
})
gb.setAttributes({ deviceId: v4() })
let { variationId } = gb.run({
  key: "my-experiment",
  variations: [false, true],
  coverage: 1,
  weights: [0.5, 0.5],
  hashAttribute: "deviceId"
})
ah .. that's probably it .. let me retry ..
but sorry, from my features json created by the GB UI, here's an excerpt
Copy code
{
  "defaultValue": false,
  "rules": [
    {
      "variations": [
        false,
        true
      ],
      "coverage": 1,
      "weights": [
        0.5,
        0.5
      ],
      "key": "my-experiment",
      "hashAttribute": "deviceId"
    }
  ]
}
there's no 
hashVersion: 2
in it, which means 
isOn('my-experiment')
will default to hashVersion 1 ? I have checked the GB UI, and i do not see any options to specify hashVersion in both the feature page and the settings page.
f
right, its a bit hidden, as you shouldn’t change it
🙏 1
l
Ah .. with that, i assume that as long as the default hashVersion is not 2 yet, 
isOn
will make use of v1 that might trigger SRM from the variation allocations ?
f
If you edit the experiment rule
you’ll see it
it defaults to v2 if your SDK supports it
l
Great news .. super thanks @fresh-football-47124.. I will check further.
Hi again @fresh-football-47124, Is there a way for me to easily determine based on the bucketing result alone whether it is an SRM case or not ? I mean what i have been doing is just loop 200k isOn() and then get the ratio. Is this enough to determine it's an SRM ? Because it seems like SRM has something to do with p-value, or probably random sampling from both variations ? Sorry i'm quite novice in this.
f
We use a Chi-squared test
yes, 200k is more than enough samples
🙏 1
l
I will study the metrics sql further and the statistics involved to really understand and debug SRM .. thank you so much for your guidances @fresh-football-47124 🙏
27 Views
Open in Slack
PreviousNext
Powered by