https://www.growthbook.io/ logo
Powered by
#ask-questions
Title
# ask-questions
l

loud-hydrogen-50191

09/19/2023, 3:03 AM
Hello dear GB team, • If i were to create a new gb instance with js sdk, hardcoding via gb.setFeature() with 50/50 allocation for variation true / false • and then do a loop 200k times • and within the loop i will create a new UUID for the hashing attribute • gb.setAttribute(deviceUuid) • call isOn() for the featureId i have set to gb instance • will the result be 100k on variation A and 100k on variation B ? 1. I have tested this with different loop counts: 100k, 200k, 400k .. 2. And it seems to me that the bigger the loop count, the bigger the chance for the variations to reach 50/50 3. Once i did 200k run, and the lowest variation percentage is 49.569 on one side 4. With this percentage, seems like this will trigger SRM .. please correct me if i'm wrong ?
f

fresh-football-47124

09/19/2023, 3:12 AM
Hi Albert, correct - it’s random, but should be very close to 50/50
we do have automatic SRM checks
l

loud-hydrogen-50191

09/19/2023, 3:14 AM
Hi again Graham .. thanks for the quick response.. May i conclude that with pure isOn(randomUuid), there's still a possibility to get SRM, eventhough the chance is pretty low. And when it happens, it's better to restart the experiment in hope of getting better bucketing that does not trigger SRM ?
f

fresh-football-47124

09/19/2023, 3:15 AM
its very rare to trigger the SRM with random assignment - I’ve not seen it - did you run your checks through the SRM calculator?
l

loud-hydrogen-50191

09/19/2023, 3:27 AM
Oh, sorry i did not run it through the SRM calculator. Do we have an online calculator, or do we need to check the GB code to run it manually ? I was concluding this based on my previous experience with SRM where i can see the message 
*Warning: Sample Ratio Mismatch (SRM) detected*. We expected a 50/50 split, but observed a 49.6/50.4 split (p-value = 0.0000353). There is likely a bug in the implementation.
with 
49.6
on the error message, and 
49.569
i was getting with one of my 200k isOn() .. i naively conclude that 49.56 is < 49.6, and thus should trigger SRM.
f

fresh-football-47124

09/19/2023, 3:27 AM
yes, I guess that’s correct
@helpful-application-7107 or @future-teacher-7046 thoughts?
l

loud-hydrogen-50191

09/19/2023, 3:28 AM
Thank you for clarifying ! And also .. is it possible that different implementations of UUID might trigger better allocation than other implementations ?
f

fresh-football-47124

09/19/2023, 3:29 AM
hrmm, are you using a new version of GrowthBook?
it shouldn’t but we added extra protection for this case with the v2 hashing algorithm
l

loud-hydrogen-50191

09/19/2023, 3:34 AM
The run i made was with the latest single bundle inclusion with <script> tag
Copy code
<script src='<https://cdn.jsdelivr.net/npm/@growthbook/growthbook/dist/bundles/index.js>'></script>
f

fresh-football-47124

09/19/2023, 3:35 AM
try it with hashVersion: 2
l

loud-hydrogen-50191

09/19/2023, 3:35 AM
The relevant code
Copy code
var gb = new window.growthbook.GrowthBook({
  enableDevMode: true
})
gb.setAttributes({ deviceId: v4() })
let { variationId } = gb.run({
  key: "my-experiment",
  variations: [false, true],
  coverage: 1,
  weights: [0.5, 0.5],
  hashAttribute: "deviceId"
})
ah .. that's probably it .. let me retry ..
but sorry, from my features json created by the GB UI, here's an excerpt
Copy code
{
  "defaultValue": false,
  "rules": [
    {
      "variations": [
        false,
        true
      ],
      "coverage": 1,
      "weights": [
        0.5,
        0.5
      ],
      "key": "my-experiment",
      "hashAttribute": "deviceId"
    }
  ]
}
there's no 
hashVersion: 2
in it, which means 
isOn('my-experiment')
will default to hashVersion 1 ? I have checked the GB UI, and i do not see any options to specify hashVersion in both the feature page and the settings page.
f

fresh-football-47124

09/19/2023, 3:40 AM
right, its a bit hidden, as you shouldn’t change it
🙏 1
l

loud-hydrogen-50191

09/19/2023, 3:41 AM
Ah .. with that, i assume that as long as the default hashVersion is not 2 yet, 
isOn
will make use of v1 that might trigger SRM from the variation allocations ?
f

fresh-football-47124

09/19/2023, 3:42 AM
If you edit the experiment rule
you’ll see it
it defaults to v2 if your SDK supports it
l

loud-hydrogen-50191

09/19/2023, 3:44 AM
Great news .. super thanks @fresh-football-47124.. I will check further.
Hi again @fresh-football-47124, Is there a way for me to easily determine based on the bucketing result alone whether it is an SRM case or not ? I mean what i have been doing is just loop 200k isOn() and then get the ratio. Is this enough to determine it's an SRM ? Because it seems like SRM has something to do with p-value, or probably random sampling from both variations ? Sorry i'm quite novice in this.
f

fresh-football-47124

09/19/2023, 6:19 AM
We use a Chi-squared test
yes, 200k is more than enough samples
🙏 1
l

loud-hydrogen-50191

09/19/2023, 7:30 AM
I will study the metrics sql further and the statistics involved to really understand and debug SRM .. thank you so much for your guidances @fresh-football-47124 🙏
3 Views
Powered by