Next question-- When using a percentage rollout, what algorithm is used for distribution of the variations, and how close is the percentage expected to be for a smaller sample size? While testing a percentage based rollout of 50/50 (this is unrelated to my experiment above, this was a "rollout" rule) with 100 feature evals, I would see lopsided results with a much as 81/19 distribution for a 50/50 rollout using a random unique GUID for my

id

attribute. The percentage evens out more if I do 5,000 evals, but I was curious what the expectation was. I was trying to write a unit test to confirm a 50/50 rollout worked, but the test kept failing from wide variations in distribution at low-ish sample sizes. (Java SDK, if that matters)

GrowthBook uses deterministic hashing for the distribution of variations in a percentage rollout. (Let me know if you want more info about that process). With smaller sample sizes, as you. saw, the distribution may not always be 50/50—but we'll approach that split the larger the sample gets. It's generally done this way to avoid having to do expensive lookups/rebalancing.

Ok that's basically what I expected. I was just surprised to see it so lopsided in a sample size of 100 tests.