https://www.growthbook.io/ logo
Join Slack
Powered by
[Bug] Sticky Bucketing does not seem to work as ex...
# ask-questions
f
[Bug] Sticky Bucketing does not seem to work as expected Prerequisites: • GB 
Build: 3.4.0+5957d6e (2025-01-17)
• ReactSDK 
"@growthbook/growthbook-react": "^1.4.1",
Copy code
// Create a GrowthBook instance
const growthbook = new GrowthBook({
  apiHost: <host>,
  clientKey: "sdk-key",
  enableDevMode: true,
  stickyBucketService: new LocalStorageStickyBucketService(),
});
• Pro plan is enabled • Settings > General > Sticky Bucketing Enabled ✅ Steps to reproduce 1. Create feature 2. Create experiment with two variants A, B 3. Hash attribute 
id
4. Set weights A=100, B=0 5. Using react SDK always send static 
id
attribute (
user1
) 6. Receive variant A 7. Update weights A=0,B=100 8. Send evaluation with same attributes Actual Result Received variant B Expected Result Received same variant A Would appreciate any help to troubleshoot We also tested on PythonSDK with custom StickyBucketing service - same behaviour
Local storage: 
gbFeaturesCache
Copy code
[
  [
    "https://<host>||sdk-xDZ6yW1x9Uw9HUkQ",
    {
      "data": {
        "features": {
          "test_sticky_2": {
            "defaultValue": {},
            "rules": [
              {
                "coverage": 1,
                "hashAttribute": "id",
                "bucketVersion": 4,
                "seed": "697b3444-8061-47f2-9bc6-10f70d93e178",
                "hashVersion": 2,
                "variations": [
                  {
                    "A": "A"
                  },
                  {
                    "B": "B"
                  }
                ],
                "weights": [
                  0,
                  1
                ],
                "key": "test_sticky_2",
                "meta": [
                  {
                    "key": "0",
                    "name": "A"
                  },
                  {
                    "key": "1",
                    "name": "B"
                  }
                ],
                "phase": "7",
                "name": "test_sticky_2"
              }
            ]
          }
        },
        "dateUpdated": "2025-02-24T10:52:47.904Z"
      },
      "version": "2025-02-24T10:52:47.904Z",
      "staleAt": "2025-02-24T11:02:33.021Z",
      "sse": true
    }
  ]
]
gbStickyBuckets__id||user1
Copy code
{
  "attributeName": "id",
  "attributeValue": "user1",
  "assignments": {
    "test_sticky_2__1": "0"
  }
}
f
@happy-autumn-40938, thoughts?
👀 1
h
At first glance it looks like your sticky bucket 
"test_sticky_2__1":"0"
is only assigned "0" (control) is using a stale cache key ("1" from 
__1
). Your experiment definition however is already on cache key "4" (
"bucketVersion": 4,
). Basically your sticky buckets were forcibly invalidated after you made the variation weight changes. As for why... generally when you shift variation weights around, the GB statistics engine doesn't work well when user traffic split deviates from the expected weights (except in the context of a Bandit which has special logic for fixing the stats). To avoid statistical issues, the GB app will often take a conservative approach to managing the problem - recommending that you choose the "New phase, re-randomize traffic" release plan when changing variation weights, which forcibly invalidates sticky buckets. If you know what you're doing though and are ok incurring the statistical penalty, there's an escape hatch that lets you change variation weights while keeping previous sticky buckets. When going through the "Make Changes" flow, choose "Advanced: multiple changes at once" and then make your weight changes — new release plans are available. The one you'd likely want to choose is "Same phase, apply changes to new traffic only (Sticky Bucketing)".
👀 1
f
Got it thanks! For more context, we are migrating our existing experiment and multiarmbandit to the growthbook. For smooth transition we have to use our current custom MultiArmBandit implementation together with integrator service that update the experiment weights via Experiment API https://docs.growthbook.io/api/#tag/experiments/operation/updateExperiment Could you please advice is this approach would be working or maybe you would suggest other workaround?
h
Not sure I understand the strategy. Are you wanting to (1) manage the bandit weight updates outside of GrowthBook and just use GB for reporting and payload serving? Or are you wanting to (2) convert your custom bandits to be fully managed by GrowthBook? In either case, manually updating variation weights via the REST API may lead to problems. In case 1, you're going to run into statistical problems (multiple exposures, SRM) because GB doesn't adjust for these issues when using sticky bucketing in a standard experiment. In case 2, the bandit would be fully managed by GB and would essentially be locked down from manual mutation via REST API (it might technically let you post the update, but it would likely cause problems). Is it possible to restart your bandits and have them managed entirely within GB (or perhaps only create new bandits in GB)?
👀 1
f
At first stage, yes I want to manage the bandit weight updates outside of GrowthBook and just use GB for reporting and payload serving I expected that sticky bucketing would still work in this case, so even the weights change and the new evaluation would return new variant the sticky bucketing would step in, see the already existing variant and use it You are saying its not going to work right?
h
It might technically work, in terms of the treatment the user will receive, but it won't work well from a statistical / test power standpoint. Regular experiments don't handle multiple exposures and SRM caused by variation weight changes gracefully, so instead multiple exposures are thrown out of the analysis. Only a native Bandit (GB's Bandit) is equipped to preserve sticky bucketing without sacrificing statistical power.
👀 1
f
we analyze the experiment outside as well, so it is only important that new users will do evaluations with latests weights, previous users should receive old variants
h
if you care most about user experience and not statistical power, then your proposal will be sufficient. If you want to use the bandit to accelerate your experiment (reach a winner faster) then you'd probably be better off only doing this for new experiments/bandits
👀 1
or, if you have a list of user ids who were already exposed to specific buckets, you could create saved groups from those and automatically assign static variations to those groups using feature rules. Then put a new bandit (100% managed by GB) further down the rule list
👀 1
f
We are looking to switch entirely to GB Bandit, but with iterations. Currently cannot switch at once. I am going to do some more integration testing with weights updates via Experiment API and Sticky Bucketing Service
22 Views
Open in Slack
PreviousNext
Powered by