Hi! This question originally came up in #C05T9EY0UKC, but it turns out that the behavior is not SDK-specific, so I wanted to clarify it here. We use experiments with Sticky Bucketing enabled. One of the attributes we use for experiment targeting is

first_visit_date

, which determines whether a user is considered new and eligible for an experiment.

first_visit_date

is initially set when the user opens the app for the first time. However, if the user later signs in to an existing account, this attribute is updated to the

first_visit_date

associated with that account. This creates the following scenario: • A user is initially eligible for an experiment targeting new users. • They are assigned to a variation, and the assignment is persisted through Sticky Bucketing for the duration of the experiment phase. • The user then signs in, and their attributes are updated. • Based on the new

first_visit_date

, the user no longer matches the experiment’s targeting conditions. • However, because of Sticky Bucketing, they remain in the experiment and continue receiving the assigned variation. Is there any mechanism that would allow us to remove such a user from the experiment? We are not only looking to exclude these users from the experiment analysis, which would be relatively straightforward. We would also like them to stop receiving the experiment variation once they no longer match the targeting conditions. One idea would be to add a rule before the experiment with conditions that are the inverse of the experiment’s targeting conditions, so that these users are immediately excluded from further targeting. However, this would not remove them from the experiment results or exposure data. Ideally, we would also like to have a callback similar to

experiment_view

, but for when a user leaves or becomes ineligible for an experiment — something like

experiment_leave

.

Hi @late-ambulance-66508 good morning 👋 Thanks for the detail — yes, that behaviour is expected with sticky bucketing. Once a user has a sticky assignment, GrowthBook can keep serving that variation even if their attributes later change and they no longer match the original experiment targeting. To stop that happening going forward, the best option is to add a higher-priority “gate” rule before the experiment: 1. If user is no longer eligible → force the default/non-experiment value 2. Else → allow them into the experiment rule That will stop ineligible users from receiving the experiment variation going forward. It won’t remove their previous exposure/results though, so for analysis you’d still need to filter those users out separately. There isn’t currently a built-in

experiment_leave

callback. If you control the sticky bucket store, you could also clear that user’s sticky assignment when they become ineligible, but that would need to be handled in your implementation.

Hi @flaky-noon-11399, Thanks for your reply. Yep, that's exactly what I've described. As for the

experiment_leave

- I know, it was more like feature request. 🙂

Hey @late-ambulance-66508. Natasha is right about the way to fix the delivery. It does open up a risk for your analysis, though: Signing in happens after exposure, so if you exclude on it, you're conditioning on something the treatment might have caused. Say the feature makes people more likely to sign in. Then you drop more users from the treatment variation than from control, and the ones you drop are plausibly the ones the feature worked best on. If it moved someone enough to make them sign in when they otherwise wouldn't, it probably moved the rest of their behavior too, purchases and so on. This should drag an effect estimate down toward zero. The SRM test could catch it if the sign-in rates differ enough, but it's not bulletproof. Sign-in rates could differ only marginally, but if these users are the ones who are impacted the most, it could materially change the treatment effect estimate without raising the SRM flag. There is no obvious fix for this, unfortunately. This situation with signed-out users where eligibility changes at sign-in is notoriously tricky, and the best mitigation would depend on the context. Happy to bounce ideas if you want?

Hi @crooked-rainbow-90115 Thanks, that makes sense. I think the key distinction in our case is delivery vs. analysis. I agree that if sign-in or purchase happens after exposure and may be affected by the treatment, we should not simply remove those users from the experiment results. I would keep the original exposure and their subsequent outcomes in the analysis. The delivery problem itself is a bit simpler for us. We mostly have two cases, assuming that we fetch feature flags on app start and again whenever relevant attributes change. 1. Existing user who initially looks like a new user A previously registered user opens the app on a new/signed-out device and goes through onboarding. At that point we only know the device-level

first_visit_date

, so they qualify for an experiment targeting new users and see UX optimized for newcomers. Later they sign in, and

first_visit_date

is replaced with the account-level value. At that point they no longer satisfy the experiment targeting, but Sticky Bucketing keeps serving the assigned variation. This creates a couple of practical inconsistencies for us: • If we later roll out the winning variation as a regular rule in a feature flag using the same

first_visit_date

targeting, the user is now evaluated with the correct account-level attribute and no longer matches. So the UX they were seeing during the experiment can suddenly change after rollout. • Cross-device behavior becomes inconsistent as well. A user may first sign in on an iPhone, then later use an iPad or another phone, and effectively see different UX depending on which sticky assignment/history exists on that device. 2. Eligibility changes naturally during the user lifecycle Another common example is an experiment targeting users without a subscription. If the user purchases a subscription, they no longer belong to the population the feature is intended for, but Sticky Bucketing can keep them in the experimental experience. Of course, there are experiments where the treatment is specifically meant to drive that transition — e.g. increasing purchase conversion — and in those cases we absolutely want to preserve the original assignment/exposure for analysis. So I wouldn't expect something like

experiment_leave

to automatically remove a user from experiment results. What would be useful for us is an optional callback/event when a user stops matching an experiment's targeting after attributes are updated. Then the application could decide whether to stop serving the variation / clear the sticky assignment, while still keeping the original exposure in the experiment data. In other words, I'd like to keep the causal analysis intact, but have more control over the runtime lifecycle of an experiment assignment.

Makes sense, and I think that's the right design. What would worry me is a callback that also pulled those users out of the results, since the attribute change might be caused by the treatment itself. Yours keeps the exposure, so the analysis stays valid whatever gets served later on. Has anyone else here dealt with the identity mess before and after sign-in? Curious what people randomize on when the experiment mainly targets signed-out users (or more generally users who might become ineligible mid-experiment).