visual, url, or feature flag based?

if you made the flag, there is no pre-assignment happening. So if it was not in use, no one was being assigned. We use deterministic hashing to do the assignment, which means as long as the hashing key (the experiment tracking key) is the same and the assignment attribute is the same, they'll get assigned the same variant. You can also have sticky bucketing enabled which means that even if the split percentage changes, they'll still get the original assigned variant.

can you share your SDK implementation?

you need to see the code or just want to know the sdk version?

engineers can try to fire it outside of that or manually on flag usage

and that causes problems

One thing we wanted to rule out first actually, we are using a device id for assignment, but what we store in our database is actually a user id in most cases (we do this so we can use mixpanel’s identiy merge feature and include non-logged in users). When we query the database we use an identity table to join all the ids associated with a particular user together. So there could be a case of some users seeing multiple variants, but I’d expect that not to be a problem, as those users are filtered out of the results and each arm of the test should be affected equally. In this case there are only around 500 such users, so it wouldn’t explain the SRM anyway. Does that sound alright?

Here is the relevant [swift] code snippet:

growthbook = GrowthBookBuilder(
            apiHost: Environment.current.growthbook.apiHost,
            clientKey: Environment.current.growthbook.clientKey,
            attributes: Self.getAttributes(from: user),
            trackingCallback: { experiment, experimentResult in

                // Ensure the experiment is active and the user is part of it. Although the documentation states this method is "Called whenever someone is put into an experiment", this makes it safer in case that's not always true.
                guard experiment.isActive, experimentResult.inExperiment else {
                    return
                }

                Logger.shared.print("In experiment", mode: .local, info: ["Experiment": experiment.key, "Value": experimentResult.value.rawValue])

                Self.trackExperiment(experiment: .init(key: experiment.key), withResult: experimentResult)
            }, refreshHandler: { cacheRefreshed in
                currentValueSubject.send(cacheRefreshed) // mapped to `growthBookRefreshed`
            }
        ).initializer()
    }

We’ve noticed that there are a lot more

Experiment Joined

events sent to mixpanel for the two variants in this experiment than for the baseline (per unique user). We’d expect that because the code will be hit more often for users who are not in the baseline. Could that be related to the issues you mentioned?