https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
f

flaky-dinner-61668

11/22/2023, 3:19 PM
Does Growthbook only work on splitting users? Wouldn't it be valuable to have the ability to see how the same user responds to different treatments?
r

rhythmic-agent-34208

11/22/2023, 4:43 PM
You can do both! 1. GrowthBook uses deterministic hashing to assign users to experiment variations, ensuring that the same user always gets the same variation given ​*the same experiment settings*​ (experiment key and user hashing id). This approach is designed to maintain consistency across multiple pages or applications for a given user. Therefore, by default, a user would not be exposed to different variations within the same experiment. 2. However, if you want to test how the same user responds to different variations, you would need to set up ​*separate experiments*​ with different experiment keys or user hashing ids. This way, you could potentially track a user's response to different variations across these separate experiments.
h

helpful-application-7107

11/22/2023, 5:45 PM
I think Aaron is asking about the case where you can compare the same user in both treatments in the context of an experiment. This is an approach used in switchback experiments, but it opens a full set of statistical challenges that standard experimentation stats can't really solve. In fact, getting that "ability to see how the same user responds to both" is impossible, since the user in control in week2 after getting the experiment in week1 is not the same as the user in week1 getting the control. Experimentation explicitly tries to create this "counterfactual" user, but does so across experiments.
f

flaky-dinner-61668

11/22/2023, 8:03 PM
Are you saying it should statistically be treated as a completely new user? Wouldn't that be ok if we're not testing users, but rather the scenarios themselves, and if the metric is tied to the scenario?
h

helpful-application-7107

11/22/2023, 8:10 PM
This is a pretty nuanced discussion, but basically lets say you randomize and analyze on
scenario_id
instead of on
user_id
. You can do this in GrowthBook if you set it up to hash on a different id type. In this setting, you are violating a central assumption that undergirds experimentation of no spillovers (also referred to as the stable unit treatment value assumption, or SUTVA). Because whether
user_id
got treatment in
scenario 1
affects their reaction to treatment or control in
scenario 2
, the results of
scenario 1
and
scenario 2
are no longer independent. This is the subject of a large body of work in experimentation, and is an issue that experimentation platforms like those at Lyft (where if they affect driver 1 in palo alto that also affects driver 2), Amazon (where if they affect the price of good 1, it may affect the sales of good 2) have to deal with explicitly. There are almost always some spillovers, but we tend to hope they are small. Once you start experimenting within user, it becomes very unlikely they are small.
4 Views