flaky-dinner-6166811/22/2023, 3:19 PM
rhythmic-agent-3420811/22/2023, 4:43 PM
helpful-application-710711/22/2023, 5:45 PM
flaky-dinner-6166811/22/2023, 8:03 PM
helpful-application-710711/22/2023, 8:10 PM
instead of on
. You can do this in GrowthBook if you set it up to hash on a different id type.
In this setting, you are violating a central assumption that undergirds experimentation of no spillovers (also referred to as the stable unit treatment value assumption, or SUTVA). Because whether
got treatment in
affects their reaction to treatment or control in
, the results of
are no longer independent.
This is the subject of a large body of work in experimentation, and is an issue that experimentation platforms like those at Lyft (where if they affect driver 1 in palo alto that also affects driver 2), Amazon (where if they affect the price of good 1, it may affect the sales of good 2) have to deal with explicitly. There are almost always some spillovers, but we tend to hope they are small.
Once you start experimenting within user, it becomes very unlikely they are small.