Can a segment change over time without messing up ...
# ask-questionsf
flaky-dinner-61668
11/09/2023, 10:36 PMCan a segment change over time without messing up the experiment results? for example if I want to target a specific set of users based on an event that's happening near them, but then update the targets based on the latest event, is the significance measured based on anyone who was ever in the segment, or the current segment? Also, how quickly does the segment update? If we want notify a split of these users, can the segment update on the fly?
f
fresh-football-47124
11/10/2023, 8:34 AM
If you enable streaming for your SDK, the changes to the feature flag rules will propagate instantly
fresh-football-47124
11/10/2023, 8:34 AM
there are two potential problems with changing targeting mid experiment
fresh-football-47124
11/10/2023, 8:35 AM
1. the samples may not be similar enough to give a meaningful result. (ie, if the experiment works for one sample, and fails on the second, your net effect may be 0, or not reflect reality (see Simpsons paradox)
2. Keeping the sample ratios the same may be harder this way.
f
flaky-dinner-61668
11/13/2023, 5:02 PMMy concern with splitting the entire user base is that our experiment only affects a small subset of users based on a rare scenario. Therefore, we risk unbalanced sample sizes for the users who are actually affected by the split.
flaky-dinner-61668
11/13/2023, 5:04 PMI thought to only split users based on the scenario itself, but we will know who they are when it happens, and the treatment is immediate communication based on the event.
r
rhythmic-agent-34208
11/15/2023, 12:04 AMHi Aaron, I'm going to speak with our Data Scientist about your most recent questions and I'll follow up once I have more info 🙂
h
helpful-application-7107
11/15/2023, 10:02 PMThis really depends on what you mean by segment.
Do you mean an attribute that is set in the SDK that you maintain ownership of? If you:
• set an attribute in the SDK whenever that event happens near them
• target your experiment only if they have that attribute
Then your experiment sample will only be made of people for who:
• You set that attribute in your app based on whatever criterion you wanted, who then, while still having that attribute, hit the feature flag in the code path
These people will stay in the experiment for data purposes even if their attribute value changes (e.g. you turn it off in the app because it has been X time since the event you care about).
That is the desired behavior from an analysis perspective, but to make sure you're measuring the treatment effect, you might want to design your metrics to use "conversion windows" so that you only measure behavior in the X hours after experiment exposure.
f
flaky-dinner-61668
11/22/2023, 3:21 PMHi @helpful-application-7107, thanks for your thorough response. Could users end up in both groups in that case? Does the metric need to be mapped to the attribute instead of the user? See my question here: https://growthbookusers.slack.com/archives/C01T6Q1SVFV/p1700666364635739
25 Views