Can a segment change over time without messing up the experiment results? for example if I want to target a specific set of users based on an event that's happening near them, but then update the targets based on the latest event, is the significance measured based on anyone who was ever in the segment, or the current segment? Also, how quickly does the segment update? If we want notify a split of these users, can the segment update on the fly?
11/10/2023, 8:34 AM
If you enable streaming for your SDK, the changes to the feature flag rules will propagate instantly
there are two potential problems with changing targeting mid experiment
1. the samples may not be similar enough to give a meaningful result. (ie, if the experiment works for one sample, and fails on the second, your net effect may be 0, or not reflect reality (see Simpsons paradox)
2. Keeping the sample ratios the same may be harder this way.
11/13/2023, 5:02 PM
My concern with splitting the entire user base is that our experiment only affects a small subset of users based on a rare scenario. Therefore, we risk unbalanced sample sizes for the users who are actually affected by the split.
I thought to only split users based on the scenario itself, but we will know who they are when it happens, and the treatment is immediate communication based on the event.
11/15/2023, 12:04 AM
Hi Aaron, I'm going to speak with our Data Scientist about your most recent questions and I'll follow up once I have more info 🙂
11/15/2023, 10:02 PM
This really depends on what you mean by segment.
Do you mean an attribute that is set in the SDK that you maintain ownership of? If you:
• set an attribute in the SDK whenever that event happens near them
• target your experiment only if they have that attribute
Then your experiment sample will only be made of people for who:
• You set that attribute in your app based on whatever criterion you wanted, who then, while still having that attribute, hit the feature flag in the code path
These people will stay in the experiment for data purposes even if their attribute value changes (e.g. you turn it off in the app because it has been X time since the event you care about).
That is the desired behavior from an analysis perspective, but to make sure you're measuring the treatment effect, you might want to design your metrics to use "conversion windows" so that you only measure behavior in the X hours after experiment exposure.