Two items I couldn't find any documentation for but would like to understand how they work exactly:
1. Activation Metric
2. Type of Phase (ramp, main, holdout)
10/15/2021, 7:01 AM
1. activation metrics are used when your experiment assignment and the event that shows the experiment variations are not the same. A good example for this is a modal window that might trigger, but you have to assign all users to that page to the variation. By using the 'modal open' event as the activation metric, you make sure that you're not including users who did not see either variation when doing the analytics.
2. Phasing is entirely personal choice- many experimentation programs start with a ramp up, to check for implementation bugs, then a main experiment to test for significance, then a holdout group to make sure that the results that were seen in the experiment persist and were not due to some other effects (like primacy/novelty effects)
for the activation metrics, we only use that on the analysis side, so we check to make sure all users included in the results also had that activation metric event
10/15/2021, 7:21 AM
Okay got it, very helpful. Thanks!
I understand phasing as a concept but how is it implemented here? i.e. when I change the setting in the UI then what happens to my experiment and how it's being delivered?
10/15/2021, 8:18 AM
depends if you're changing the variation weights - but mostly each phase is treated as a separate experiment in terms of the results- so its just used for the date range for the data in the results
if you want your experiment to change when adjusting things in the UI, you can use a webhook to catch the change. (or the visual editor script will pick this up too)
10/18/2021, 9:07 AM
okay thanks. So, changing the pase will not change how the experiment is being delivered unless I configure it to do this myself?