https://www.growthbook.io/ logo
w

witty-laptop-42616

10/15/2021, 6:36 AM
Two items I couldn't find any documentation for but would like to understand how they work exactly: 1. Activation Metric 2. Type of Phase (ramp, main, holdout)
f

fresh-football-47124

10/15/2021, 7:01 AM
Hi Silver: 1. activation metrics are used when your experiment assignment and the event that shows the experiment variations are not the same. A good example for this is a modal window that might trigger, but you have to assign all users to that page to the variation. By using the 'modal open' event as the activation metric, you make sure that you're not including users who did not see either variation when doing the analytics. 2. Phasing is entirely personal choice- many experimentation programs start with a ramp up, to check for implementation bugs, then a main experiment to test for significance, then a holdout group to make sure that the results that were seen in the experiment persist and were not due to some other effects (like primacy/novelty effects)
for the activation metrics, we only use that on the analysis side, so we check to make sure all users included in the results also had that activation metric event
w

witty-laptop-42616

10/15/2021, 7:21 AM
Hi Graham! Okay got it, very helpful. Thanks! I understand phasing as a concept but how is it implemented here? i.e. when I change the setting in the UI then what happens to my experiment and how it's being delivered?
f

fresh-football-47124

10/15/2021, 8:18 AM
depends if you're changing the variation weights - but mostly each phase is treated as a separate experiment in terms of the results- so its just used for the date range for the data in the results
if you want your experiment to change when adjusting things in the UI, you can use a webhook to catch the change. (or the visual editor script will pick this up too)
w

witty-laptop-42616

10/18/2021, 9:07 AM
okay thanks. So, changing the pase will not change how the experiment is being delivered unless I configure it to do this myself?
f

fresh-football-47124

10/18/2021, 3:49 PM
yes, for the most part.
6 Views