Hi Growthbook! We are setting up an experiment to measure the impact on conversation rate. Now, the challenge is that treatment and control groups will have to take different action in order to convert:
• Treatment group will have to press a button (CTA on Mixpanel)
• Control group will have to schedule a call or schedule a test (Step completed on Mixpanel)
Our question is: Is it possible to set up an experiment with different success metrics?
I believe that from a statistical perspective it should be, how do we set it up on growthbook?
10/03/2023, 5:42 PM
there are a couple of ways I can think of:
1. Use an OR in the metric definition- assuming that the events are exclusive - Mixpanel actually allows for writing JS in the metric conditions. You can get pretty custom here.
2. add one experiment with two different metrics - the statistical significance will not be visible, but you can run the numbers through another calculator (a lot less ideal)
10/03/2023, 7:48 PM
A third could be creating a new metric based on a new event that tracks this specific dual-use conversion for both variations. That way you are containing the experiment-specific changes within a new namespace without introducing unusual data into your standard metric.