Hey guys, a question and a request: 1. Question: We’re finding that we want to test the effect of a feature on new users, and existing users as two separate groups, since new users would never know what it was like before, and existing users have to adapt to the change. What is the best way to set this up as far as feature flagging and experimenting goes? Right now, I’m creating two separate feature flags (one for new users, one for existing), creating separate override rules for each as part of setting up the A/B test and then creating two experiments in the experiments list. This feels hacky, is there perhaps a better way to do this? 2. Request: In the experiments, results tab, in the metrics table, can we see the type of metric it is e.g. binomial or count or duration, etc.. Especially when we turn on “converted only” for a count metric, it means something pretty different than a binomial metric and my non-tech team members may get confused

For #1, I think long term we would make that a built-in dimension for Mixpanel (new vs existing). That requires a few other changes first like supporting user properties for dimensions. For #2, that's a good idea to have more context for the type and behavior of metrics. We're considering splitting that table into 2 views, a simplified one for non-technical people, and a data-heavy one. Right now, it's already pretty overwhelming to non-technical people so adding more info to the table might make it worse.

For #1: what I’m hearing is that we should try to use dimensions, which I could set with super properties on each event e.g. a user’s first app open. So there I can save you the trouble of supporting user profile properties. However, how would I convert a value that has a discrete distribution to a boolean (i.e. is before or after a certain time stamp)? For #2: sure something like that would be helpful. If there’s any small thing you can add that could help with specifically type of metric (especially differentiating between binomial and count) would be nice

In SQL, we have support for template placeholders, so you can add something like

WHERE date < '{{ experimentStart }}'

. I think we can do something similar for Mixpanel.

yeh if setting something like that up wouldn’t kill ya, would be great, because then I can use dimensions with the event super properties

I made a PR for this. https://github.com/growthbook/growthbook/pull/342 Doing some final testing now, but should be good to go soon