Returning vs New User Tests: The Messy Measurement… We’re running server-side tests and coming across tricky questions: What happens: 1. new user lands on site, gets added into new user test 2. user eventually becomes a returning user, at which point they remain in the new user test 3. BUT since they are now evaluated against the conditions of the returning user test, they now get added into that test also The options I can see: 1. we only run either new user tests OR returning user tests at the any given time 2. we never change the returning user flag. so a returning user is just anyone that visited the site before ABT launched, and stays like that forever 3. we don’t do new/returning user tests, because what is the actual reason. thinking it through, I’m struggling to find a valid reason. time-based cohorts makes sense, new vs returning sounds like a really arbitrary reason to cause headaches.

1 is usually how people do it.

thanks!

3 is interesting, but would depend on the specific hypothesis - I'm having trouble thinking of how this will help

@fresh-football-47124 you’re referring to the first set of numbers here right? these ones?

no, sorry, the solutions

ah i see

three for you shouldn't happen, because that user will be marked as enrolled one way or another