some-planet-4410411/17/2023, 2:20 PM
rhythmic-agent-3420811/17/2023, 2:20 PM
helpful-application-710711/21/2023, 6:32 PM
y = b0 + b1 * t1 + b2 * t2 + b3 * t1 * t2
is 0/1 for a/b in ab test 1 and
is 0/1 for a/b in ab test 2. Then if
is different from 0 there is evidence of an interaction effect. There's an analog in ANOVA but I find that people describe anova models in different ways that makes communicating with them harder than communicating with regression models.
some-planet-4410411/23/2023, 3:17 PM
Sorry if something sounds not exactly clear - I’m still relatively new to statistics and just trying to find my way around
old_control, new_control, old_treatment, new_treatment
helpful-application-710712/01/2023, 4:05 PM
I heard that same results can be obtained through regression though.Yes, I'm just stating it in regression format because I find it easier to communicate about given my backgorund. Multi-factor anova will be identical to regression in this setting.
but also for testing the presence of novelty and primacy effects? In case I decide to segment my users by new vs. returning and see if the effect of my change is the same for both new and returning users (by testing these four combinations)Generally yes, this is the rough approach to estimating "heterogeneous treatment effects" where you look at treatment effects within dimension slices, and then specifically run a test for whether the effect in group A is different from the effect in group B.