#ask-questions

Title

# ask-questions

s

some-planet-44104

11/17/2023, 2:20 PMQuestion to the community (not directly related to GrowthBook)
Can I use multi-factorial ANOVA for testing for presence of interaction effect in my A/B test?
Say I have two tests running on the same page with 2 variations each (Control and Treatment), this leaves us with with 4 possible combinations of variants if I’m not mistaken. Is it even reasonable to perform ANOVA for detecting interaction in this setting? Would love to hear an opinion on that.

r

rhythmic-agent-34208

11/17/2023, 2:20 PM⭐ Our official support hours are 6:30am - 5pm Pacific Time, Monday through Friday. You may occasionally hear from us outside of these hours.
Your support request has been logged in our system. Our support team will get back to you very soon!

f

fresh-football-47124

11/17/2023, 9:49 PM

Hi Yevhen - this kind of interaction effect detection is on our roadmap. As for the specific technique, I’ll have to let our data science team weigh in.

h

helpful-application-7107

11/21/2023, 6:32 PMYes, but it depends on what you look at in the ANOVA context. ANOVA and regression are basically identical in many ways, so I can easily point out how to do this with a regression.
In a regression context you want to basically run the following regression:

`y = b0 + b1 * t1 + b2 * t2 + b3 * t1 * t2`

where `t1`

is 0/1 for a/b in ab test 1 and `t2`

is 0/1 for a/b in ab test 2. Then if `b3`

is different from 0 there is evidence of an interaction effect.
There's an analog in ANOVA but I find that people describe anova models in different ways that makes communicating with them harder than communicating with regression models.s

some-planet-44104

11/23/2023, 3:17 PMHi! I meant multi-factor ANOVA. that specifically tests for interaction, as described here:
https://onlinestatbook.com/2/analysis_of_variance/multiway.html
I heard that same results can be obtained through regression though.
And I imagine the same approach can be used not only for testing for interaction effect in the context of different experiments on the same page (for instance), but also for testing the presence of novelty and primacy effects? In case I decide to segment my users by new vs. returning and see if the effect of my change is the same for both new and returning users (by testing these four combinations)

Copy code

`old_control, new_control, old_treatment, new_treatment`

Sorry if something sounds not exactly clear - I’m still relatively new to statistics and just trying to find my way aroundh

helpful-application-7107

12/01/2023, 4:05 PMI heard that same results can be obtained through regression though.Yes, I'm just stating it in regression format because I find it easier to communicate about given my backgorund. Multi-factor anova will be identical to regression in this setting.

but also for testing the presence of novelty and primacy effects? In case I decide to segment my users by new vs. returning and see if the effect of my change is the same for both new and returning users (by testing these four combinations)Generally yes, this is the rough approach to estimating "heterogeneous treatment effects" where you look at treatment effects within dimension slices, and then specifically run a test for whether the effect in group A is different from the effect in group B.

2 Views