Hello! Is it statistically safe to run an experiment covering both desktop and mobile web users ("device" dimension) and then analyze results by filtering by Device Dimension ? The other - more conservative - approach would be to run two differente experiments, one targetted to "device = desktop" users, the other to "device = mobile" users. The cons I see about runnning two different experiments is that we will need more time to gather sufficient data. I am using a GA4 Big Query Data Source.

Hi Federico, I am a data scientist at GrowthBook. This is a good question. You should run just 1 experiment. If your feature causes some of your mobile users to start using desktop (or vice versa), then your dimension-level results can be biased. Your aggregate results across all dimensions should be fine either way. If you see an SRM failure at the dimension level, but not for overall results, then this is an indication you should not analyze by device type. Hope this helps, Luke

If your feature causes some of your mobile users to start using desktop (or vice versa), then your dimension-level results can be biased.

It should be noted that if you use the built-in

device

dimension from the EAQ, this is not a concern, because we always take the first

device

seen for each user in the experiment.