Hey all,
We just started our first real experiment last night after a successful A/A test. The A/B experiment targets desktop only, whereas the A/A test was all users. This is the only difference we can see between them, yet the A/B test is receiving no data at all.
I've tried making changes to the A/A test so as not to make things worse with the A/B test, and it just got stranger:
• Added attribute-targeting for desktop only in a new phase - the experiment went unhealthy with a 77/23% split, and if I changed the dimension to device I see data for phone and tablet
• Changed the attribute-targeting to chrome only in a new phase - the experiment health changed to a 98/2% split! Viewing with the dimension to browser showed data for other browsers
• Removed the attribute-targeting completely - the experiment was still unhealthy with a 54/46% split
• Created a new phase with no changes - healthy!
Each time I created a new phase I re-randomised traffic.
We are at a loss for what is causing this, as nothing seems to be a constant