rich-psychiatrist-1629107/07/2023, 7:45 PM
helpful-application-710707/07/2023, 9:27 PM
rich-psychiatrist-1629107/07/2023, 9:33 PM
helpful-application-710707/07/2023, 9:35 PM
rich-psychiatrist-1629107/07/2023, 9:36 PM
helpful-application-710707/07/2023, 9:36 PM
rich-psychiatrist-1629107/07/2023, 9:38 PM
helpful-application-710707/07/2023, 9:46 PM
rich-psychiatrist-1629107/07/2023, 9:53 PM
helpful-application-710707/07/2023, 9:58 PM
but all of the tests I've run have been false positives with different changes in upliftOnly one of the tests you've showed me is a "false positive" in that it was stat sig. The non-zero uplift that isn't statsig is a feature of your sample size. If you collected more data those uplift percentages should shrink towards zero, but not always.
rich-psychiatrist-1629107/07/2023, 10:08 PM
helpful-application-710707/07/2023, 10:20 PM
the emails have 4 different experiments reaching statsig and it was on a random uplift, and it has happened to all 8 experiments at one point or anotherThis is akin to the well known peeking problem. If you're running 10 a/a tests, and you check them every 6 hours then the probability of false positives is very, very high.
how much more data collection do you estimate would be necessary for them to shrink towards 0? so far they have just shifted in different directions and not towards 0 as I collect more dataIt's hard to say. You're seeing uplift of about 4% in your A/A tests. This is about a 0.5 percentage point absolute difference (e.g. your variations are like 12% and 12.5% conversion rates). In this way, I'd say your effects are pretty close to 0. But let's talk about sample sizes. Imagine this 4% was real. In this case, if the true effect was 4%, in order to reliably get a stat sig result showing that this effect is real, you would need ~75k users per variation! So if you wanted to rule out effect sizes smaller than 4% reliable, you'd need even more than 75k users per variation.
adventurous-computer-4181907/10/2023, 11:35 AM
rich-psychiatrist-1629107/10/2023, 3:40 PM