https://www.growthbook.io/ logo
#ask-questions
Title
# ask-questions
g

green-helicopter-75175

12/18/2023, 11:23 AM
Hi 👋, how long does you normally have to run a A/A test do see a ‘null’ result? I have been running a test almost 1 month now, and I’m having a 95,5% chance to win on a invisible change. (change is 1,88% ⬆️ )
This is how my Metrics looks like after the last update from Matomo:
f

fresh-football-47124

12/18/2023, 4:06 PM
interesting - what is the sample size?
does your data have a lot of outliers?
h

helpful-application-7107

12/18/2023, 9:47 PM
Hi Thomas. This kind of result can often occur by chance. Indeed, these data indicate a 2-10 percent chance this occurred by chance. If you already have a lot of traffic early in your experiment, you might not ever get a null result even if everything is set up well. This can happen if in those early days you got unlucky and got a bad split with a lot of traffic.
I suggest restarting your AA test (note: you cannot just add a new phase)! Unfortunately we don't have a super easy way to do this. You can do this by going to Edit Targeting and maybe changing your targeting to 99% or so, then staring a new phase (!)with re-randomization(!) You can take this time to dig into your setup again in case there is an issue, but this could easily be a false positive as well.
Alternatively you can let this run and just start another one. Then on average that 1.88% in that experiment should come down.
g

green-helicopter-75175

12/19/2023, 7:38 AM
Sample size. Please see image. Regarding outliers, how can I check that? But I guess if there is any outliers, then they would occur on both variants?
Since I was having problems understanding the first AA test, I did started two more AA tests on the 7. of December. And here is the data from them:
r

rhythmic-agent-34208

12/19/2023, 9:59 PM
Hi Thomas, you can mitigate the effects of outliers by using capped values: https://docs.growthbook.io/app/metrics#capped-value
h

helpful-application-7107

12/19/2023, 10:16 PM
Hey Thomas, given your other two tests it looks likely that your first set of results was a false positive. This happens from time to time when running A A tests!
g

green-helicopter-75175

02/27/2024, 8:47 AM
Hi @brief-honey-45610 and @helpful-application-7107 , thanks for your prev response. But we are still having trust problems to our GB setup. And now, after 2 months of not using the service (just collecting data), we where suppose to check the results, and we cannot check the results because the queries cannot run because of ‘no space left on device’. Our disk is quite big, and last time I checked we had 74GB free.. Maybe our metric queries is not 100% perfect, but still, our server guy said that it is many heavy queries that is running. So now we cannot get any test results. Do you have anyone that can check our setup to verify that it is correct? Example of an error:
h

helpful-application-7107

02/27/2024, 2:18 PM
Hey Thomas. Are you using MySQL? Can you also paste your query here? (feel free to remove your actual table paths if you would like, or DM me if you don't want to share it in this thread).
g

green-helicopter-75175

02/27/2024, 2:23 PM
Have DM you 🙂
5 Views