Hi :wave:, how long does you normally have to run ...
# ask-questionsg
green-helicopter-75175
12/18/2023, 11:23 AMHi 👋, how long does you normally have to run a A/A test do see a ‘null’ result? I have been running a test almost 1 month now, and I’m having a 95,5% chance to win on a invisible change. (change is 1,88% ⬆️ )
green-helicopter-75175
12/18/2023, 1:54 PMThis is how my Metrics looks like after the last update from Matomo:
f
fresh-football-47124
12/18/2023, 4:06 PM
interesting - what is the sample size?
fresh-football-47124
12/18/2023, 4:07 PM
does your data have a lot of outliers?
h
helpful-application-7107
12/18/2023, 9:47 PMHi Thomas. This kind of result can often occur by chance. Indeed, these data indicate a 2-10 percent chance this occurred by chance. If you already have a lot of traffic early in your experiment, you might not ever get a null result even if everything is set up well. This can happen if in those early days you got unlucky and got a bad split with a lot of traffic.
helpful-application-7107
12/18/2023, 9:51 PMI suggest restarting your AA test (note: you cannot just add a new phase)! Unfortunately we don't have a super easy way to do this. You can do this by going to Edit Targeting and maybe changing your targeting to 99% or so, then staring a new phase (!)with re-randomization(!)
You can take this time to dig into your setup again in case there is an issue, but this could easily be a false positive as well.
helpful-application-7107
12/18/2023, 9:52 PMAlternatively you can let this run and just start another one. Then on average that 1.88% in that experiment should come down.
g
green-helicopter-75175
12/19/2023, 7:38 AMSample size. Please see image. Regarding outliers, how can I check that? But I guess if there is any outliers, then they would occur on both variants?
green-helicopter-75175
12/19/2023, 7:43 AMSince I was having problems understanding the first AA test, I did started two more AA tests on the 7. of December. And here is the data from them:
r
rhythmic-agent-34208
12/19/2023, 9:59 PMHi Thomas, you can mitigate the effects of outliers by using capped values:
https://docs.growthbook.io/app/metrics#capped-value
h
helpful-application-7107
12/19/2023, 10:16 PMHey Thomas, given your other two tests it looks likely that your first set of results was a false positive. This happens from time to time when running A A tests!
g
green-helicopter-75175
02/27/2024, 8:47 AMHi @brief-honey-45610 and @helpful-application-7107 , thanks for your prev response.
But we are still having trust problems to our GB setup.
And now, after 2 months of not using the service (just collecting data), we where suppose to check the results, and we cannot check the results because the queries cannot run because of ‘no space left on device’. Our disk is quite big, and last time I checked we had 74GB free..
Maybe our metric queries is not 100% perfect, but still, our server guy said that it is many heavy queries that is running.
So now we cannot get any test results.
Do you have anyone that can check our setup to verify that it is correct?
Example of an error:
h
helpful-application-7107
02/27/2024, 2:18 PMHey Thomas. Are you using MySQL? Can you also paste your query here? (feel free to remove your actual table paths if you would like, or DM me if you don't want to share it in this thread).
g
green-helicopter-75175
02/27/2024, 2:23 PMHave DM you 🙂
26 Views