Hi, we've integrated Growthbook in our repo(React+...
# sdk-reactl
little-animal-79425
12/16/2023, 8:45 AMHi, we've integrated Growthbook in our repo(React+Redux) and were running a simple A/A test to check if we'd done the setup correctly. But for one GA event the results are skewed even though growthbook is getting the right count of data for the same event under Metrics tab. We confirmed the metric count is correct by querying the DB but on querying the DB for the experiment result the count is different from what's shown in Growthbook. Does anyone have any idea what could be causing the experiment result to be skewed?
r
rhythmic-agent-34208
12/20/2023, 1:09 AMHi Niraj, thanks for your patience!
- Are you using GA4 and BigQuery?
- Are you using the cloud-hosted or self-hosted version of GrowthBook?
- Could you send some screenshots that show the full screen of the unexpected results?
- Can you show me the Metric's SQL (screenshot or copy-paste code snippet)?
l
little-animal-79425
12/21/2023, 3:21 PMHi August,
1. Yes, I'm using GA4 and Big Query
2. Self hosted version
little-animal-79425
12/21/2023, 3:26 PM3. Here's the screenshot of the experiment results
little-animal-79425
12/21/2023, 3:27 PM4. Here's the
donation Copy code
SELECT
user_id,
user_pseudo_id as anonymous_id,
TIMESTAMP_MICROS(event_timestamp) as timestamp
FROM
<analytics_table>
WHERE
event_name = 'donation'
AND _TABLE_SUFFIX BETWEEN '{{startYear}}{{startMonth}}{{startDay}}' AND '{{endYear}}{{endMonth}}{{endDay}}'r
rhythmic-agent-34208
12/29/2023, 9:14 PMHi Niraj, thanks for your patience over the holidays.
In this particular case, the Goal Metric that is causing problems ("donation") has a very small baseline of only 0.02% (100 users out of 464,000). The Variation is at 0.03% (119 users out of 463,000). So it's a difference of only 19 users on a very small sample size and it's causing a 90% chance-to-win.
This can be attributed to expected statistical variation because the sample size of users is so small (around 100 to 120).
We usually recommend at least a 95% threshold since you will see 90% on average once out of every 10 A/A tests just by random chance. With 95%, it only happens 1 out of 20 times
l
little-animal-79425
01/11/2024, 7:07 PMHey, so looking at it after the holidays the chance to win has 98.5% and the gap keeps increasing. So I'm not sure if the 95% threshold will work because it's cross it now. I've attached the latest experiment result screenshots
little-animal-79425
01/11/2024, 7:07 PMimage.png
little-animal-79425
01/11/2024, 7:10 PMYes I can understand it's a small sample size but we're a bit unsure as to whether we've set up growthbook properly or not. Looking at the above results in a sense makes things even more confusing
r
rhythmic-agent-34208
01/12/2024, 1:52 AMHi Niraj, I've asked our Data Scientist to chime in here. He should be able to comment during US business hours tomorrow.
rhythmic-agent-34208
01/12/2024, 5:11 PMHey Niraj, in this case it's likely that this result is just due to chance. It's hard to know for sure, but it's possible you just got unlucky with a few more `donation` metrics in the variation than in the control.
I don't see anything else that would likely cause the deviation, especially if you're sure that the test is an a/a test.
The easiest way to confirm would be to click Make Changes -> Start a New Phase -> New Phase and re-randomize and see if the results look better on a second run.
l
little-animal-79425
01/17/2024, 2:37 PMI see. I'll take another look at it
38 Views