Hello! I'm doing a test run of growthbook just lik...
# experimentation
s
Hello! I'm doing a test run of growthbook just like @little-balloon-64875 couple months ago and hitting the same issue. I've added around 20-30 events and I have 10 total users in experiment, but I'm not seeing a report graph. I'm using bigquery with GA4. I've set up fact table and metrics. My goal is as following: in our app we have two export versions: 3.0 and 4.0. And we have a glow effect around the 4.0 export. So I want to see, if I turn off that glow effect, would I get less 4.0 exports or not. Practically it probably doesn't need to be compared to 3.0 I guess, all I care is increase or decrease of number of 4.0 exports. Can anyone tell if I'm doing anything wrong here? Maybe there's amount of users/events threshold? I've set up minimum metric total to be 0 so it should pick the data up. In posthog, as comparison, it was quite easy and just 4-5 users in different groups were enough to see the chart. Sharing to give better picture of what I'm trying to get. (Ignore the warning, I did that kinda on purpose)
s
Howdy! I'm not exactly sure what you're having trouble with, but it looks like things are set up correctly. The main issue I can see is that with only 10 users, you won't be able to have any real results. A general rule of thumb is to have at least 100-200 conversions per variation.
s
@strong-mouse-55694 thanks man. By 100-200 conversions you mean users or events? Like, say, in current situation at most I can make 30-ish internal users to generate 2-3 key events, which makes 60-90 events. I can try to push more manually through my own account and couple of other test accounts, but I would barely do 100-120ish events in a day or two. So I wanna know if I'm even in a situation where I can get what I need or I have to start using experiments in my production blindfolded.
s
So, if you're trying to optimize for exports, you'd need 100-200 exports per variation before you could reach statistical significance. But there a few different factors at play: https://docs.growthbook.io/faq#how-much-traffic-do-i-need-to-run-ab-tests Is it the case you only have 30 internal users but your external user base is larger? Or is it just 30 users?
s
it's around 30-ish internal that I can involve, prod user base is much larger:)
1
okay I'll give it a shot, will find a way to trigger 100-200 exports. Thanks!
l
I will say that once I was confident enough that data was flowing through, I ran an "invisible" test on our homepage where we get the majority of our traffic, both Control and Variant 1 were hidden divs but it still fired off events saying the experiment was viewed. This was just to get a better sense of how it was receiving and handling data before we ran an actual test.
🙌 1
s
oh this is good approach, thanks @little-balloon-64875. We might consider it as first prod test option, but I wanted to make a visually appealing PoC in QA. Thanks anyway:)
🙌 1
s
Another practice we encourage here is to run an A/A test, which ensures that all your fundamentals are working: https://docs.growthbook.io/kb/experiments/aa-tests#what-is-an-aa-test