https://www.growthbook.io/ logo
Join Slack
Powered by
Hey Growthbook team! I would like some help unders...
# experimentation
p
Hey Growthbook team! I would like some help understanding why our 1st bandits experiments are not capturing data. Our exploratory stage lasted 3 days and we have been running a couple of experiments like these for almost 1 month with the same issue. I can see traffic but no data coming from our main decision metric. I have checked w/other data sources and I can confirm that this is not reflective of what's happening (the decision metric is moving for those exposed). It's our 1st time running this type of experiments so would be great to have some guidance, pls!
d
To give some more context it seems like we're having trouble in this part of the bandit query
Copy code
__banditPeriodWeights AS (
        SELECT
            bps.bandit_period,
            bps.dimension,
            SUM(bps.users) / MAX(dt.total_users) AS weight
        FROM
            __banditPeriodStatistics bps
                LEFT JOIN __dimensionTotals dt ON (bps.dimension = dt.dimension)
        GROUP BY
            bps.bandit_period,
            bps.dimension
    )
Where the 
bps.users
is magnitudes smaller than 
dt.total_users
so the weight is 0 and that makes 
main_sum
0 as well. What's cocnfusing to me is that this seems to be expected though since 
total_users
is the sum of 
users
for all the periods
p
@strong-mouse-55694 would love your perspective to see what the issue could be and we can make progress on this. Let us know if there's a better channel to post this! TY!
h
Hey!
The weight shouldn't be 0, but it should be small.
Have you copied out the query and run it with just this part of the SQL to check if it's returning 0? I would think the most likely explanation is actually that the metric data isn't being joined for some reason, but there are two things that can help us debug it. Can you click the three dots in the top right of the Update button, click View Queries, and paste the table for the main decision metric here? Can you also check to see what happens if, in Redshift, you run just the part to 
__userMetricAgg
and return that in Redshift? Do you see users with any conversion rates there?
d
Could it be an issue of it being an int division instead of float? I ran manually and doing 
SUM(bps.users)::FLOAT / MAX(dt.total_users) AS weight
makes the weights seem normal. We're indeed using Redshift, I'll copy some data when I have a chance here for debugging
h
Oof, it could be that, yes.
Could it be an issue of it being an int division instead of float? I ran manually and doing 
SUM(bps.users)::FLOAT / MAX(dt.total_users) AS weight
makes the weights seem normal.
I'm pretty sure we ran tests on our redshift set up, but let me check.
Yes, confirmed this is a bug with Redshift. Our apologies, we are working on a bugfix.
We've just landed a fix here. you can use the check results refresh to see if it's resolved. This means that your bandits won't have been updating and re-allocating traffic, but should begin doing so now as if they had a very long update burn-in window.
🙌 2
d
All good, thanks Luke!
p
Thanks @helpful-application-7107!
2 Views
Open in Slack
Previous
Powered by