https://www.growthbook.io/ logo
#give-feedback
Title
# give-feedback
b

brash-raincoat-88597

02/03/2023, 3:08 AM
Hello. I have been using Growthbook for our analysing our experiments and one of the analysis seem to behave weird. This experiment has run for a few days, but the violin plot is just one line at the center, the Chance to Beat is always 50%, and the risk of choosing either control or variant is always zero. We already have quite a number of users and the mean of the metric is quite different for the variant as compared to the control. I wonder how I can troubleshoot if there’s anything we did wrong in the way we set this up.
f

fresh-football-47124

02/03/2023, 4:16 AM
can you share the numbers or screen shot?
b

brash-raincoat-88597

02/03/2023, 4:26 AM
Control: 33502 users, mean 161.790 (5420287/33502) Variant: 6178 users, mean 177.191 (1094683/6178)
f

fresh-football-47124

02/03/2023, 4:34 AM
ya, that seems right - you have very high traffic - what kind of metric is it?
b

brash-raincoat-88597

02/03/2023, 4:36 AM
The type is revenue, the rest is set as default. A user may have multiple values, and the values are averaged.
f

fresh-football-47124

02/03/2023, 4:40 AM
can you just make sure the metric definition is set correctly to revenue and has the right aggregation?
b

brash-raincoat-88597

02/03/2023, 4:46 AM
Yeap it was correctly set. It seems I figured out why. There were crazy outliers that skewed the standard deviation (the Control’s STDDEV is 3141 and Variant’s is 3868). After I cap the values in the metric and updated the data, the violin plots and risk of choosing started showing non-zeros.
The
View Queries
inside the three dot menu is very useful 👍
f

fresh-football-47124

02/03/2023, 4:47 AM
cool
winsorization
b

brash-raincoat-88597

02/03/2023, 4:50 AM
yeah. i may need to discuss with the stakeholders on what to cap it to, but it seems to work well. thanks for the help.
10 Views