https://www.growthbook.io/ logo
m

mysterious-iron-16289

04/21/2023, 10:06 AM
Hi, we run Build: f58a91e (2023-03-31). When using the frequentist statistics, we still get the Bayesian behaviour for the metrics (see screenshot). This means we can’t change the critical p-value for our results to be considered significant, as one could in previous versions. Is anyone else observing this behaviour?
Update: I can still set the critical p-vaue on the global level in the gerneal settings, but not for the individual metrics. Individual metrics let me configure bayesian behaviour, though.
h

helpful-application-7107

04/21/2023, 4:00 PM
Update: I can still set the critical p-vaue on the global level in the gerneal settings, but not for the individual metrics.
Yes, that's a limitation of our settings. However, the analog to the critical p-value threshold in the bayesian engine is also only configurable at the global level (Confidence Level). The Sample Size and Max/Min Percent Change Thresholds apply to both engines, while the Risk thresholds only apply to the Bayesian engine
m

mysterious-iron-16289

04/24/2023, 6:57 AM
Ahoi @helpful-application-7107. thanks for the answer! THisBased on your input, there is a difference between the expected behaviour and how the current GrowthBook version behaves. The risk thresholds for metrics are still configurable even when using the frequentist engine. I created a bug ticket for this, since I didn’t address it explicitly my original message.
h

helpful-application-7107

04/24/2023, 12:25 PM
I'll respond in the ticket. I agree it may not be the best user experience, but I'm not sure I would categorize this as a bug.
m

mysterious-iron-16289

04/24/2023, 12:26 PM
OK, I’ll follow the comments on the ticket.
3 Views