Hi @future-teacher-7046, @fresh-football-47124 and @helpful-application-7107, we’re using the enterprise feature for multiple hypothesis corrections. Concretely, we decided to use to control the false discovery rate (FDR) with your implementation of the Benjamini-Hochberg method. We found that showing the adjusted p-values with the unadjusted confidence intervals is highly confusing to a user. Would you consider adjusting the confidence intervals, too? As I understand it, you can have simultaneously adjusted p-values and confidence intervals for the adjusted critical p-value per metric as well without making a math error. You just scale the individual confidence levels per hypothesis. FYI @adventurous-dream-15065

Hi @mysterious-iron-16289, unfortunately, there are not directly analogous CIs for the Benjamini-Hochberg procedure (as far as I know) given the way the procedure works. Our FWER control method (holm-bonferroni) also does not have analogous CIs. The simple Bonferroni correction does, but that test is considerably more conservative. However, I understand that seeing a non-statsig p-value next to a CI that is completely on one side of 0 can be confusing.. We could consider hiding CIs, an alternative procedure that maybe does have analogous CIs but is less widely adopted, or doing something ad-hoc to the CIs, but for these widely used methods with good statistical properties, there are not directly analogous CIs.

Really? I thought that you just would calculate the critical p-value instead of an adjusted p-value - the formula is the same. I attached the formula I was thinking about. Denoting the k-th hypothesis in an ordered list of m experiments, where _p_k_ is the p-value of the k-th experiment, alpha the critical value. The values with the tilde are adjusted. You basically provide \tilde{p_k} right now. And when you’re smaller than alpha, then you should have a CI with 0 lying outside of the interval with hypothesis confidence level 1 - \tilde{\alpha}/2. Or am I mistaken?

And when you’re smaller than alpha, then you should have a CI with 0 lying outside of the interval with hypothesis confidence level 1 - \tilde{\alpha}/2. Or am I mistaken?

This sounds correct to me. But the question is what is the general way you can construct that CI?

I know, BH has some more steps, so that you can have significance at the k-th level even when the inequality is not true, given the k+n-th hypothesis becomes significant according to that formula.

Yeah, and these kinds of stepped procedures, to my understanding, don't have analogous CIs.

That’s true but the p-value has an ci equivalent. As I understand it, a confidence interval is related to the p-value in the sense that the difference between the measured difference and 0 is given by the attached formula. SEM is the standard error of the mean and Z the critical value at c.l. 1-p/2. Hence, at the critical value p=alpha the CI touches 0. When p<alpha its smaller, 0 is outside the CI, when p>alpha it’s inside. So for me it looks like you could use this to construct the CIs.

Hmmm, I'm not sure that would work, but I can think about it some more. Benjamini and Yekutieli did a lot of work on corrected CIs that, when reading it, seemed to me to further my assumption that there were no appropriate CIs for the BH procedure itself.

The above formula certainly works, if you have just one hypothesis. And regarding the interpretation, it should be viable if the hypotheses are uncorrelated. However, BH assumes zero or positive correlation, where the latter might affect the statement. It would be good to find out what the key point of Benjamini and Yekuieli were to investigate the point further. Maybe they even make a statement why this naive approach is not working. We can get back to it tomorrow, if you like. We would really like to have a correct visualisation, since they current can be misleading.

The above formula certainly works, if you have just one hypothesis.

But, this is the crux of the issue, no?

Let us know if you have an idea about this topic. If we come up with something, we can notify you, too.

Yeah, this is the paper I was referencing earlier. They talk about how this works for BH Selected CIs, and don't seem to discuss in general the case where we need to construct CIs for both rejected and non-rejected hypotheses. Maybe their implementation is enough not to matter and CIs constructed in this way for non-rejected hypotheses will be in line what what we'd expect.

I just realised that page 74 was absent in my copy of the paper 😱 Now it’s way easier to understand. I see your point. However, the correction suggested is connected to FDR, a type I error. Adjusting the FCR for not selected CIs would be a type II error correction. I would argue that from the point of controlling the FDR it is not relevant how wide exactly the CI for the non-selected parameters are. It’s just important, that they encompass 0. What do you think?

I would argue that from the point of controlling the FDR it is not relevant how wide exactly the CI for the non-selected parameters are. It’s just important, that they encompass 0.

I agree somewhat. From the perspective of building a reliable machine used by many different companies and analysts, I am a bit concerned about ad-hoc CIs for these non-selected tests. That said, this is probably better than the status quo: unadjusted CIs & a tooltip warning that they aren't adjusted.

I agree, ad-hic adjusted with tooltip is way less misleading than the status quo. Regarding proper CI: we could, potentially, write a mail to the authors of the paper and ask for their opinion. I guess they have thought about the non-selected CIs at least a bit. Even if it is not necessarily an interesting topic for their research. Worst case we get no answer 😉

@helpful-application-7107 do you have an update on this issue? The Black Friday season is coming up and we are looking forward to share results with a wider audience than just experimentation pros. For that reason it would be great, if the visualizations in GrowthBook more closely resemble the numbers. Thank you!

No! I'm sorry 😕 I've tried to turn to it a couple of times but we've prioritized other work.

FYI, @mysterious-iron-16289

Hi @adventurous-dream-15065, we implemented adjusted CIs as default for tests with corrected p-values. When the p-value is 1, we extend the CI range to be infinite, which follows the ad-hoc adjustment we are using. We also return the unadjusted CIs in the tooltip. Feedback is welcome.

That sounds amazing, I will look at the result on Wednesday and will get back to you with feedback! FYI, @mysterious-iron-16289