Quick question about whether it's possible to convert the Bayesian ChanceToBeat to a frequentist p-value. The robots are saying "you can informally think of Chance to Beat Control as roughly equivalent to 1 minus the p-value." I'm somewhat skeptical of that conversion. But is it true-ish? 🙂 cc @helpful-application-7107 @strong-mouse-55694 ps. I'm building an API based tool to scan past experiments, looking for replication worthy signals. https://www.linkedin.com/posts/jlinowski_experimentation-replication-optimization-activity-7405330721030041600-KeYL/ if anyone is interested in giving it a spin, ping me.

he robots are saying "you can informally think of Chance to Beat Control as roughly equivalent to 1 minus the p-value." I'm somewhat skeptical of that conversion. But is it true-ish? 🙂

It is true-ish for positive effects, but only if the user is not using flat priors. CTW = 97.5% is strong evidence for a positive effect, this is ~similar (in a very loose sense) to a positive lift AND a 0.025 p-value. CTW = 2.5% is strong evidence for a negative effect, this is ~similar (in a very loose sense) to a negative lift AND a 0.025 p-value. CTW therefore is also encoding the direction of the lift, and there is no direct comparison between the two of them, and the more "correct" this rough conversion is the further you get from 50/50.