A protective effect in observational data may be all bias: new metric attributes a 16.9% mortality drop to systematic error

Observational causal studies can now report a third quantity: how much of an observed association comes from systematic bias, rather than only the effect size or whether a signal exists. Earlier practice either reported a relative risk with a confidence interval, answering only how large the effect is, or a calibrated P-value, answering only whether a signal survives systematic error, leaving the analyst to judge whether the residual estimate is bias-dominated. Across 1,920,000 simulated repetitions, the calibrated P-value alone declared 36.02% of truly bias-dominated repetitions usable, adding the BAF point estimate lowered this to 16.48%, requiring the whole credible interval below 0.5 lowered it to 10.16%, and adding a bias-dominance probability below 15% lowered it to 3.02%.

BAF writes the bias share as a continuous quantity between 0 and 1 with a credible interval, turning a binary credibility judgement into a three-channel report. The calibrated P-value returns a binary answer that conflates whether a signal exists with whether the signal is credible; BAF reuses the same empirical null but reports the share of the calibrated signal attributable to bias. In the MIMIC IV beta-blocker target trial emulation, the empirical null from 12 negative controls had center -0.168 and scale 0.113; question 1 gave BAF 0.875 with a 95% credible interval of 0.385 to 0.992, and question 2 gave BAF 0.460 with an interval of 0.186 to 0.619.

Even under the strictest attainable observational design, an association repeatedly reported as protective can be entirely explained by bias. The emulation followed the TARGET guideline, used doubly robust targeted maximum likelihood estimation, and carried 12 negative controls, a combination often treated as sufficient for effect conclusions. Question 1 gave an uncalibrated one-year relative risk of 0.831, a 16.9% mortality reduction, which became a calibrated relative risk of 0.983 with a calibrated P-value of 0.894 and a verdict of not usable as effect evidence; leave-one-out, learner-library replacement, and propensity-score perturbations left that verdict intact.

The metric targets observational causal studies that use a negative-control panel; the simulation spans true protective effects from marginal to large with bias centers matching healthy-adherer magnitudes, and the case study draws on adults with a first ICU stay and a recorded 12-lead ECG in MIMIC IV between 2008 and 2019. The two clinical questions illustrate the rule's behaviour and are not powered to answer the underlying clinical questions about beta-blocker therapy, which the authors state require replication in independent data.

BAF inherits the bias-exchangeability assumption of the calibration framework, which is not testable in any single study; a 30% violation in the simulation raised the misuse rate from 1.39% to 4.36%. The interval is a credible interval rather than a confidence interval and carries no frequentist coverage guarantee, with realized coverage of 55.7% in the merged simulation pool against a nominal 95%. Agreement with truth is moderate (concordance correlation coefficient 0.838) with significant proportional underestimation that grows with the true value, so direct interpretation of the BAF point estimate is not advised and the verdict relies on a truth-calibrated lookup table. That lookup was built from the same 1,920,000 repetitions used to evaluate the rules, so its reported performance is in-sample agreement, bounded by held-out validations.