A five-part data-quality profile flags bad clinical training data, but cannot rank how much harm each defect does

Data-quality assessment moves from isolated statistics such as missingness rates and class ratios to a five-part profile tied to a specific task and prediction target, with each part paired to a measured change in a trained model. The prevailing practice reports feature-level missingness, class-imbalance ratios or demographic tables as proxies for dataset fitness without connecting them to model behavior. On three MIMIC-IV cohorts, sepsis (N=2,000), ICU mortality (N=3,000) and KDIGO-staged acute kidney injury (N=2,293), every controlled corruption condition trains a logistic regression and measures log-loss and expected calibration error change on a held-out target set.

The composite score moved in the expected direction in all three controlled cohorts, while each of three common single-number heuristics pointed the wrong way in at least one cohort. In the mortality cohort label noise pushes the observed class balance toward even, so an imbalance monitor reads that corruption as an improvement; in the sepsis cohort the raw missingness rate ranks the missingness condition above the institutional shift, although the shift produces the larger log-loss gap (0.127 versus 0.007). Spearman rank correlations over four conditions per cohort give 0.800 for the composite degradation score in all three cohorts, while the missingness, imbalance and parity-gap monitors are each wrong-signed in at least one cohort.

The temporal-integrity component detects leakage, but the apparent discrimination gain from leaked features does not persist on clean deployment data. With future information mixed into training, apparent AUROC exceeded the clean reference by 0.025 (95% interval 0.011–0.039), while the same model on clean deployment data was 0.000 different from the clean reference (interval −0.009 to +0.007). In a separate train-and-test leakage experiment on the mortality cohort, late-window creatinine and blood urea nitrogen were substituted into admission features, and across 20 repeated stratified splits the deployment penalty was concentrated in a 0.009 log-loss worsening (interval 0.002–0.017), with a calibration interval spanning zero.

Across 21 analyzable eICU hospitals the composite degradation score was associated with per-site calibration but not detectably with discrimination. The unadjusted correlation between the degradation score and expected calibration error was 0.60, falling to 0.45 after controlling for site size, 0.29 after also controlling for mortality prevalence, and 0.17 (interval −0.22 to +0.52) when sites were equalized in size. A 300-replicate null experiment gave a median correlation of 0.21, placing the observed 0.60 at the 94th percentile (P=0.057), so estimation artifacts alone can produce a substantial positive correlation.

The demonstrated scope is an exploratory benchmark on three controlled MIMIC-IV cohorts and the multi-institution eICU database, with logistic regression as the primary model and tree models as sensitivity checks; the author lists leave-one-component-out ablations, site-held-out evaluation, false-reassurance estimation and a budget-matched comparison of remediation policies as prerequisites for any further predictive or actionable claim.

The author reports that removing the clean baseline drops the rank correlation between the composite and measured gaps from 0.800 to 0.500 in both the sepsis and mortality cohorts; in the leave-one-family-out evaluation the full component vector's mean advantage was +0.009 (interval −0.035 to +0.051) over the best scalar baseline and +0.005 (interval −0.020 to +0.042) over an always-zero predictor, so the null of no added value cannot be rejected at five families. The label-noise component returned 0.997 under a 100% label flip on separable data and 0.884 on clean nonlinear data, which the author cites in calling it a proxy rather than a noise-rate estimate.