Output parity alone misses how predictions are built: adding an attribution-disparity penalty improves all six medical-imaging fairness trade-offs

The fairness constraint moves from the model output alone to the feature contributions behind it: an attribution-disparity penalty is added on top of output-level equalized-odds regularization, so different groups rely on more closely aligned predictive evidence rather than merely similar final scores. Prior output-level fairness constrained disparities in predicted scores and errors across groups but did not characterize whether groups relied on similar task-relevant feature contributions in representation space. Across 960 DINOv2-LoRA models trained on HAM10000 and Harvard-GF, adding the same attribution term to each of three output-level equalized-odds surrogates gave positive paired 95% confidence intervals for hypervolume in all six comparisons, with relative gains of 1.49% to 10.57%.

The method writes attribution disparity as a differentiable regularizer trained jointly with the existing output fairness term, with separate coefficients controlling the two terms. The attribution term is normalized by representation dimensionality so the penalty does not scale directly with that dimensionality; output-only and attribution-only settings serve as direct ablations. All methods share a DINOv2-Small backbone with LoRA adaptation and an explicit linear binary head, trained for 20 epochs at batch size 128 with AdamW and cosine learning-rate scheduling; HAM10000 uses five-fold lesion-level cross-validation and Harvard-GF its predefined split with five training seeds.

Under a predefined utility constraint, every augmented model family contains an operating point with lower equalized-odds difference, and those points also show lower attribution disparity and less cancellation. A smaller output gap does not by itself imply more aligned feature contributions, so the analysis also checks attribution disparity and the cancellation ratio as diagnostics. Relative equalized-odds difference fell by 9.0% to 38.7%, AUROC was maintained or slightly improved in all six comparisons, and the size of the balanced-accuracy change is not given as a specific value in the available text.

The conclusions rest on held-out Pareto frontiers over prespecified regularization grids and cover two binary medical imaging tasks: malignant versus benign skin-lesion classification on HAM10000 with gender as the sensitive attribute, and glaucoma detection on Harvard-GF retinal nerve fiber layer thickness images with race as the sensitive attribute. The authors note that the exact output-attribution decomposition relies on a linear classification head and that extension to nonlinear prediction heads requires further study.

Evaluation is limited to two binary medical imaging tasks, although they differ in imaging modality and sensitive attribute, and the exact attribution decomposition depends on a linear classification head. The cancellation ratio is positioned as a diagnostic rather than a fairness criterion, and low cancellation is desirable only when output and attribution disparities are both small.