CT pretraining favors what context can predict: pathology signal fades in hidden-target prediction while scan-level calls barely move
Synopsis
A ViT-B/3D encoder trained on 113,611 thoraco-abdomino-pelvic CT volumes, TAP-JEPA, matches state-of-the-art CT encoders on thoracic and abdominal classification benchmarks; replaying its exact masking, teacher and predictor shows hidden-target prediction lowers mean Skill from 0.388 to 0.271 for pathology but only from 0.921 to 0.873 for anatomy. Pathology moves 34.7% toward matched controls along the teacher direction while controls move 13.6% toward pathology, with larger shifts farther from visible context, and deleting all final-layer pathology tokens changes the decision margin by at most 0.45% in four of seven scan-level probes.
Interpretation
In CT self-supervised pretraining, a predictive objective retains far more anatomical than pathological information, and that gap can be recomputed inside the training objective itself. Prior transfer benchmarks show what a representation supports downstream, not which local information the predictive objective preserves; this work compares recurrent anatomy and sporadic pathology within one encoder after hidden-target prediction. On ATLAS and ReX, both unseen during pretraining, the normalized Skill shortfall is 36.7% and 31.8% for pathology versus 11.2% and 6.2% for anatomy, a paired difference of 25.5 and 25.6 percentage points.
Hidden-target prediction pulls pathology features toward matched normal controls in the same organ, and the pull grows as visible context moves farther away. Annotated pathology had not been traced through the training objective into the prediction output, so the relative displacement of pathology and control in prediction space was unquantified. Separation is attenuated in all six pathology categories, from 17.9% for scarring/fibrosis to 72.4% for liver lesions; pathology-side displacement exceeds control-side in five categories, for example atelectasis/consolidation at 54.5% attenuation with 48.2% pathology-side and 6.3% control-side, and liver attenuation rises from 67.4% below 24 mm to 83.4% at 96 mm or more.
A visible pathology region contributes less than an equally sized matched control region to predicting surrounding hidden anatomy. Target-side asymmetry shows what pathology is predicted to look like, but the context side still needed a test of whether visible pathology is treated as useful evidence. On shared hidden targets, replacing control features raises loss more than replacing pathology features in all six categories, with four differences significant after joint correction: liver, scarring, atelectasis and ground-glass.
Scan-level pathology decisions rely only partly on the local pathology tokens, and acquisition metadata or co-occurring labels alone carry much of the same predictive power. It was unclear how much a scan-level decision depends on the lesion's own tokens. Scan-level Skill spans 0.198–0.607, while deleting all final-layer pathology-overlapping tokens reduces the reference margin by at most 0.45% for liver, kidney, scarring and nodules; probes using only acquisition metadata or co-occurring labels reach 0.68–1.24 times encoder Skill in five categories.
Perspective
The analysis characterizes predictability bias in one large-scale 3D CT JEPA, pretrained on 113,611 thoraco-abdomino-pelvic CT volumes from NLST, CT-RATE, Merlin and AbdomenAtlas, with regional analyses on 1,863 unseen ATLAS abdominal scans and 950 ReX thoracic scans. The downstream interventions measure direct dependence on final-layer pathology tokens, and the context-side analysis concerns full replacement on shared surrounding targets. The authors propose tracing where local pathology information is retained, redistributed or attenuated across representation depth and perturbation strength, and using controlled changes to masking and predictive objectives to identify which design choices preserve context-unresolved clinical variation.
The authors scope the results to one large-scale 3D CT JEPA and note that the downstream interventions measure direct dependence on final-layer pathology tokens while the context-side analysis covers full replacement on shared surrounding targets. The association between positional typicality and local readout dependence is described as exploratory, consistent in direction with the predictability-bias hypothesis but not causal. Kidney is reported descriptively because it fails the support gate, and the nodule replacement difference does not remain significant after joint correction. The authors also note that acquisition metadata and co-occurring labels are themselves predictive, so scan-level Skill does not localize to the lesion itself.