95% intervals calibrated on random splits fail outside the training envelope: hard-shift coverage falls to 0.40–0.75 across seven engineering datasets

Uncertainty quantification for engineering regression now has a comparison graded by deployment shift: the same datasets are cut into in-distribution, soft-shift and hard-shift rungs along curing age, glazing area ratio, ambient temperature, Froude number, ship speed, fault mode or calendar time, rather than by random split alone. Earlier comparisons of regression uncertainty quantification relied on random train–test splits, which make training and test points exchangeable by construction, exactly the condition under which conformal methods guarantee coverage and other methods are easiest to calibrate. On seven public engineering datasets, all four conformal methods reach at least 0.946 coverage on every dataset at the in-distribution rung, the lowest being weighted CQR at 0.946 on C-MAPSS.

As shift grows, marginal coverage hides local failure: under soft shift most methods lose only a little average coverage, but worst-subgroup coverage sits 0.05–0.10 below the mean. Marginal coverage can stay near nominal while some operating regimes are covered far less often; for Concrete, Yacht and Naval the worst subgroup is the part of the target furthest from the source. The deep ensemble covers 0.91 on average and 0.81 in its worst subgroup, the Gaussian process 0.87 and 0.79, and Split CP 0.89 and 0.79; only Conformal GPR (0.94 and 0.90) and weighted CQR (0.95 and 0.91), whose widths respond to the shift, keep the gap small.

Under hard shift, mean coverage falls to 0.40–0.75 for the 11 methods that always return an interval, and a Friedman test detects no systematic difference in method ranks. At the in-distribution rung these methods still come close to nominal coverage; at the hard rung they fail together, and no family is consistently ahead. On the five extrapolation datasets the 11 methods average 0.46 coverage, ranging from 0.00 to 1.00; on the two conditional-shift datasets they average 0.81, ranging from 0.53 to 0.99.

Weighted CQR is the only method that stays near nominal on the points it answers, and it does so by abstaining, with abstention driven by the effective calibration sample size rather than by density-ratio estimation error. In extrapolation regions where the source has no mass the density ratio is unbounded, so no method learning only from source data can guarantee coverage, and abstention is the designed response of weighted conformal prediction. Under hard shift it abstains on 85% of target points on average and on 98% across the five extrapolation datasets; on the six soft-shift datasets with a known closed-form density ratio, exact weights abstain on 28% of points against 26% with estimated weights, with answered-point coverage of 0.94 and 0.95.

The conclusions rest on seven public engineering regression datasets, three from simulators and one from towing-tank tests, with only Air Quality a real field deployment over time; each dataset has a single shift variable and a single threshold, and the hard rung groups support extrapolation, an unseen fault mode and a temporal deployment under one operational severity category.

The hard rung mixes support extrapolation with conditional shift, so extrapolation and conditional datasets are reported separately where the distinction matters; seven datasets give dataset-level tests little power, so a non-significant Friedman result is read as no detectable difference rather than equivalence; weighted CQR's answered-point coverage is conditional on answering and rests on about 8 points per seed on Concrete.