StreamLTT swaps the risk being certified: on streaming deployments its frontier nears the risk target while false certification tracks the set level
Synopsis
In streaming deployment a fixed policy's loss can drift with observed history, so certifying a fixed population risk no longer describes the risk actually encountered. StreamLTT certifies average predictable risk, the conditional risk of each decision given the information available before its outcome, averaged along the stream, and conditional centering turns empirical deviations into a martingale sum. Across seven forecasting datasets and seven deployment risks, its variance-adaptive version pushes the certified frontier close to the risk target while empirical family-wise error stays near the prescribed level.
Interpretation
The object of policy certification changes from a fixed population risk to a conditional risk averaged along the deployment stream, so certification follows the information available before each decision instead of assuming independent, stationary losses. Standard Learn-then-Test targets a fixed population risk estimated from calibration losses; under dependent and nonstationary sequential deployment, a fixed policy's risk evolves with observed history, and a single stationary marginal risk no longer describes the risks encountered during deployment. Across seven real forecasting datasets and seven deployment risks at δ=0.10, StreamLTT-A has a mean risk gap of 0.0090 versus 0.0302 for IID-LTT and 0.0300 for both the long-run-variance and circular-block-bootstrap alternatives, with empirical family-wise error 0.0902.
Subtracting each loss's one-step predictable risk makes the gap between empirical and target risk a martingale average, which supports a finite-sample test using only a deterministic loss bound and a variance-adaptive test using predictable quadratic variation. Conditional centering yields a martingale-difference sequence, so the tests need no specification of dependence among losses; the finite-sample version uses only the loss bound, while the adaptive version normalizes by conditional variance accumulated along the stream. Evaluation covers 49 dataset-risk combinations with 2,000 repetitions per configuration at δ of 0.10 and 0.20; the adaptive test requires a predictable-variance stabilization assumption, and for binary losses the variation is estimated by a plug-in estimator from online one-step risk estimates.
Combining policy-level p-values with family-wise error control yields a simultaneously certified policy region from which a deployment policy can be selected without repeating certification. Policy-level tests handle dependence along the deployment stream while family-wise error control handles multiplicity across candidate policies, so the whole certified region lies inside the safe region with probability at least 1−δ. At δ=0.10, StreamLTT-A retains 76.7% of the candidate family across all 49 combinations with mean empirical family-wise error 0.0902; raising δ to 0.20 moves empirical family-wise error to 0.1888 and the mean risk gap from 0.0090 to 0.0078.
Perspective
The demonstrated scope is seven binary deployment risks on seven real forecasting datasets spanning energy, traffic and temperature forecasting, with a frozen temporal convolutional network as the predictor, a candidate policy family fixed before certification, and losses with deterministic bounds; the certification stream is processed in original temporal order using only information available before each outcome is revealed.
Validity of the variance-adaptive test relies on the predictable quadratic variation stabilizing at a deterministic scale and is asymptotic; the finite-sample version needs only a loss bound but normalizes by the worst-case loss range, so it is conservative, with empirical family-wise error of only 0.0001 and 0.0006 at δ of 0.10 and 0.20. The attainable certified frontier itself varies by dataset, as seen for the opposite-direction risk on Jena and the severe-miss risk on several streams.