Zhiheng Zhang's fluctuation-supervised pretraining (FSP) cuts macro RMSE by 7.0% versus S-learner across 24 nonlinear continuous-covariate cells and by 54.2% versus latent-effect supervision under effect shift
Synopsis
The work introduces fluctuation-supervised pretraining (FSP), labeling each synthetic table by its average treatment effect plus its efficient influence-function fluctuation while deployment remains a frozen forward pass; the author proves an endpoint transition along the path T_{λ,P}=θ(P)+λP_nψ_P, where every fixed λ<1 retains label ambiguity of order (1-λ)^2/n whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^{-2}; experiments show that across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells, validation-selected Summary FSP deploys 11.
Figure 1 : Teach the sampling response once, reuse it across new tables. Panel a shows why latent-effect supervision can have the wrong repeated-sampling center under prior shift; Panel b shows that FSP moves EIF structure from per-dataset correction to reusable synthetic supervision; Panel c shows why the full-fluctuation endpoint is statistically special and how finite pretraining error propagates into inferential guarantees.
arXivInterpretation
The paper introduces fluctuation-supervised pretraining (FSP), in which each synthetic table is labeled by its average treatment effect plus its efficient influence-function fluctuation, with deployment remaining a frozen forward pass. Relative to causal tabular foundation models supervised by latent effects, which reward posterior shrinkage, FSP instead encodes the repeated-sample response needed in a fixed deployment population. This is the method setting stated in the abstract, accompanied by a theoretical characterization along the path T_{λ,P}=θ(P)+λP_nψ_P.
The author proves an endpoint transition: every fixed λ<1 retains label ambiguity of order (1-λ)^2/n, whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^{-2}. The result links the choice of label construction to an attainable risk order rather than resting on empirical comparison alone. This is a theoretical proof reported in the paper; the abstract also gives complementary lower bounds separating local n^{-1} ATE risk from the log N/M excess risk of generic finite-dictionary episode learning.
The paper provides a finite-pretraining bound combining label, network, episode-sampling, and optimization errors, whose sampling defect controls fixed-mechanism bias, mean squared error, variance, Gaussian approximation, and, with variance-head accuracy, studentized coverage. This connects the pretraining error decomposition to downstream causal-inference properties, including coverage. The bound and lower bounds are stated in the abstract; known-effect semisynthesis tests coverage.
Experiments trace the learned sampling response: across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells; validation-selected Summary FSP deploys 11.6× faster per table in a warm one-thread benchmark; under effect shift, matched Raw FSP lowers mean-checkpoint RMSE by 54.2% and teacher defect by 99.0% versus latent-effect supervision and RMSE by 10.2% versus the released CausalPFN-S checkpoint. These numbers map the theoretical endpoint transition onto specific comparators (S-learner, latent-effect supervision, the CausalPFN-S checkpoint) and specific settings (weak overlap, effect shift, deployment speed). Specific percentages and cell counts reported in the abstract; two randomized-study evaluations show that lower RMSE can coexist with residual attenuation.
Perspective
The work targets the pretraining stage of causal tabular foundation models: synthetic tables are labeled by average treatment effect plus efficient influence-function fluctuation, and deployment remains a frozen forward pass, so its conclusions apply to effect-estimation pipelines aimed at a fixed deployment population. The theory derives the endpoint transition and finite-pretraining bound along the path T_{λ,P}=θ(P)+λP_nψ_P, while the experiments cover 24 nonlinear continuous-covariate cells, 12 weak-overlap cells, an effect-shift setting, a warm one-thread deployment benchmark, known-effect semisynthesis coverage tests, and two randomized-study evaluations. For a reader, this offers a reusable perspective on label construction and evaluation: it ties pretraining supervision to downstream bias, mean squared error, variance, Gaussian approximation, and coverage, and it provides comparable baselines (S-learner, latent-effect supervision, the CausalPFN-S checkpoint).
The abstract notes that two randomized-study evaluations show lower RMSE can coexist with residual attenuation, which suggests readers should be careful when reading RMSE improvements directly as better effect estimation. The abstract does not give the specific data sources, sample sizes, or confidence intervals for each experiment, nor how variance-head accuracy is determined, so the scope of the coverage conclusion still depends on details in the full text. In addition, the abstract does not state how FSP performs on non-tabular data, non-continuous covariates, or non-synthetic mechanisms, which remain open to observation.