RWKV fades on long horizons: alternating in attention blocks lets a 36M model match a 1B forecaster on GIFT long-horizon MASE

In non-autoregressive multi-step forecasting, the pure recurrent model RWKV-7 degrades systematically at long horizons, and interleaving attention blocks largely removes that degradation. RWKV-7 matches similarly sized Transformers in language modeling, so it should have been competitive for long-horizon time series forecasting; under non-autoregressive inference it instead collapses in amplitude over the prediction region, while the Transformer shows no comparable collapse under the same data and training configuration. On a synthetic signal RWKV-7 collapses in amplitude over the prediction region while the Transformer does not under the same data and training configuration; on GIFT-Eval, HART lowers long-horizon MASE from 0.770 for pure RWKV to 0.752 and medium-horizon MASE from 0.749 to 0.727, with comparable short-horizon performance.

HART lets prediction positions retrieve historical tokens through attention while keeping recurrent modeling of local temporal structure. The backbone alternates one Transformer block with one RWKV block, so attention performs global retrieval of history and the generalized delta rule accumulates observational information in the historical region. The three models are pre-trained with identical data, optimization strategy, and random seed, and evaluated on 97 GIFT-Eval entries and 98 TIME-Benchmark task-frequency combinations using MASE relative to the seasonal naive forecast, aggregated by geometric mean.

Layer-wise measurements show lower write norms in the prediction region than in the historical region for most layers, while transition-matrix norms stay similar across the two regions. That pattern is consistent with the write-compensation hypothesis of Placeholder Idling Decay: with no new observations in the prediction region, placeholder-driven writes that compensate insufficiently let historical information attenuate across state transitions. Averaged over 4,096 synthetic sequences with a 16,384-step history, a 1,024-step prediction region, and patch length 16, the prediction-to-history write-norm ratio is 0.60 to 0.87 in the first seven layers, 0.84 to 0.97 in Layers 8 to 11, and reverses to 1.34 in Layer 12, while the transition-norm ratio stays within 1.00 to 1.03 across layers.

The report addresses researchers and engineers working on time series forecasting and sequence modeling, covering the RWKV-7 mechanism analysis under non-autoregressive multi-step forecasting and the HART architecture, with evaluation limited to GIFT-Eval, TIME-Benchmark, and six synthetic signals. The mechanistic analysis covers RWKV-7 only; applying the transition-injection decomposition to other recurrent architectures requires separate validation.

The norm diagnostics characterize matrix magnitudes and do not directly measure how much historical information is retained or the contraction rate of transition products. The forecasting comparisons do not isolate Placeholder Idling Decay from all other architectural effects, because architecture, training data, and inference design are not separated. HART does not dominate every signal regime: on SE4, RWKV reaches a long-horizon MASE of 1.058 against 1.148 for HART and 1.151 for the Transformer. The teacher-forced control changes the available information, so it neither isolates Placeholder Idling Decay nor represents deployable autoregressive forecasting.