# Offline Reinforcement Learning

Offline reinforcement learning usually trains its critic by regressing onto a bootstrapped target, stabilized with a target network. AL-LBLP keeps the Lifted Bellman Linear Programming form, which writes in-sample Bellman optimality as a linear program over the dataset, and solves it with the method of multipliers: the fixed hinge coefficients become multipliers learned by dual ascent, leaving one penalty parameter. With a single critic, no target network and no ensemble, a Gaussian actor, a flow actor and best-of-n sampling average 55.8, 55.7 and 60.1 on the 50 state-based OGBench tasks.

Offline reinforcement learning usually trains its critic by regressing onto a bootstrapped target, stabilized with a target network. AL-LBLP keeps the Lifted Bellman Linear Programming form, which writes in-sample Bellman optimality as a linear program over the dataset, and solves it with the method of multipliers: the fixed hinge coefficients become multipliers learned by dual ascent, leaving one penalty parameter. With a single critic, no target network and no ensemble, a Gaussian actor, a flow actor and best-of-n sampling average 55.8, 55.7 and 60.1 on the 50 state-based OGBench tasks.

In heterogeneous multi-task offline reinforcement learning, tasks differ in state dimension, action space and dynamics, so a directly shared backbone hits both interface mismatch and gradient conflict. HeteroGrad-DT gives each task its own input tokenizer and prediction head around a shared sequence backbone, and a Gradient Flow Transformer pools per-task gradients over a synchronization interval and learns fusion weights to coordinate the shared update. On the DeepMind Control Suite with up to 30 tasks, its average normalized return exceeds representative multi-task methods, with a 32.3% relative gain over standard shared-backbone training and 8.6% over a PCGrad baseline using the same task-aware interfaces.

In heterogeneous multi-task offline reinforcement learning, tasks differ in state dimension, action space and dynamics, so a directly shared backbone hits both interface mismatch and gradient conflict. HeteroGrad-DT gives each task its own input tokenizer and prediction head around a shared sequence backbone, and a Gradient Flow Transformer pools per-task gradients over a synchronization interval and learns fusion weights to coordinate the shared update. On the DeepMind Control Suite with up to 30 tasks, its average normalized return exceeds representative multi-task methods, with a 32.3% relative gain over standard shared-backbone training and 8.6% over a PCGrad baseline using the same task-aware interfaces.

A course of repetitive transcranial magnetic stimulation (rTMS) for treatment-resistant depression runs about 36 sessions, each requiring a choice of stimulation type, intensity and laterality, and clinicians have lacked evidence-based guidance for those session-level decisions. A reinforcement-learning policy trained on electronic health records from 1609 adults across 19 clinic groups in one behavioral health network was tested on 370 patients at six held-out clinics: sessions where clinicians matched its recommendation showed 0.788 points greater next-session PHQ-9 improvement (95% CI 0.634–0.940). Following every recommendation was projected to add 3.22 points over a first episode, but this is a retrospective association without prospective validation.

A course of repetitive transcranial magnetic stimulation (rTMS) for treatment-resistant depression runs about 36 sessions, each requiring a choice of stimulation type, intensity and laterality, and clinicians have lacked evidence-based guidance for those session-level decisions. A reinforcement-learning policy trained on electronic health records from 1609 adults across 19 clinic groups in one behavioral health network was tested on 370 patients at six held-out clinics: sessions where clinicians matched its recommendation showed 0.788 points greater next-session PHQ-9 improvement (95% CI 0.634–0.940). Following every recommendation was projected to add 3.22 points over a first episode, but this is a retrospective association without prospective validation.

The work introduces ICDP, an offline reinforcement learning framework that exactly decomposes joint-support degradation over ego and surrounding-agent futures into an ego-support component and a residual interaction-support component, recovers the latter through contrastive density-ratio estimation, and constrains policy improvement without explicit joint-density modeling, surrounding-agent prediction, or reactive simulator or world-model rollouts; on nuPlan and InterPlan closed-loop evaluations and real-world truck experiments, ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios.

The work introduces ICDP, an offline reinforcement learning framework that exactly decomposes joint-support degradation over ego and surrounding-agent futures into an ego-support component and a residual interaction-support component, recovers the latter through contrastive density-ratio estimation, and constrains policy improvement without explicit joint-density modeling, surrounding-agent prediction, or reactive simulator or world-model rollouts; on nuPlan and InterPlan closed-loop evaluations and real-world truck experiments, ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios.

SWAP casts robot policy routing as an offline reinforcement learning problem, training an IQL routing critic that picks, after each action chunk, the candidate pretrained VLA policy with the highest predicted long-horizon return given the current observation; on real-world DROID tabletop manipulation it raises success from 63.3% for the best single policy to 96.7% while cutting average action steps for successful trajectories from 269 to 193 (a 28.3% reduction), and it outperforms fixed-policy and random-routing baselines in LIBERO and LIBERO-Plus simulation.

SWAP casts robot policy routing as an offline reinforcement learning problem, training an IQL routing critic that picks, after each action chunk, the candidate pretrained VLA policy with the highest predicted long-horizon return given the current observation; on real-world DROID tabletop manipulation it raises success from 63.3% for the best single policy to 96.7% while cutting average action steps for successful trajectories from 269 to 193 (a 28.3% reduction), and it outperforms fixed-policy and random-routing baselines in LIBERO and LIBERO-Plus simulation.

The work augments a Decision Transformer with Test-Time Training (TTT) layers, forming Decision Titan, and analyzes its performance and memory mechanism in the X-Maze environment built to test sequential memory: the model learns long-term dependencies with ranges 20x longer than the context window and generalizes to lengths 1.7x the training data, but temporal generalization depends on the time embeddings used and the ability to learn long-term dependencies depends on how the relevant information is encoded.

The work augments a Decision Transformer with Test-Time Training (TTT) layers, forming Decision Titan, and analyzes its performance and memory mechanism in the X-Maze environment built to test sequential memory: the model learns long-term dependencies with ranges 20x longer than the context window and generalizes to lengths 1.7x the training data, but temporal generalization depends on the time embeddings used and the ability to learn long-term dependencies depends on how the relevant information is encoded.