# Proximal Policy Optimization

11 related articles

Subscribe to updates

arXiv Oct 29, 2026
Differentiable decision trees stop spreading across dozens of features: a Euclidean penalty cuts Wine's effective node features from 8.94 to 1.54
Differentiable decision trees keep explicit paths and end-to-end training, but an oblique soft split at each internal node can depend on many input features at once, weakening node-level readability. EU-DDT adds a normalised squared Euclidean regulariser to the training objective that constrains how far each node's relative feature weights depart from uniform, without changing the sigmoid soft routing. Over five matched random seeds, mean effective features fall from 8.94 to 1.54 on Wine and from 45.28 to 9.06 on Digits, while mean test accuracy moves from 100.0% to 93.9% and from 92.3% to 94.6%, a dataset-dependent trade-off between concentration and performance.
Differentiable decision trees keep explicit paths and end-to-end training, but an oblique soft split at each internal node can depend on many input features at once, weakening node-level readability. EU-DDT adds a normalised squared Euclidean regulariser to the training objective that constrains how far each node's relative feature weights depart from uniform, without changing the sigmoid soft routing. Over five matched random seeds, mean effective features fall from 8.94 to 1.54 on Wine and from 45.28 to 9.06 on Digits, while mean test accuracy moves from 100.0% to 93.9% and from 92.3% to 94.6%, a dataset-dependent trade-off between concentration and performance.
University of Birmingham
Qiqi Yang（First author, Corresponding author）
EU-DDT Euclidean feature-concentration regularisation Differentiable decision trees Intrinsic interpretability Proximal Policy Optimization
arXiv Oct 12, 2026
PPO's clip was only one geometry: swapping in KL divergence with atomic estimation lifts math average from 27.7 to 41.5
PPO's clipping in LLM reinforcement learning is generalized into a two-dimensional design space of divergence geometry and estimator topology, where KL divergence with atomic estimation raises average scores by about 50% on math and 14% on coding over standard GRPO across eight competition-level benchmarks.
Tsinghua University ByteDance (China)
Zhicheng Cai（First author）
Generalized Proximal Policy Optimization Divergence estimator topologies Proximal Policy Optimization Reinforcement Learning Mathematical Reasoning
researchsquare.com Oct 14, 2026
Signal controllers learn whether a jam is a burst or a cut road: waiting-time recovery falls from 329 to 245 seconds under random blockages
Network signal control no longer compresses the whole road network into one shared latent vector. Policies trained under nominal flow often cannot tell a brief fluctuation from a sustained capacity loss when blockages, corridor failures or rain arrive. Here each intersection re-weights its neighbours from their joint state, then reads a history window to judge whether a change persists, cutting waiting-time recovery from 329 to 245 seconds under random blockages and by 82 seconds under a central closure.
Network signal control no longer compresses the whole road network into one shared latent vector. Policies trained under nominal flow often cannot tell a brief fluctuation from a sustained capacity loss when blockages, corridor failures or rain arrive. Here each intersection re-weights its neighbours from their joint state, then reads a history window to judge whether a change persists, cutting waiting-time recovery from 329 to 245 seconds under random blockages and by 82 seconds under a central closure.
Wissem YOUSFI（First author）Rim Slama（Corresponding author）
Traffic Signal Control Multi-Agent Reinforcement Learning Graph Attention Network Proximal Policy Optimization Robustness and Resilience under Traffic Disruptions
arXiv Oct 7, 2026
Agents stop reading each other's rewards: watching peers' behavior is enough to cooperate and split returns more evenly
Multi-agent reinforcement learning no longer needs peers' private rewards: each agent evaluates others' transitions with its own learned reward model, learns cooperation in Escape Room, Clean Up, and Commons Harvest, and often divides returns more evenly than agents that see the true rewards.
Nvidia (United States)
Mohamed Ayman Mohamed（First author）
Self-Referenced Social Preferences Sequential Social Dilemmas Multi-Agent Reinforcement Learning Proximal Policy Optimization
arXiv Oct 8, 2026
DRIVE turns successful-trajectory diversity into an RL objective, improving VLA out-of-domain generalization in simulation and on real robots
The authors analyze how reinforcement learning fine-tuning (RFT) reshapes exploration in vision-language-action (VLA) policies, finding that RFT contracts the overall behavior distribution while making successful trajectories more diverse, easier to elicit, and broader in task-valid solution coverage; they then propose DRIVE, which converts temporally aligned similarity among grouped rollouts under matched task conditions into a success-conditioned intrinsic reward, improving out-of-domain success on LIBERO-Plus, ManiSkill3, and RoboTwin 2.0 and raising average out-of-domain success from 64.1% to 73.3% on a dual-arm AgileX PiPER-X platform.
The authors analyze how reinforcement learning fine-tuning (RFT) reshapes exploration in vision-language-action (VLA) policies, finding that RFT contracts the overall behavior distribution while making successful trajectories more diverse, easier to elicit, and broader in task-valid solution coverage; they then propose DRIVE, which converts temporally aligned similarity among grouped rollouts under matched task conditions into a success-conditioned intrinsic reward, improving out-of-domain success on LIBERO-Plus, ManiSkill3, and RoboTwin 2.0 and raising average out-of-domain success from 64.1% to 73.3% on a dual-arm AgileX PiPER-X platform.
Australian National University Nanjing University Harbin Institute of Technology University of Technology Sydney
Haoru Li（First author）
Diversity-Driven RL Fine-Tuning Vision-Language-Action model Global Alignment Kernel Potential-Based Reward Shaping Proximal Policy Optimization
arXiv Oct 2, 2026
DexPolicy makes exploration noise an explicit function of training steps, lifting deterministic dexterous-manipulation success from 49.4% to 68.1% (FPO) in simulation and on hardware
The work introduces DexPolicy, which in trajectory-guided dexterous manipulation reinforcement learning such as ViViDex makes the action-noise scale an explicit annealing function of training steps while holding loss, architecture, reward, and optimizer settings fixed; across PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO), mean deterministic Target success over five YCB objects and three training seeds rises from 32.0% to 35.7%, 14.1% to 45.4%, and 49.4% to 68.1%, and on a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials raise mean Target success from 8.3% to 43.3%, 10.0% to 63.3%, and 25.0% to 85.0%.
The work introduces DexPolicy, which in trajectory-guided dexterous manipulation reinforcement learning such as ViViDex makes the action-noise scale an explicit annealing function of training steps while holding loss, architecture, reward, and optimizer settings fixed; across PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO), mean deterministic Target success over five YCB objects and three training seeds rises from 32.0% to 35.7%, 14.1% to 45.4%, and 49.4% to 68.1%, and on a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials raise mean Target success from 8.3% to 43.3%, 10.0% to 63.3%, and 25.0% to 85.0%.
DexPolicy Scheduled Exploration dexterous robot hand manipulation Proximal Policy Optimization GRPO
arXiv Oct 1, 2026
TERRA reconstructs terrain from kinematics alone, letting a muscle-actuated body complete stairs, ramps and seats
TERRA presents an end-to-end pipeline that, from scene-less kinematic trajectories alone, combines terrain priors, estimated contacts and negative free-space evidence to recover task-relevant support geometry, adds anatomical, tendon-continuity and contact constraints during retargeting, and uses motion-terrain pairs from five datasets to train a single muscle-actuated control policy, improving terrain accuracy, sharply reducing anatomical and interaction violations, and achieving the highest completion rate over supported terrain families across reconstruction, retargeting and held-out tracking benchmarks.
TERRA presents an end-to-end pipeline that, from scene-less kinematic trajectories alone, combines terrain priors, estimated contacts and negative free-space evidence to recover task-relevant support geometry, adds anatomical, tendon-continuity and contact constraints during retargeting, and uses motion-terrain pairs from five datasets to train a single muscle-actuated control policy, improving terrain accuracy, sharply reducing anatomical and interaction violations, and achieving the highest completion rate over supported terrain families across reconstruction, retargeting and held-out tracking benchmarks.
Merkourios Simos（First author）
TERRA Terrain Reconstruction Motion Retargeting Proximal Policy Optimization
arXiv Sep 30, 2026
StructRL lifts long-horizon VLA success from 41.5% to 49.1% with verifiable subtask rewards
StructRL is an online reinforcement learning framework that automatically decomposes each long-horizon instruction into subtasks verifiable by binary environment-state criteria and organizes them into ordered dependency groups, granting intermediate rewards only after all prerequisites of a subtask are complete and scaling each reward by completion pace; across RoboCasa365 and LIBERO-Long with the GR00T-N1.5 and π0.5 VLA backbones it consistently outperforms the evaluated online RL baselines (e.g., on GR00T-N1.5, RoboCasa365 rises from 41.5% to 49.1% and LIBERO-Long from 92.4% to 96.6%).
StructRL is an online reinforcement learning framework that automatically decomposes each long-horizon instruction into subtasks verifiable by binary environment-state criteria and organizes them into ordered dependency groups, granting intermediate rewards only after all prerequisites of a subtask are complete and scaling each reward by completion pace; across RoboCasa365 and LIBERO-Long with the GR00T-N1.5 and π0.5 VLA backbones it consistently outperforms the evaluated online RL baselines (e.g., on GR00T-N1.5, RoboCasa365 rises from 41.5% to 49.1% and LIBERO-Long from 92.4% to 96.6%).
Pennsylvania State University
Ziyi Yin（First author）
StructRL Vision-Language-Action Models Long-Horizon Robotic Manipulation Proximal Policy Optimization Reward Gating
arXiv Sep 30, 2026
EasyPPO traces two critic failure modes in PPO and stays stable across three tasks, gaining 14.89%, 2.28% and 9.47% over PPO
The work identifies two critic failure modes that destabilize PPO for LLM reinforcement learning — filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, and heterogeneous return noise across prompts lets high-variance prompts dominate critic updates in finite batches — and introduces EasyPPO, which combines actor-only overlong filtering, noise-normalized critic regression weighting each prompt by the inverse standard deviation of its sampled returns, and moderately smaller critic mini-batches with gradient clipping, remaining stable across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24 and multi-turn search on Search-R1, with best validation scores improving over PPO by 14.89%, 2.
The work identifies two critic failure modes that destabilize PPO for LLM reinforcement learning — filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, and heterogeneous return noise across prompts lets high-variance prompts dominate critic updates in finite batches — and introduces EasyPPO, which combines actor-only overlong filtering, noise-normalized critic regression weighting each prompt by the inverse standard deviation of its sampled returns, and moderately smaller critic mini-batches with gradient clipping, remaining stable across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24 and multi-turn search on Search-R1, with best validation scores improving over PPO by 14.89%, 2.
Princeton University University of California, Berkeley
Xuanyi Zhou（First author）
EasyPPO Proximal Policy Optimization Reinforcement Learning with Verifiable Rewards Training Stability
arXiv Sep 29, 2026
A reinforcement learning agent produced all-quadrilateral meshes on all 96 held-out domains, reaching the provable irregularity floor on 90
The work casts quadrilateral block decomposition as a Markov decision process over a half-edge mesh, uses the vertex-irregularity lower bound implied by the discrete Gauss–Bonnet identity (called par) as both reward target and termination test, and trains via behaviour cloning on trivially constructible optimal meshes followed by PPO, producing an all-quadrilateral mesh on all 96 held-out domains, a usable one on 95.7 on average and a provably optimal one on 90, whereas Gmsh's strongest configuration at the same element count completes 51, is usable on 38 and optimal on none.
The work casts quadrilateral block decomposition as a Markov decision process over a half-edge mesh, uses the vertex-irregularity lower bound implied by the discrete Gauss–Bonnet identity (called par) as both reward target and termination test, and trains via behaviour cloning on trivially constructible optimal meshes followed by PPO, producing an all-quadrilateral mesh on all 96 held-out domains, a usable one on 95.7 on average and a provably optimal one on 90, whereas Gmsh's strongest configuration at the same element count completes 51, is usable on 38 and optimal on none.
University of California, Berkeley
Arjun Narayanan（First author）
Quadrilateral Block Decomposition Reinforcement Learning Proximal Policy Optimization geo2d