# Proximal Policy Optimization

Reinforcement-learning portfolio policies can propose allocations but lack a calibrated, replayable way to certify before execution that a proposal stays inside a bounded risk envelope. Gated-CSRC has a language model read timestamped news only into structured semantic risk states, converts those states into factor-shock ellipsoids through rolling conformal calibration, and lets a second-order cone gate correct an allocation only when its budget is breached. In walk-forward tests on nine U.S. sector SPDR ETFs, raw PPO earned 12.7% annual return with a 4.05% semantic violation rate; gating cut violations to zero while retaining 11.7% annual return.

Reinforcement-learning portfolio policies can propose allocations but lack a calibrated, replayable way to certify before execution that a proposal stays inside a bounded risk envelope. Gated-CSRC has a language model read timestamped news only into structured semantic risk states, converts those states into factor-shock ellipsoids through rolling conformal calibration, and lets a second-order cone gate correct an allocation only when its budget is breached. In walk-forward tests on nine U.S. sector SPDR ETFs, raw PPO earned 12.7% annual return with a 4.05% semantic violation rate; gating cut violations to zero while retaining 11.7% annual return.

Policy optimization in language-model post-training starts from a capable base model, yet REINFORCE, the basis of PPO and GRPO, can drive the probability of optimal responses toward zero even as reward improves, and convergence can take exponentially many steps even with exact gradients and KL regularization to the base model. Periodically resetting the KL reference to the most recent policy, formalized as Approximate Policy Mirror Descent (APMD), lets each subproblem be solved locally under strong regularization while successive resets accumulate into global optimality. For softmax linear policies, this yields the first global convergence and coverage-preserving guarantees for PPO and GRPO variants with reference resets and for Kimi-style squared-loss algorithms.

Policy optimization in language-model post-training starts from a capable base model, yet REINFORCE, the basis of PPO and GRPO, can drive the probability of optimal responses toward zero even as reward improves, and convergence can take exponentially many steps even with exact gradients and KL regularization to the base model. Periodically resetting the KL reference to the most recent policy, formalized as Approximate Policy Mirror Descent (APMD), lets each subproblem be solved locally under strong regularization while successive resets accumulate into global optimality. For softmax linear policies, this yields the first global convergence and coverage-preserving guarantees for PPO and GRPO variants with reference resets and for Kimi-style squared-loss algorithms.

Deciding whether to look at the lane or the speedometer becomes a problem of allocating limited visual resources under partial observability. Earlier models treated lane keeping and speed control separately, and none formalized sampling decisions alongside coupled lateral and longitudinal control. A hierarchical computational rationality model learns lane keeping, speed maintenance and supervisory attention allocation together; across 30 simulated runs, all eight measure means fell within one human standard deviation, though the model produced no lane deviations.

Deciding whether to look at the lane or the speedometer becomes a problem of allocating limited visual resources under partial observability. Earlier models treated lane keeping and speed control separately, and none formalized sampling decisions alongside coupled lateral and longitudinal control. A hierarchical computational rationality model learns lane keeping, speed maintenance and supervisory attention allocation together; across 30 simulated runs, all eight measure means fell within one human standard deviation, though the model produced no lane deviations.

Regularized interior-point solvers for convex quadratic programs have long relied on hand-set regularization weights, barrier parameters and inner tolerances. RL-QProxIP uses a reinforcement-learning policy to output contraction factors for those parameters at each outer iteration without changing the underlying algorithm, so the solver keeps its convergence guarantees. On the Maros–Mészáros test set the policy is the fastest solver on about 70% of instances, while the base QProxIP needs about 3.5 times the time to reach the same fraction; the source states two conflicting versions of their final coverage, so no preference is taken here.

Regularized interior-point solvers for convex quadratic programs have long relied on hand-set regularization weights, barrier parameters and inner tolerances. RL-QProxIP uses a reinforcement-learning policy to output contraction factors for those parameters at each outer iteration without changing the underlying algorithm, so the solver keeps its convergence guarantees. On the Maros–Mészáros test set the policy is the fastest solver on about 70% of instances, while the base QProxIP needs about 3.5 times the time to reach the same fraction; the source states two conflicting versions of their final coverage, so no preference is taken here.

Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.

Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.

Lane directions and speed limits across an urban network are now set jointly at every decision step by one transformer shared by all road controllers, passed through a check grounded in graph orientation theory that keeps every junction mutually reachable at all times. On a grid with commuter rush hours, the learned policy makes about two thirds of roads one-way in the peak direction and cuts daily delay by roughly 80% against holding the initial two-way configuration all day, and by roughly 70% against a tuned two-way load balancer. Restricted to two-way roads at test time, it gains only about as much as that balancer, and its learned speed limits add delay.

Lane directions and speed limits across an urban network are now set jointly at every decision step by one transformer shared by all road controllers, passed through a check grounded in graph orientation theory that keeps every junction mutually reachable at all times. On a grid with commuter rush hours, the learned policy makes about two thirds of roads one-way in the peak direction and cuts daily delay by roughly 80% against holding the initial two-way configuration all day, and by roughly 70% against a tuned two-way load balancer. Restricted to two-way roads at test time, it gains only about as much as that balancer, and its learned speed limits add delay.

The humanoid Reflex catches human-thrown boxes using only onboard RGB-D vision, reaching 85.7% catch success in simulation versus 87.9% with privileged box observations, and 65% and 40% on the real robot at 2.5 and 5 m.

The humanoid Reflex catches human-thrown boxes using only onboard RGB-D vision, reaching 85.7% catch success in simulation versus 87.9% with privileged box observations, and 65% and 40% on the real robot at 2.5 and 5 m.

The authors represent the Unitree Go1 as a cell complex whose four limb-level and one body-level rank-2 cells make multi-joint mechanical units explicit, and apply Hodge-based message passing over nodes, edges, and faces; under degradation training, the node-edge-face Hodge actor achieves the highest return on unseen actuator degradations, with higher survival and lower velocity-tracking error.

The authors represent the Unitree Go1 as a cell complex whose four limb-level and one body-level rank-2 cells make multi-joint mechanical units explicit, and apply Hodge-based message passing over nodes, edges, and faces; under degradation training, the node-edge-face Hodge actor achieves the highest return on unseen actuator degradations, with higher survival and lower velocity-tracking error.

The work proposes an autonomous robotic ultrasound framework driven by an LLM agent that retrieves guideline steps from scanning handbooks, reasons over current observations and scanning state, and dynamically invokes tools for trajectory planning, robot execution, contact adjustment, voice guidance, and trajectory refinement, with reinforcement-learning (PPO) fine-tuning to improve reasoning quality and the correctness of tool selection and parameterization; validated verbally on 10 unseen ultrasound scanning guidelines, fine-tuning raised step-wise accuracy from 0.6512 to 0.8973 and overall success rate from 0.5384 to 0.

The work proposes an autonomous robotic ultrasound framework driven by an LLM agent that retrieves guideline steps from scanning handbooks, reasons over current observations and scanning state, and dynamically invokes tools for trajectory planning, robot execution, contact adjustment, voice guidance, and trajectory refinement, with reinforcement-learning (PPO) fine-tuning to improve reasoning quality and the correctness of tool selection and parameterization; validated verbally on 10 unseen ultrasound scanning guidelines, fine-tuning raised step-wise accuracy from 0.6512 to 0.8973 and overall success rate from 0.5384 to 0.