Safety controllers stop chasing average reward: training on the weakest certificate margin yields zero test violations in three domains
Synopsis
Learned controllers for safety-critical systems usually maximize average reward first and add safety through penalties, constraints or runtime filters. CeDL instead trains without reward: it combines Lyapunov stability, control-barrier safety, temporal-logic robustness and feasibility margins into one certificate margin, and an adversarial falsifier repeatedly surfaces the scenarios closest to violation. Across simulated URLLC wireless resource allocation, energy-safe sensor-network routing and robotic obstacle avoidance, CeDL reports zero observed test-set violations and larger minimum certificate margins than reward-based RL, constrained RL, model predictive control, shielded RL and classical controllers.
Interpretation
The learning objective shifts from maximizing expected return to maximizing the worst-case certificate margin, so the controller is trained directly to strengthen its weakest auditable behavior. Earlier safe-learning methods mostly attach safety to reward maximization as a penalty, a constraint, a post-training verification step or an external correction layer, and certificates often target a single property such as stability or pointwise safety. In the URLLC ablation, full CeDL reaches a 0.00% deadline violation rate and a 0.187 minimum certificate margin, while average-margin training gives 0.29% and 0.031.
Policy and certificate are trained together, with the certificate inside the loss rather than a post-training diagnostic. Certificate-based control usually assumes the certificate is known in advance or enforces a given safety condition around a nominal controller. The certificate takes the minimum over Lyapunov, barrier, temporal-robustness and feasibility terms, approximated by a smooth minimum for backpropagation; in the wireless case it is parameterized by a quadratic prior over queues, deadline debt and violation accumulation plus a neural residual.
An adversarial falsifier searches the declared operating domain for near-violation scenarios and stores low-margin cases in a hard-scenario bank as counterexample-guided training signals. Conventional practice relies on random sampling or nominal validation, with counterexamples used for checking rather than training. Removing adversarial falsification raises the URLLC deadline violation rate from 0.00% to 0.18%, the 99.9% delay from 3.12 ms to 4.96 ms, and lowers the minimum certificate margin from 0.187 to 0.064.
A runtime safety shield serves only as a final feasibility-preserving layer, pulling the action to the nearest feasible one when the nominal policy nears the certified boundary. In conventional shielded RL the policy is trained mainly for reward and the shield corrects unsafe actions after the decision. Shield activation is 6.4% for CeDL versus 18.7% for shielded RL in the URLLC case, and 7.1% versus 21.4% in the robotic case.
Perspective
The claims hold over a declared bounded operating domain: state, control, disturbance and parameter sets are assumed compact and the relevant maps locally Lipschitz over the operating region, and CeDL does not assert global safety over arbitrary unmodeled environments. Evaluation covers three simulated domains, URLLC wireless resource allocation, energy-safe wireless sensor-network routing and robotic obstacle avoidance, with test sets of 10,000, 8,000 and 6,000 scenarios, of which 2,000, 1,500 and 1,500 are adversarial.
Zero violations is an observed test-set result; a domain-level certificate condition requires positive certificate margins across all admissible scenarios. The authors also state that guarantees depend on the declared operating domain, the expressiveness of the certificate class, the quality of adversarial scenario generation and the feasibility of the runtime shield, and list tighter generalization bounds, scalable distributed implementations and hardware-in-the-loop validation as future directions.