# Double Q-learning

A course of repetitive transcranial magnetic stimulation (rTMS) for treatment-resistant depression runs about 36 sessions, each requiring a choice of stimulation type, intensity and laterality, and clinicians have lacked evidence-based guidance for those session-level decisions. A reinforcement-learning policy trained on electronic health records from 1609 adults across 19 clinic groups in one behavioral health network was tested on 370 patients at six held-out clinics: sessions where clinicians matched its recommendation showed 0.788 points greater next-session PHQ-9 improvement (95% CI 0.634–0.940). Following every recommendation was projected to add 3.22 points over a first episode, but this is a retrospective association without prospective validation.

A course of repetitive transcranial magnetic stimulation (rTMS) for treatment-resistant depression runs about 36 sessions, each requiring a choice of stimulation type, intensity and laterality, and clinicians have lacked evidence-based guidance for those session-level decisions. A reinforcement-learning policy trained on electronic health records from 1609 adults across 19 clinic groups in one behavioral health network was tested on 370 patients at six held-out clinics: sessions where clinicians matched its recommendation showed 0.788 points greater next-session PHQ-9 improvement (95% CI 0.634–0.940). Following every recommendation was projected to add 3.22 points over a first episode, but this is a retrospective association without prospective validation.

Recursive QAOA re-measures at every variable elimination, so shot cost compounds along the recursion even though steps differ in difficulty. The work casts step-wise shot allocation in depth-1 recursive QAOA as a sequential decision problem: a hand-crafted rule keyed to local difficulty sets a baseline, and a tabular Double Q-learning agent learns residual corrections under a Lagrangian constraint while the elimination rule stays fixed. Under a fixed per-step cap, on 130 held-out policy-instance pairs the heuristic cuts total shots by about 23% and the learned policy by 36.1% versus uniform allocation, with an effective shots-per-success ratio of 0.676, and the gain holds at about 36.6% on problem sizes unseen in training.

Recursive QAOA re-measures at every variable elimination, so shot cost compounds along the recursion even though steps differ in difficulty. The work casts step-wise shot allocation in depth-1 recursive QAOA as a sequential decision problem: a hand-crafted rule keyed to local difficulty sets a baseline, and a tabular Double Q-learning agent learns residual corrections under a Lagrangian constraint while the elimination rule stays fixed. Under a fixed per-step cap, on 130 held-out policy-instance pairs the heuristic cuts total shots by about 23% and the learned policy by 36.1% versus uniform allocation, with an effective shots-per-success ratio of 0.676, and the gain holds at about 36.6% on problem sizes unseen in training.

The work proposes UniIntervene++, which formulates the evolving RL policy, trajectory correction, and a task-structured CodePolicy as Options in a semi-Markov decision process and learns their relative values online, combines periodic unassisted probing with coupled experience learning, and achieves 89.67% average success across five real-world manipulation tasks, at least 6 percentage points above the best baseline, while reducing the human intervention rate to 0.77%.

The work proposes UniIntervene++, which formulates the evolving RL policy, trajectory correction, and a task-structured CodePolicy as Options in a semi-Markov decision process and learns their relative values online, combines periodic unassisted probing with coupled experience learning, and achieves 89.67% average success across five real-world manipulation tasks, at least 6 percentage points above the best baseline, while reducing the human intervention rate to 0.77%.