A generative adversarial network hands reinforcement learning its own subgoals: on Ant-v5, TD3 solves about 31% faster and ends 53% higher
Synopsis
Continuous-control reinforcement learning struggles with exploration and long-horizon credit assignment, and the usual fix, action noise or entropy regularization, explores without direction. GANGSTER uses a conditional generative adversarial network to propose candidate subgoals from the current state, scores them with structured losses for desirability, reachability and diversity, and feeds the chosen subgoal back only as intrinsic reward shaping, leaving the core TD3 and SAC update rules untouched. On MuJoCo's Ant-v5, GANGSTER with TD3 cuts time-to-solve by about 31% and raises final performance by 53% over baseline TD3; on HalfCheetah-v5 with SAC it raises final performance by 10% and lifts post-solution consistency above 99%.
Interpretation
GANGSTER turns subgoal discovery into a conditional generation step: given the agent's current state, a generator emits 100 candidate intermediate goals at once, and a scoring function picks one as the exploration anchor for that stretch of training. Earlier hierarchical reinforcement learning either required hand-specified hierarchies, task-specific abstractions or auxiliary controllers, or discovered subgoals through heuristics, fixed clustering or implicit representations, leaving little explicit control over the quality of intermediate goals. On Ant-v5 with TD3, GANGSTER reaches a time-to-solve of 209 episodes against 303.4 for baseline TD3, about 31% shorter, and a final 50-episode average of 1850.62 against 1205.99, about 53% higher.
The generator is not trained on adversarial loss alone; three interpretable terms are added to its objective, pulling subgoals toward high-value states, toward the current state, and apart from one another within a batch. The three terms divide the work: the desirability term supplies the main performance signal, the reachability term penalizes distant or infeasible goals to steady training, and the diversity term keeps the generator from collapsing onto a few repeated targets. The ablation runs on two algorithm-environment pairings, TD3 with Ant-v5 and SAC with HalfCheetah-v5, covering single terms, pairwise combinations and equal weights; in both, desirability-leaning configurations reach the highest final performance, and most SAC-side configurations keep consistency above 97%.
The selected subgoal changes only the reward: each step adds an intrinsic reward equal to the reduction in distance to that goal, scaled by a weight, to the environment reward before the transition enters the replay buffer. The actor-critic update equations of TD3 and SAC stay as they are, so the baselines keep their stability and only the data distribution and reward shaping change. The cross-environment evaluation covers Pendulum-v1, Hopper-v5, HalfCheetah-v5, Walker2d-v5 and Ant-v5, training each algorithm-environment pairing for 500 episodes across 5 random seeds, with the intrinsic reward weight set between 0.1 and 0.3.
Perspective
The evaluation is confined to the MuJoCo continuous-control tasks shipped with Gymnasium, namely Pendulum-v1, Hopper-v5, HalfCheetah-v5, Walker2d-v5 and Ant-v5, with each algorithm-environment pairing trained for 500 episodes across 5 random seeds; the method assumes meaningful subgoals can be represented directly in state space.
Gains are uneven: on low-complexity Pendulum-v1 GANGSTER brings no measurable benefit and with SAC even lengthens time-to-solve; on HalfCheetah-v5 with TD3 the baseline solves faster, a significant difference; on Walker2d-v5 convergence is faster and more stable but final reward sits slightly below baseline. The structured-loss weights and the intrinsic reward scale need tuning, which the authors note may affect reproducibility, and per-step compute time rises 45% to 216%, highest on Hopper-v5.