# Efficient Exploration

Continuous-control reinforcement learning struggles with exploration and long-horizon credit assignment, and the usual fix, action noise or entropy regularization, explores without direction. GANGSTER uses a conditional generative adversarial network to propose candidate subgoals from the current state, scores them with structured losses for desirability, reachability and diversity, and feeds the chosen subgoal back only as intrinsic reward shaping, leaving the core TD3 and SAC update rules untouched. On MuJoCo's Ant-v5, GANGSTER with TD3 cuts time-to-solve by about 31% and raises final performance by 53% over baseline TD3; on HalfCheetah-v5 with SAC it raises final performance by 10% and lifts post-solution consistency above 99%.

Continuous-control reinforcement learning struggles with exploration and long-horizon credit assignment, and the usual fix, action noise or entropy regularization, explores without direction. GANGSTER uses a conditional generative adversarial network to propose candidate subgoals from the current state, scores them with structured losses for desirability, reachability and diversity, and feeds the chosen subgoal back only as intrinsic reward shaping, leaving the core TD3 and SAC update rules untouched. On MuJoCo's Ant-v5, GANGSTER with TD3 cuts time-to-solve by about 31% and raises final performance by 53% over baseline TD3; on HalfCheetah-v5 with SAC it raises final performance by 10% and lifts post-solution consistency above 99%.

This work studies online iterative Nash learning from human feedback (NLHF) and identifies exploration as a key obstacle: standard iterative NLHF can incur an exponential dependence on the inverse KL-regularization parameter, showing that implicit exploration through policy updates can be insufficient; it proposes ENPO, which combines SFT-type regularization with adversarial policy exploration to eliminate this exponential dependence without minimax oracles or explicit preference-model estimation, further introduces BENPO, which uses additional oracles to achieve an O(log T) regret bound, and develops DENPO, a practical variant of ENPO for fine-tuning LLMs; with Llama-3-8B-Instruct, DENPO shows consistent improvements over the evaluated RLHF and NLHF baselines across multiple benchmarks.

This work studies online iterative Nash learning from human feedback (NLHF) and identifies exploration as a key obstacle: standard iterative NLHF can incur an exponential dependence on the inverse KL-regularization parameter, showing that implicit exploration through policy updates can be insufficient; it proposes ENPO, which combines SFT-type regularization with adversarial policy exploration to eliminate this exponential dependence without minimax oracles or explicit preference-model estimation, further introduces BENPO, which uses additional oracles to achieve an O(log T) regret bound, and develops DENPO, a practical variant of ENPO for fine-tuning LLMs; with Llama-3-8B-Instruct, DENPO shows consistent improvements over the evaluated RLHF and NLHF baselines across multiple benchmarks.