# Reverse-KL Regularization

This paper proposes and analyzes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles that, instead of directly optimizing a differentiable preference loss, perturbs the current policy and judges whether each perturbation raises the likelihood of preferred responses and lowers that of dispreferred ones to extract an update direction, thereby exploiting low-margin "noisy" preference pairs; the authors establish a convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and oracle compatibility, introduce an online version that uses unlabeled policy generations for reverse-KL control, and prove a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward

This paper proposes and analyzes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles that, instead of directly optimizing a differentiable preference loss, perturbs the current policy and judges whether each perturbation raises the likelihood of preferred responses and lowers that of dispreferred ones to extract an update direction, thereby exploiting low-margin "noisy" preference pairs; the authors establish a convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and oracle compatibility, introduce an online version that uses unlabeled policy generations for reverse-KL control, and prove a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward