Flipped preference labels no longer get blanket down-weighting: RW-DPO gives each comparison a reliability weight, and its win rate holds as flips rise
Synopsis
Preference data contain reversed, tied and otherwise noisy comparisons, yet Direct Preference Optimization (DPO) fits every reported comparison as correct, and its quality falls as the share of flipped labels grows. RW-DPO attaches an unknown reliability weight to each comparison, lets it enter the likelihood linearly, and estimates weights jointly with the policy under a minimax-concave penalty, so comparisons the model agrees with stay at weight one and only contradicted ones are adjusted. In DPO fine-tuning of Qwen2.5-1.5B with 10% to 30% randomly flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method.
Interpretation
Preference optimization no longer treats every human comparison as a correct label; each comparison carries a reliability weight that the data can revise. DPO treats every reported preference as correct, and its quality falls steadily with the share of flipped labels; methods built on a global flip rate treat every comparison alike and cannot tell which ones are wrong. In DPO fine-tuning of Qwen2.5-1.5B-ultrachat200k with LoRA on UltraFeedback comparisons with a share of labels flipped at random, DPO's win rate against clean-label DPO falls from 50.0 at 10% flips to 34.5 at 30%.
The weight multiplies the model-annotator agreement linearly, so one comparison moves continuously from fully trusted through uninformative to reversed. A weight of one recovers the DPO term, zero makes the comparison uninformative, and a negative weight corrects its direction; a genuine tie is one point on that continuum. Weights are estimated on a standardized margin, scaled by a robust standard deviation (MAD) and not centred, so the decision threshold does not drift as the policy's margins sharpen; the weight step has a closed-form solution after one standard approximation, at negligible cost over DPO.
The penalty acts only where the model contradicts the annotation, and comparisons it agrees with keep weight exactly one and train as in DPO. Earlier per-comparison corrections each lack something: some discard a comparison but cannot reverse it, some blend every comparison with its reversal by a fixed rule and never fully trust one, and some penalties bias the estimates. In linear-index simulations, relative error is 0.381 under random flips and 0.526 under feature-dependent flips, below DPO's 0.632 and 0.666; when only clear-cut comparisons are flipped, it flags 95% of flips while flagging 0.8% of clean comparisons, at precision 0.91, against 0.35 for PLC-DPO and 0.19 for the mean-shift fit.
On the language model the correction concentrates on flipped comparisons rather than spreading weight evenly. PLC-DPO blends every comparison with its reversal by a fixed rule, so the share of its weight falling on flipped pairs equals the flip rate; RW-DPO places weight only on comparisons it suspects are flipped, so flipped pairs receive a share above the flip rate itself. Language-model margins identify flips less sharply than simulated ones, with flag precision at most one half, so RW-DPO down-weights suspected flips rather than reversing them; after the second epoch sharpens the policy's margins, its flag precision rises.
Perspective
The demonstrated scope is simulation of the linear-index model and two-epoch LoRA fine-tuning of Qwen2.5-1.5B-ultrachat200k on UltraFeedback with 10% to 30% randomly flipped labels; win rates are judged by Skywork-Reward-V2-Llama-3.1-8B against clean-label DPO at the same training budget. The guarantees target the linear-index model with binary true weights, while the language-model runs use a stochastic mini-batch variant.
The guarantees rest on a linear-index model with binary true weights, and ties and graded reliability are left for future work; direction and scale are identified by different comparisons, only close calls carry scale information, and flips among close calls cannot be told apart from ordinary noise and bias every scale estimator, this one included. The stochastic mini-batch scheme used on the language model is not covered by the consistency theory, and its flag precision is at most one half.