# RW-DPO

Preference data contain reversed, tied and otherwise noisy comparisons, yet Direct Preference Optimization (DPO) fits every reported comparison as correct, and its quality falls as the share of flipped labels grows. RW-DPO attaches an unknown reliability weight to each comparison, lets it enter the likelihood linearly, and estimates weights jointly with the policy under a minimax-concave penalty, so comparisons the model agrees with stay at weight one and only contradicted ones are adjusted. In DPO fine-tuning of Qwen2.5-1.5B with 10% to 30% randomly flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method.

Preference data contain reversed, tied and otherwise noisy comparisons, yet Direct Preference Optimization (DPO) fits every reported comparison as correct, and its quality falls as the share of flipped labels grows. RW-DPO attaches an unknown reliability weight to each comparison, lets it enter the likelihood linearly, and estimates weights jointly with the policy under a minimax-concave penalty, so comparisons the model agrees with stay at weight one and only contradicted ones are adjusted. In DPO fine-tuning of Qwen2.5-1.5B with 10% to 30% randomly flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method.