Preference optimization gains a linguistic constraint: folding information uniformity into the reward brings model wording closer to human habits

Preference optimization gains a second optimization direction drawn from linguistics: instead of only deciding which response is preferred, training also asks that information inside a response be distributed the way humans distribute it. SimPO is reference-free and learns only from comparisons between response pairs, so it has no constraint on the optimization direction: it knows some responses are preferred but not which linguistic property to attend to. On AlpacaEval 2, Arena-Hard v0.1 and MT-Bench, UIDPO and the reproduced SimPO each win half of the benchmarks, with differences inside the random error of training and evaluation.

Information uniformity is split into a global and a local term, which constrain the long-range range of information density and the abrupt changes between adjacent words respectively. The earlier uniform information density formulation measured global and local uniformity with variance and expected density to be uniform throughout a text, but real discourse jumps at topic boundaries and full-length uniformity is rare. On Llama-3-8B-Instruct and Mistral-7B-Instruct, keeping only the global term scores below the untuned Instruct model on both, while mixing the two terms raises the length-controlled win rate on AlpacaEval 2* for Llama-3-8B-Instruct from 45.8% to 63.2%.

The local term becomes the mean of second-order absolute differences of log probabilities, which is less sensitive to outliers and prefers piecewise linear rather than globally smooth information curves. The earlier local uniformity used the variance of first-order differences, which does not distinguish between two information curves and is sensitive to outliers. On AlpacaEval 2*, a mixing coefficient of 0.25 gives the best length-controlled win rate for both Llama-3-8B-Instruct and Mistral-7B-Instruct, indicating the value is consistent across model backbones.

On English complement clauses, the finetuned model supplies a necessary that more often, while the reduction in redundant that is smaller. SFT and SimPO count 495 and 617 correct necessary that instances against 676 for UIDPO, and 843 and 960 total correct instances against 1021. Across 4817 complement-clause instances extracted from the Switchboard corpus, pairwise Cochran's Q and McNemar tests among SFT, SimPO and UIDPO all give p-values below 0.05; the authors also note that in overall propensity to produce that, UIDPO sits between SFT and SimPO.

The report covers UIDPO preference optimization experiments on two instruct models, Llama-3-8B-Instruct and Mistral-7B-Instruct, trained on llama3-ultrafeedback-armorm and evaluated on AlpacaEval 2, Arena-Hard v0.1 and MT-Bench, with the linguistic analysis drawn from complement clauses in the Switchboard corpus. It is aimed at researchers working on preference optimization objectives and linguistic constraints.

The authors report that the gap between UIDPO and SimPO is within random error, so the result is comparability rather than a stable improvement; Mistral-7B-Instruct is inconsistent with preference data generated by Llama-3-8B-Instruct, and the authors present its results for reference; the authors state that the conditions under which the original uniform information density assumption still works are currently unclear.