# Direct Preference Optimization

Safety alignment in open-weight models is highly sparse: pruning a small set of safety-critical neurons or ablating a single refusal direction can make a model produce harmful content. ABS-Align gradually prunes those identified safety-critical neurons during preference finetuning, forcing refusal behavior to be re-established through other parts of the network, and restores the pruned neurons afterward. Across Gemma-2-2B-IT, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct and Qwen-3-4B, attack success generally falls against TwinBreak, NeuroStrike, refusal-direction ablation and ten SORRY-Bench black-box attacks, with reductions of about 90% for some attacks, while TruthfulQA improves and general capabilities change little.

Safety alignment in open-weight models is highly sparse: pruning a small set of safety-critical neurons or ablating a single refusal direction can make a model produce harmful content. ABS-Align gradually prunes those identified safety-critical neurons during preference finetuning, forcing refusal behavior to be re-established through other parts of the network, and restores the pruned neurons afterward. Across Gemma-2-2B-IT, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct and Qwen-3-4B, attack success generally falls against TwinBreak, NeuroStrike, refusal-direction ablation and ten SORRY-Bench black-box attacks, with reductions of about 90% for some attacks, while TruthfulQA improves and general capabilities change little.

Multi-step reaction mechanism prediction has largely stayed at the level of single elementary steps and generalizes poorly to unseen reaction spaces. GRMP couples a chemical mechanism knowledge graph, completed with expert templates, to a language model: a graph convolutional network generates candidate pathways that become preference data for direct preference optimization, and at inference a relation retrieval module pulls graph evidence to refine the pathway. GRMP reaches pathway accuracy of 48.1% on ChemMechQA and 57.9% on oMe-Gold, the highest among the compared methods.

Multi-step reaction mechanism prediction has largely stayed at the level of single elementary steps and generalizes poorly to unseen reaction spaces. GRMP couples a chemical mechanism knowledge graph, completed with expert templates, to a language model: a graph convolutional network generates candidate pathways that become preference data for direct preference optimization, and at inference a relation retrieval module pulls graph evidence to refine the pathway. GRMP reaches pathway accuracy of 48.1% on ChemMechQA and 57.9% on oMe-Gold, the highest among the compared methods.

Preferred Networks released PLaMo 3.0 Fin, an 8B-parameter language model specialized for finance, built from PLaMo 3 NICT 8B Base through continued pretraining on financial corpora, supervised fine-tuning and direct preference optimization, with Japanese reasoning traces and external tool calls. Its average score on the Japanese financial-knowledge benchmark japanese-lm-fin-harness rose from 48.0 for the base model to 55.8, above Gemma 4 E4B and gpt-oss-20b and below Qwen3.5-9B and Ministral 3 14B under PFN's own measurement setup.

Preferred Networks released PLaMo 3.0 Fin, an 8B-parameter language model specialized for finance, built from PLaMo 3 NICT 8B Base through continued pretraining on financial corpora, supervised fine-tuning and direct preference optimization, with Japanese reasoning traces and external tool calls. Its average score on the Japanese financial-knowledge benchmark japanese-lm-fin-harness rose from 48.0 for the base model to 55.8, above Gemma 4 E4B and gpt-oss-20b and below Qwen3.5-9B and Ministral 3 14B under PFN's own measurement setup.

Preferred Networks has released weights for PLaMo 3 Translate, a translation-specific model, at 0.8B, 2B, 8B and 31B on Hugging Face, along with FP8, W4A16, NVFP4, GGUF and MLX formats for quantized and local inference. The smallest 0.8B model outperforms existing translation-specific models more than 30 times larger on Japanese-English and Japanese-to-multilingual translation, while the 2B model surpasses all existing translation-specific models on English-to-multilingual translation. Training uses two-stage supervised fine-tuning plus DPO, with smaller models distilled from the 31B teacher.

Preferred Networks has released weights for PLaMo 3 Translate, a translation-specific model, at 0.8B, 2B, 8B and 31B on Hugging Face, along with FP8, W4A16, NVFP4, GGUF and MLX formats for quantized and local inference. The smallest 0.8B model outperforms existing translation-specific models more than 30 times larger on Japanese-English and Japanese-to-multilingual translation, while the 2B model surpasses all existing translation-specific models on English-to-multilingual translation. Training uses two-stage supervised fine-tuning plus DPO, with smaller models distilled from the 31B teacher.

Preference data contain reversed, tied and otherwise noisy comparisons, yet Direct Preference Optimization (DPO) fits every reported comparison as correct, and its quality falls as the share of flipped labels grows. RW-DPO attaches an unknown reliability weight to each comparison, lets it enter the likelihood linearly, and estimates weights jointly with the policy under a minimax-concave penalty, so comparisons the model agrees with stay at weight one and only contradicted ones are adjusted. In DPO fine-tuning of Qwen2.5-1.5B with 10% to 30% randomly flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method.

Preference data contain reversed, tied and otherwise noisy comparisons, yet Direct Preference Optimization (DPO) fits every reported comparison as correct, and its quality falls as the share of flipped labels grows. RW-DPO attaches an unknown reliability weight to each comparison, lets it enter the likelihood linearly, and estimates weights jointly with the policy under a minimax-concave penalty, so comparisons the model agrees with stay at weight one and only contradicted ones are adjusted. In DPO fine-tuning of Qwen2.5-1.5B with 10% to 30% randomly flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method.

Existing instruction-hierarchy training sets cover constraint-constraint and constraint-goal conflicts but not two goals that conflict with each other. Decomposing a task into goals and constraints yields 9,544 goal-goal preference pairs built from two text-processing instructions that each process the other's message, and fine-tuning Qwen3.5-9B and Gemma4-12B-it on them with direct preference optimization raises mean conflict resistance by 34.3 and 19.4 points on four instruction-hierarchy benchmarks unseen in training, with small mean utility changes across eight benchmarks.

Existing instruction-hierarchy training sets cover constraint-constraint and constraint-goal conflicts but not two goals that conflict with each other. Decomposing a task into goals and constraints yields 9,544 goal-goal preference pairs built from two text-processing instructions that each process the other's message, and fine-tuning Qwen3.5-9B and Gemma4-12B-it on them with direct preference optimization raises mean conflict resistance by 34.3 and 19.4 points on four instruction-hierarchy benchmarks unseen in training, with small mean utility changes across eight benchmarks.

E-commerce ad image optimization moves from regenerating the whole creative to fine-grained attribute editing: 28 click-sensitive visual attributes are selected, editing suggestions are aligned to real click preferences, and a one-week online test over roughly 20 million impressions lifts aggregate CTR by 6.86%.

E-commerce ad image optimization moves from regenerating the whole creative to fine-grained attribute editing: 28 click-sensitive visual attributes are selected, editing suggestions are aligned to real click preferences, and a one-week online test over roughly 20 million impressions lifts aggregate CTR by 6.86%.

This work ports multimodal Direct Preference Optimization (mDPO) from vision-language models to the audio domain, building preference pairs that contrast intact audio with acoustically perturbed audio (reversal, random noise, time and frequency masking) and training Qwen2-Audio-7B-Instruct, raising DCASE 2025 audio QA accuracy from 56.0% to 70.0% and AH Existence accuracy from 55.6% to 83.0%.

This work ports multimodal Direct Preference Optimization (mDPO) from vision-language models to the audio domain, building preference pairs that contrast intact audio with acoustically perturbed audio (reversal, random noise, time and frequency masking) and training Qwen2-Audio-7B-Instruct, raising DCASE 2025 audio QA accuracy from 56.0% to 70.0% and AH Existence accuracy from 55.6% to 83.0%.