Literary translation now rewards a large model's scores: reinforcement learning on source sentences alone lifts a 7B model past QwQ-32B-preview on metaphor-heavy text
Synopsis
Literary translation now gets its reinforcement-learning reward from a large model scoring against written criteria, instead of human preference labels or paired reference translations. Earlier DRT reasoning translation models were supervised on synthesized long thought and translation data, and reinforcement-learning routes such as R1-T1 and MT-R1 used machine translation metrics like COMET and CometKiwi as reward. Built on Qwen2.5-7B, DeepTrans-7B improves over its backbone by 16.3%, 13.8% and 9.0% on GEA5, GEA100 and GRF on MetaphorTrans, and beats sub-30B baselines on most metrics.
Interpretation
The reinforcement-learning stage of literary translation can use source sentences alone, without paired reference translations. Earlier DRT reasoning translation models were supervised on synthesized long thought and translation data, while reinforcement-learning routes such as R1-T1 and MT-R1 still used machine translation metrics like COMET and CometKiwi as reward. On MetaphorTrans, DeepTrans-7B, trained with reinforcement learning on source sentences alone, outperforms DRT-7B, which is trained on synthesized data, on both automatic metrics and GPT-4o evaluation.
A large model supplies the reward by scoring against written criteria, judging both the thought process and the translation. The criteria grade the thought process as a lack of analysis, slight analysis or detailed analysis, score the translation on a 100-point scale, and zero the overall reward when the output format is wrong. The reward model is DeepSeek-v3 and training uses GRPO, with 8 H20 GPUs serving the reward model and another 8 optimizing the policy, costing about 2K GPU hours for the reinforcement-learning stage.
Both training stages matter: cold-start supervised fine-tuning first, then reinforcement learning to shape translation strategy. DeepTrans-7B (Cold Start), trained only with cold-start fine-tuning, and DeepTrans-7B (Direct RL), which skips cold start, both fall below the full model on every metric. Across MetaphorTrans, O. Henry and Orbital, the full DeepTrans-7B reaches GRF of 88.84, 87.95 and 87.95, above both variants.
Perspective
Experiments cover a single translation direction, English to Chinese, with Qwen2.5-7B as the 7B backbone, evaluated on the MetaphorTrans test set and on two literary books, O. Henry and Orbital; the authors propose extending to multilingual translation, controlling translation style, more efficient translation rewards, and balancing long and short thought.
The authors note that the DeepSeek-v3 reward model needs substantial compute to run, that resource limits kept experiments at 7B parameters, and that only English-to-Chinese was covered; mid-training the model drifted toward a classical Chinese style, which the authors treat as a reward-model preference rather than a goal and recommend future work evaluate.