Autoregressive training drops strict left-to-right targets: Max(MI(S;t)) reorders by mutual information and lifts arithmetic accuracy to 94.96%
Synopsis
Autoregressive language models normally learn next-token prediction in a fixed left-to-right order; Max(MI(S;t)) instead ranks target tokens by their mutual information with the source and already-selected targets before training, exposing informative targets earlier. On GPT-style models it reaches 94.96% average accuracy across arithmetic tasks, against 74.84% for next-token prediction and 86.29% for reverse order, close to the 97.33% upper bound from exhaustively testing 24 orders; on WikiText-2 it lowers average perplexity from 13.58 to 10.77. Only the training-target organization changes; architecture and autoregressive decoding stay the same.
Interpretation
The order of target tokens during training becomes a design choice rather than a fixed by-product of next-token prediction. The default left-to-right order is often far from the best tested order, ranking 18th, 20th and 15th in 3-digit addition, 2-digit multiplication and multi-label classification respectively. After exhaustively evaluating all 24 orders for each four-output task, Max(MI(S;t)) ranks 5th, 1st and 7th in the same three settings, with accuracy close to each optimum.
Ranking uses the mutual information between each candidate target token and the current source, meaning how much information they share. A fixed hand-crafted order such as reverse order helps only on addition, falling to 4th and 13th in multiplication and multi-label classification. On GPT-style models Max(MI(S;t)) averages 94.96% arithmetic accuracy versus 74.84% for next-token prediction and 86.29% for reverse order; on multi-task Qwen-Math fine-tuning it averages 96.0% versus 92.2% and 95.0%.
For fixed-length outputs, the method repeatedly picks the remaining target position with the highest mutual information to build a task-specific prediction order. Arithmetic and multi-label targets are small and discrete, so mutual information comes from empirical counts and preprocessing needs only one offline counting-and-sorting pass. Average accuracy on English multi-label classification is 83.40%, above 82.25% for next-token prediction and 81.36% for reverse order; on the Chinese-translated ToxicComment it averages 76.34% versus 74.16% and 72.86%.
For open-vocabulary text generation, the same principle becomes duplicating one selected information-rich word at the start of each sentence as a prefix. Standard next-token prediction averages 13.58 perplexity on WikiText-2, TF-IDF word selection 11.90 and conditional-probability selection 11.44. Max(MI(S;t)) lowers average perplexity to 10.77; on XSUM summarization its average ROUGE is 0.2323, slightly above 0.2313 for TF-IDF and 0.2309 for conditional probability.
Perspective
The demonstrated scope is GPT-style models trained from scratch plus Llama and Qwen decoder models, on arithmetic, multi-label classification, WikiText-2 continued pretraining and XSUM summarization; only the training-target organization changes, while model architecture and standard autoregressive decoding stay fixed.
Max(MI(S;t)) is a heuristic and does not guarantee the optimal prediction order; in long-sequence multiplication all strategies degrade sharply as output length grows and fail in the longest settings. The Chinese multi-label results rest on machine-translated ToxicComment, so language effects cannot be separated from translation artifacts, and larger models and other architectures still need validation.