Majority-vote test-time RL locks each question onto one answer, and a kernel probe predicts which one before training without fitted constants
Synopsis
Majority-vote test-time reinforcement learning makes each question lock onto one answer during training, often a wrong one on hard questions, and until now which answer that is became known only after training. The work treats this as a short race driven by sampling noise between the answers that are close at the start, and gives a closed-form probability for how the network kernel shifts the gap of the two leaders through updates of the trailing answers. In LoRA training on MMLU-Pro multiple-choice questions with three models from two families, predictions from a frozen-kernel chain computed before training forecast which answer each question locks into better than a logistic model refitted on the outcomes, using no fitted constants.
Interpretation
Majority-vote test-time reinforcement learning can be given a probability for each question's locked answer before training starts, instead of waiting for training to end. Earlier analyses either detected lock-in during training from flip rates or described GRPO with noisy rewards as a deterministic flow, and none gave the probability with which each answer wins a question. In LoRA training on MMLU-Pro multiple-choice questions with Qwen2.5-1.5B, Llama-3.2-1B and Qwen2.5-7B, the frozen-kernel chain fixed before training had a better Brier score than a cross-fitted logistic regression in five unseen settings, a second model family, a larger size, AdamW and the standard-deviation baseline, without any constant fitted to the outcomes.
In a network the answer logits share parameters, so updates of the trailing answers change the gap between the two leaders through the parameter kernel, and this spillover moves the boundary between the starts each leader wins. A scalar law that summarizes the kernel by one scale omits this shift, so its step has to be inflated by a factor of about 4 to 150 to fit the outcomes. The closed form accumulates the spillover along one deterministic path, and on runs where a check before training finds the two leaders stay above the other answers it matches the frozen-kernel chain, with a Brier score about 0.002 higher and an interval that contains zero.
Questions that share parameters can be updated together, their lock-ins become independent in the limit, and each question's shift adds the mean updates of the other questions along their tied paths. Looking at one question in isolation misses this cross-question push, and the coupled chain predicts each member's winner better than the isolated chain. In sets of 16 to 64 questions with shared parameters, the leader won in members at rates near the prediction, the shifts of letter frequencies correlated with the prediction at 0.87, and the closed form closed 100% and 89% of the Brier gap between the isolated law and the coupled chain at sizes 16 and 64.
Perspective
The conclusions cover single-token answers from a fixed set of ten letters, LoRA adapters that stay close to their linearization until the race is settled, and an objective with no clipping, no KL term and no exploration term; a kernel probe before training can check whether a question lies in the scope of the two-leader law.
The probabilities rank and discriminate well, but calibration depends on the model and setting, with calibration slopes for Qwen-1.5B in the unseen settings between 0.76 and 1.06, so no calibration claim is made; decisions come earlier than the frozen kernel predicts because the kernel grows late in training, and momentum lies outside the model; with shared parameters the closed form is more confident than the coupled chain because the gaps of different questions are coupled and their winners become correlated at practical step sizes; the error order for three or more answers is only observed in simulations.