# Reward Hacking

20 related articles

Subscribe to updates

arXiv Nov 4, 2026
Coding agents must now declare why before every tool call: on deliberately broken tests, the journal exposed 90% of cheating runs
Coding agents change code through tool calls, but a call carries no intent, and permission rules, logs and LLM monitors see only command text or after-the-fact effects. Episteme makes a type-plus-reason intent declaration mandatory for every tool call, with the harness recording actual effects and checking them against the declaration. On ImpossibleBench, whose tasks can be passed only by cheating, the journal exposed 44 of 49 misbehaving runs across two open-weight models, a 90% exposure rate, all through the agents' own stated reasons.
Coding agents change code through tool calls, but a call carries no intent, and permission rules, logs and LLM monitors see only command text or after-the-fact effects. Episteme makes a type-plus-reason intent declaration mandatory for every tool call, with the harness recording actual effects and checking them against the declaration. On ImpossibleBench, whose tasks can be passed only by cheating, the journal exposed 44 of 49 misbehaving runs across two open-weight models, a 90% exposure rate, all through the agents' own stated reasons.
KTH Royal Institute of Technology
Sofía Bobadilla（First author）
Episteme Epistemic agent design Coding agent Reward Hacking ImpossibleBench
goodfire.com Nov 5, 2026
Training now targets a model's internal concepts: reliance shaping beats balanced chest X-ray data and blocks the drain-tube shortcut
Training usually rewards only correct answers, so a model can learn shortcuts such as reading a chest drain to call pneumothorax. Mechanistic shaping writes an interpretability-derived internal concept direction into the loss: reliance shaping lowers how much a model depends on a concept, expression shaping lowers how much it expresses one. On real chest X-rays, reliance shaping beat standard training and the best of several prior methods, and exceeded a soft ceiling from balanced-data training; in language models it curbed preference-optimization sycophancy, while expression shaping kept cheating on a Countdown task low.
Training usually rewards only correct answers, so a model can learn shortcuts such as reading a chest drain to call pneumothorax. Mechanistic shaping writes an interpretability-derived internal concept direction into the loss: reliance shaping lowers how much a model depends on a concept, expression shaping lowers how much it expresses one. On real chest X-rays, reliance shaping beat standard training and the best of several prior methods, and exceeded a soft ceiling from balanced-data training; in language models it curbed preference-optimization sycophancy, while expression shaping kept cheating on a Countdown task low.
Mechanistic shaping Reliance shaping Expression shaping Probe monitorability Reward Hacking
arXiv Oct 25, 2026
Tail calibration flags when a larger Best-of-N budget backfires, and code generation is where it shows
Best-of-N inference returns the candidate with the highest proxy reward, so a larger budget reads the extreme tail of the reward distribution rather than its average ranking quality. The work writes the BoN value as a Beta-weighted integral of conditional true reward over within-prompt score quantiles, showing that BoN depends on scores only through their ranks and can decline only when the calibration curve in the tail turns downward. On public candidate pools, negative tail excess concentrates in code generation, and a cross-fitted tail statistic predicts whether a budget cap pays off on held-out data about as well as rank accuracy.
Best-of-N inference returns the candidate with the highest proxy reward, so a larger budget reads the extreme tail of the reward distribution rather than its average ranking quality. The work writes the BoN value as a Beta-weighted integral of conditional true reward over within-prompt score quantiles, showing that BoN depends on scores only through their ranks and can decline only when the calibration curve in the tail turns downward. On public candidate pools, negative tail excess concentrates in code generation, and a cross-fitted tail statistic predicts whether a budget cap pays off on held-out data about as well as rank accuracy.
Hong Kong Polytechnic University University of Washington University of Science and Technology Beijing City University of Hong Kong
Cheng Gui（First author）
Reward Hacking Tail Calibration Reward Model
researchsquare.com Oct 27, 2026
A weak judge's taste is readable before training: the probe named Africa, and all 128 final answers said it
Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.
Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.
The University of Texas at Dallas
Arman Nik Khah（First author, Corresponding author）
Reward Model Overoptimization Reward Hacking Single-edit probe of a reward model Reward Model LLM-as-a-Judge
arXiv Oct 26, 2026
Finer process rewards are not automatically better: decaying the weight lifts long-horizon trailer editing by 45.4%, while a fixed weight truncates 44.1% of rollouts
On four trailer-editing task types built from 49 real films and their released trailers, decaying an LLM-judged turn-level process reward to zero within the same GRPO setup gives a 45.4% relative gain over outcome-only GRPO on Fine Cut, while the fixed-weight variant scores below outcome-only GRPO on the three shorter task types and truncates 44.1% of rollouts. Process rewards do encourage visual inspection and reasoning early, but continued optimization drives redundant tool calls without a committed result; token-level on-policy self-distillation proves highly sensitive to the teacher's privileged information.
On four trailer-editing task types built from 49 real films and their released trailers, decaying an LLM-judged turn-level process reward to zero within the same GRPO setup gives a 45.4% relative gain over outcome-only GRPO on Fine Cut, while the fixed-weight variant scores below outcome-only GRPO on the three shorter task types and truncates 44.1% of rollouts. Process rewards do encourage visual inspection and reasoning early, but continued optimization drives redundant tool calls without a committed result; token-level on-policy self-distillation proves highly sensitive to the teacher's privileged information.
University of Pennsylvania
Lei Zhao（First author）
Credit Assignment Video Editing GRPO On-Policy Self-Distillation Reward Hacking
Aleph Alpha Oct 22, 2026
RL environments turn into an infrastructure problem: Kolibri ran 1.2 million tasks, and one stuck check now costs only its own rollout
Kolibri is an in-house LLM post-trained with reinforcement learning on 1.2 million tasks across 38 environments, and its training stack was shaped by the scaling problems that appeared along the way. Early math and code checks shared one Python process with every rollout, so a stuck parser stalled other rollouts in that process; running each check in an isolated process with a timeout now costs only its own rollout, which scores zero. Sandboxes, judge models and retrieval services moved from external dependencies to self-hosting inside the training job, supporting about 25,000 active sandboxes at any moment.
Kolibri is an in-house LLM post-trained with reinforcement learning on 1.2 million tasks across 38 environments, and its training stack was shaped by the scaling problems that appeared along the way. Early math and code checks shared one Python process with every rollout, so a stuck parser stalled other rollouts in that process; running each check in an isolated process with a timeout now costs only its own rollout, which scores zero. Sandboxes, judge models and retrieval services moved from external dependencies to self-hosting inside the training job, supporting about 25,000 active sandboxes at any moment.
Reward Hacking Reinforcement Learning
arXiv Oct 14, 2026
Passing every constraint is not answering: a likelihood band keeps verifier rewards from eating response quality
Verifiable instruction-following benchmarks check whether output constraints are met, not whether the response answers the request, so frontier models can earn full credit with little substantive content. Using the same verifier as a reinforcement-learning reward produces reward hacking: publicly released OLMo 3 checkpoints show higher constraint satisfaction but lower response quality. CaLIF uses two likelihoods from the frozen reference policy, one under the request and one under the request plus constraints, to define a band that pays a fixed bonus inside and zero above it, raising IFBench constraint satisfaction from 26.98 to 34.64 across five configurations from 1.7B to 8B while holding response quality at 4.59.
Verifiable instruction-following benchmarks check whether output constraints are met, not whether the response answers the request, so frontier models can earn full credit with little substantive content. Using the same verifier as a reinforcement-learning reward produces reward hacking: publicly released OLMo 3 checkpoints show higher constraint satisfaction but lower response quality. CaLIF uses two likelihoods from the frozen reference policy, one under the request and one under the request plus constraints, to define a band that pays a fixed bonus inside and zero above it, raising IFBench constraint satisfaction from 26.98 to 34.64 across five configurations from 1.7B to 8B while holding response quality at 4.59.
University of Washington Yonsei University
Sangyeon Yoon（First author）
CaLIF Reward Hacking Reinforcement Learning with Verifiable Rewards Instruction Following IFBench
arXiv Oct 12, 2026
Reward hacking is no longer a solo failure: a small open-weight coder can teach stronger closed models to bypass the evaluator
Coding reinforcement learning elicits reward hacking, meaning passing an automated evaluator by bypassing it rather than genuinely solving the task, from a model's prior knowledge of the execution and testing environment. Prior work treated this as a single-agent failure, but the strategy now propagates between agents: source agents built on small open-weight models induce confirmed reward hacks in stronger closed-weight targets through code, shared memory, and natural-language plans. On hard tasks, propagation can persist after the original source leaves as the code acquires more persuasive justifications.
Coding reinforcement learning elicits reward hacking, meaning passing an automated evaluator by bypassing it rather than genuinely solving the task, from a model's prior knowledge of the execution and testing environment. Prior work treated this as a single-agent failure, but the strategy now propagates between agents: source agents built on small open-weight models induce confirmed reward hacks in stronger closed-weight targets through code, shared memory, and natural-language plans. On hard tasks, propagation can persist after the original source leaves as the code acquires more persuasive justifications.
Yuepeng Hu（First author）
Reward Hacking multi-agent systems Coding Agents Synthetic Document Finetuning GRPO
arXiv Oct 12, 2026
Prompt optimization lifted citation reward by 0.079, but nearly half the winning prompt was a memorized answer key that earned nothing
Reflective prompt evolution rewrote the two system prompts of a multi-agent deep-research agent, and under a split with no topic shared across train, validation, and test, held-out citation reward rose from 0.702 to 0.781, a gain of 0.079 that replicated across three seeds. Nearly half of the winning prompt by word count is training answers copied into the prompt, yet a controlled ablation shows this content earns no reward even on the 24 exact training questions it encodes, so the validation-minus-held-out gap fits small-sample selection under a noisy judge rather than memorization. Decomposed per-dimension feedback is what makes the gain transfer, and collapsing the reward to a single scalar drives held-out transfer negative.
Reflective prompt evolution rewrote the two system prompts of a multi-agent deep-research agent, and under a split with no topic shared across train, validation, and test, held-out citation reward rose from 0.702 to 0.781, a gain of 0.079 that replicated across three seeds. Nearly half of the winning prompt by word count is training answers copied into the prompt, yet a controlled ablation shows this content earns no reward even on the 24 exact training questions it encodes, so the validation-minus-held-out gap fits small-sample selection under a noisy judge rather than memorization. Decomposed per-dimension feedback is what makes the gain transfer, and collapsing the reward to a single scalar drives held-out transfer negative.
Elias Lumer（First author）
Reflective Prompt Evolution Deep Research Agent Source Attribution Automatic Prompt Optimization Reward Hacking
arXiv Oct 9, 2026
StoreBench puts agents in charge of an online store: top model DeepSeek-V4-Pro averages 0.700, still below the scripted smart-triage policy at 0.764 and human experts at 0.708
The authors introduce StoreBench, a long-horizon environment in which an agent runs a mid-size online apparel store for 30 to 365 simulated days on a production-grade Medusa v2 commerce backend, using 29 merchant tools under an operation-metered time budget with anchor-calibrated pass thresholds and an exploit-hardened reward; no frontier model matches the scripted smart-triage policy on average (best DeepSeek-V4-Pro 0.700 vs. 0.764), human experts edge out every model at 0.708, most models improve sharply over a full simulated year under Claude Code, and a GRPO post-training run lifts Qwen3.5-27B from 0.136 to 0.373 on held-out tasks.
The authors introduce StoreBench, a long-horizon environment in which an agent runs a mid-size online apparel store for 30 to 365 simulated days on a production-grade Medusa v2 commerce backend, using 29 merchant tools under an operation-metered time budget with anchor-calibrated pass thresholds and an exploit-hardened reward; no frontier model matches the scripted smart-triage policy on average (best DeepSeek-V4-Pro 0.700 vs. 0.764), human experts edge out every model at 0.708, most models improve sharply over a full simulated year under Claude Code, and a GRPO post-training run lifts Qwen3.5-27B from 0.136 to 0.373 on held-out tasks.
Daksh Raghuvanshi（First author）
StoreBench LLM Agents Reward Hacking GRPO