Meta-agent-generated terminal tasks let a 9B model reach 81.3% mean pass@2 within 20 steps, dropping to 20.6% after hard tasks are added

The work presents a meta-agent pipeline for generating terminal tasks and verifiers, and uses it to diagnose three failure classes: benchmark invalidity, harness brittleness, and reward misalignment. Common practice treats a runnable Docker image and an executable test suite as evidence that a pipeline works; the work argues these do not guarantee a faithful end-to-end pipeline for terminal agent training and names the failure classes explicitly. Evidence comes from the pipeline's own diagnostic process; the abstract presents the three failure classes by name and description without independent quantitative breakdowns for each.

Prompt redesign and context extension raise baseline solvability 5.6 times. This magnitude identifies prompt and context design in the data-generation stage as a key variable affecting solvability, beyond task difficulty alone. The abstract reports the 5.6-times factor but does not state the baseline absolute value, task counts, or evaluation protocol details.

A 9B model saturates at 81.3% mean pass@2 within 20 steps on Claude Opus-generated tasks; adding hard tasks reduces mean pass@2 to 20.6% without changing the training configuration. Changing only the task difficulty distribution under the same training configuration produces this shift, which the abstract presents as strong evidence that the solvability band is model-specific. Evidence is a controlled observation under the same model and training configuration, with the abstract giving the two values 81.3% and 20.6% and the 20-step training range.

The work argues that meta-agent reliability requires solvability-band calibration, verifier audits, and infrastructure error accounting as first-class evaluation criteria rather than post-hoc diagnostics. Elevating these three from after-the-fact troubleshooting to prerequisites in the evaluation process changes how training data generated by meta-agents is evaluated. The claim is supported by the preceding failure-class diagnosis and the solvability-band observation, and is a methodological recommendation grounded in this pipeline's experience.

The work targets researchers and engineering teams who use frontier models as meta-agents to generate tasks and verifiers for terminal-agent RL training, and applies to settings where one must judge whether generated tasks fall within a target model's solvability band. Its diagnostic framework and solvability-band calibration can be used directly to design task difficulty distributions, audit verifiers, and account for infrastructure errors; the observation that a 9B model saturates at 81.3% within 20 steps and drops to 20.6% after hard tasks are added provides an actionable reference for calibrating a model-specific solvability band before training.

The abstract does not state task counts, task-source distribution, or the specific verifier audit procedure, nor does it give quantitative shares for each of the three failure classes, making their relative weight in the overall pipeline hard to judge. The baseline absolute value and evaluation conditions behind the 5.6-times gain are not listed, and the task-set sizes and definition of hard tasks behind 81.3% and 20.6% are not expanded. The conclusion that the solvability band is model-specific currently rests on observations from one 9B model, so whether it holds for other model scales or architectures remains open. In addition, this reading is based on the abstract, with figures and body details not included, so the statistical uncertainty and reproducibility of these values await confirmation in the original.