# AutoDataBench

The work introduces AutoDataBench, a controlled testbed that isolates data intelligence under fixed training pipelines and resource budgets across tool-use data diagnosis and repair, retrieval data organization, and knowledge injection, evaluates seven frontier LLMs on iterative data decisions, finds that agent-curated data approaches expert references on tool use and retrieval while strong in-distribution scores do not guarantee out-of-distribution transfer, and shows that reusing optimization trajectories as mid-training data improves downstream coding performance.

The work introduces AutoDataBench, a controlled testbed that isolates data intelligence under fixed training pipelines and resource budgets across tool-use data diagnosis and repair, retrieval data organization, and knowledge injection, evaluates seven frontier LLMs on iterative data decisions, finds that agent-curated data approaches expert references on tool use and retrieval while strong in-distribution scores do not guarantee out-of-distribution transfer, and shows that reusing optimization trajectories as mid-training data improves downstream coding performance.

The work introduces AutoDataBench, which turns "can an agent write a training task that a data pipeline would accept sample by sample" into an evaluation target: given an original benchmark task and a record of the target model attempting it, an author agent must write a new task for the same suite, and the score multiplies a gate, whether the target model's pass rate lands in a band, and coverage of a hidden rubric of behavioural modes; across three benchmarks of executable agent tasks, no agent evaluated scores above 20 out of 100 at the default 45-minute budget, and giving the strongest agent four times as long raises the score substantially while the cost of one usable task stays almost unchanged.

The work introduces AutoDataBench, which turns "can an agent write a training task that a data pipeline would accept sample by sample" into an evaluation target: given an original benchmark task and a record of the target model attempting it, an author agent must write a new task for the same suite, and the score multiplies a gate, whether the target model's pass rate lands in a band, and coverage of a hidden rubric of behavioural modes; across three benchmarks of executable agent tasks, no agent evaluated scores above 20 out of 100 at the default 45-minute budget, and giving the strongest agent four times as long raises the score substantially while the cost of one usable task stays almost unchanged.