Data science agents often score well while breaking method rules: across 220 tasks, more data leakage tracked with higher predictive performance

Evaluation of data science agents now looks beyond predictive scores and runnable code to whether the agent violates data science methodology. Earlier agentic data science evaluations measured downstream predictive performance and code correctness, giving only a preliminary picture of capability. A large-scale analysis of failure modes in common agentic frameworks and models across 220 data science tasks.

The failure modes are packaged into a reusable test suite that triggers six specific errors under controlled conditions. Previous evaluations folded methodological problems into an overall score, so a single failure class could not be isolated. The suite uses curated data science tasks to test six specific failures in a controlled manner.

Agentic pipelines were compared qualitatively against expert solutions from Kaggle competitions, where the gap shows up in problem-solving creativity and domain knowledge. Predictive performance comparisons alone do not reveal how agent reasoning differs from that of winning data scientists. A qualitative assessment of agentic data science pipelines against expert solutions on Kaggle competitions.

The evaluation targets researchers and engineering teams using data science agents, covering common frameworks and models on 220 data science tasks, plus a reusable and extensible controlled test suite for six specific failures. The material also mentions a qualitative comparison against Kaggle competition expert solutions, which readers can use to judge how methodological failures appear in a real competition setting.

The material is a summary, so the definitions of the six failures, the task composition of the test suite, and the scale of the qualitative comparison are not spelled out. The positive correlation between data leakage and predictive performance is an observed association, and the material offers no causal explanation.