Data science agents often score well while breaking method rules: across 220 tasks, more data leakage tracked with higher predictive performance
An evaluation across 220 data science tasks finds most agentic solutions contain at least one methodological failure, and that data leakage failures correlate positively with predictive performance, so accuracy alone misses method problems.