Fireworks folds six Surge benchmarks into its Specialized Intelligence Index, testing whether models can get through a workday
Synopsis
Fireworks has added six Surge benchmarks to its Specialized Intelligence Index (SII); the benchmarks are written by practitioners and test whether models can finish real work in messy environments, with ambiguous instructions and calls that require judgment. Earlier evaluations focused more on well-specified tasks such as writing a patch, running a calculation or answering a question. The new benchmarks score the decisions around execution, using expert-authored rubrics that measure correctness, verification, judgment and communication.
Interpretation
The Specialized Intelligence Index now compares open, closed and specialized models on six Surge benchmarks built around real work. Earlier evaluations were better suited to well-specified tasks such as writing a patch, running a calculation or answering a question; the Surge benchmarks target the messy environments, ambiguous instructions and judgment that real work requires. The benchmarks are written by practitioners who have done the work, including staff engineers, portfolio managers, physicians and accountants, and every task goes through multiple rounds of review for correctness, completeness and solvability.
The benchmarks focus on decisions around execution: what matters, what could go wrong, and when a technically correct answer is professionally wrong. Surge uses expert-authored rubrics to measure correctness, verification, judgment and communication rather than a single pass-or-fail score. DAYJOB environments average 22 to 36 input files per task, with some exceeding 300; 44% of tasks are estimated to require 20 or more hours of professional work, and nearly half require 5 to 10 years of experience.
The software-engineering benchmark sudo L7 places tasks inside production codebases with years of history, touching systems including identity and access control, payments, ledgers, payroll and webhooks. These tasks come from privately licensed production codebases rather than clean, self-contained problems. Roughly 70% of sudo L7 tasks come from privately licensed production codebases across more than 75 repositories and more than 15 organizations.
Perspective
The news is the evaluation arrangement by which six Surge benchmarks join the Specialized Intelligence Index, and how those benchmarks are built; the public material reports no model scores or rankings on them. It suits engineering and research readers tracking model selection who want to know which professional settings and task scales the evaluation covers.
The material says models can differ substantially in judgment, reliability and ability to finish an assignment end to end, and that models with similar headline scores can behave differently, but it gives no specific evaluation numbers supporting that statement.