Customer-support agent benchmark scores 17 models with five judges: GPT-5.4 wins everywhere, judges agree only moderately

Skill execution in customer-support agents is evaluated on its own, separate from orchestration and general tool use. Earlier tool-use evaluations mostly test whether a model picks tools across diverse tasks, whereas skill execution requires completing a stateful multi-turn workflow inside a given skill context under company policy. The evaluation covers 17 models, spanning the closed GPT-5, GPT-5.4 and GEMINI families and open-weight models from 1.7B to 14B parameters, each scored by five independent LLM judges plus a deterministic rule-based evaluator.

GPT-5.4 ranks first under all five judges, and QWEN3.5-9B is the strongest open-weight model below 20B parameters. QWEN3.5-9B beats several closed baselines, including GEMINI-2.5-FLASH-LITE, GPT-5-MINI and GPT-5-NANO, under every judge, and also beats the larger QWEN3-14B and MINISTRAL-14B. The five judges preserve the same model ranking, but absolute pass rates for the same model vary substantially across judges, with mean pairwise agreement of 84.2% and moderate Cohen's kappa.

Benchmark scenarios come from real customer-support calls rather than free generation by a frontier model. Earlier synthetic dialogue benchmarks often sample from a fixed template distribution, so scenarios can drift from the workflow distribution seen in production. The pipeline mines 858 workflow clusters from roughly 350K real transcripts, keeps 80 skills after quality validation and manual expert review, generates 1800 candidate scenarios, and selects 500 for evaluation.

Tool calls run against simulated environments that keep the production interface without touching real customer data. Connecting directly to live production tools imposes rate limits, risks changing real customer data, and often lacks an isolated test environment. 281 MCP-compatible tools are drawn from the production tool catalogue, and the mocking layer returns scenario-conditioned structured responses while validating inputs, simulating configurable failures, and maintaining conversation history and caching.

The evaluation targets enterprise customer support, covering the top 10 industries by annual recurring revenue and the top 3 organizations per industry by call volume, with skills and tool catalogues drawn from those organizations' production environments. The intended readers are teams choosing a zero-shot skill-execution model under cost and latency constraints, and engineers whose production pipelines rely on an LLM judge to gate deployment, filter training data, or monitor live agent quality.

Results are limited to customer-support workflows from the selected industries and organizations, so they may not generalize to all domains, policies, or tool interfaces. The MCP-compatible mocks preserve production-facing schemas but remain simulations, and LLM-based generation and judging can miss subtle workflow errors. All judging is single-sample, and multi-sample judging with majority aggregation could reduce some of the reported disagreement. The main open-weight evaluation is capped at sub-20B scales; the 27B extension shows scaling within a generation helps, while the gap to the strongest closed models remains substantial.