Test cases stop being hand-written one by one: five agents split the work and audit themselves, averaging a 93.4% mutation score across five datasets
Synopsis
Software test-case design has long depended on manual work, and models that treat generation as one-shot text translation struggle with multi-layered business rules. A pipeline of five roles, parser, triage, architect, writer and auditor, turns test generation into a stateful reasoning process in which the auditor corrects scripts before output. Across five datasets spanning travel, telecom, financial regulation and a government job platform, the pipeline averaged a 93.4% mutation score, 1.4% hallucination and 7.8% redundancy, at an average generation time of 21.2 hours.
Interpretation
Test-case generation becomes a stateful agent pipeline whose internal auditor checks and repairs scripts before output, instead of translating requirement text into cases in one pass. Earlier approaches treated generation as a stateless string-translation task; random forest, T5-Small, RAG, a genetic algorithm and a static Mistral 7B model each generated once and retained no context of multi-step business logic. Averaged over the ISIS Travel, Travco, Vantage Towers, ESMA FIRDS and JMP datasets, the pipeline reached a 93.4% mutation score, against 68.8% for static Mistral 7B, 57.2% for RAG and 15.6% for random forest.
The pipeline divides roles: a parser splits requirement text, a triage agent filters testable items, an architect fixes equivalence classes and boundaries, a writer emits JSON scripts, and an auditor acts as an internal compiler. The auditor carries a self-healing loop in which a JSON validator and syntax fixer repeatedly correct the payload, bounded by a circuit breaker of at most two retries that stops the endless refinement seen in early trials. In an ablation on the ESMA and JMP datasets, removing the auditor dropped syntactic correctness from 99.6% to 84.2%, and removing the architect dropped the mutation score from 93.4% to 76.1% while redundancy rose from 7.8% to 31.4%.
High-fidelity output costs compute: the agent pipeline needs A100 high-memory GPUs and takes far longer to generate than minute-scale traditional flows. Random forest, T5-Small, RAG and the genetic algorithm needed 5 to 10 minutes per dataset on T4 GPUs, static Mistral 7B averaged 16.6 hours, and the agent pipeline averaged 21.2 hours, reaching 30 hours on ESMA. Consolidated means over the five datasets show the pipeline produced 2,636 cases on average versus 4,658 for static Mistral 7B, while hallucination was 1.4% versus 13.6% and redundancy 7.8% versus 22.2%.
Perspective
The demonstrated setting is offline batch generation over five requirement documents: three commercial platforms whose requirements were reverse-engineered with Perplexity.ai, plus the ESMA FIRDS functional specification and a government job-matching platform's requirements. Metrics are mutation score, requirement traceability, hallucination rate, redundancy, branch coverage and syntactic correctness. The intended readers are software engineers and quality-assurance researchers following test automation.
Generative models are non-deterministic, so the same requirement can yield tests of differing quality across runs, which the authors mitigate with structured chain-of-thought prompting; early agent trials entered endless refinement of a single case, stopped by a hard-coded retry limit. The three commercial platforms' requirements are not original project documents but were reverse-engineered with Perplexity.ai.