Chip characterization agents stop grading tool calls: final Liberty files lift composite score from about 55% to about 95%
Synopsis
Siemens Fuse used NVIDIA NeMo Gym to build an evaluation loop for the Solido Characterization Suite agent that grades final Liberty files, raising the mean composite verification score on standard-cell characterization tasks from about 55% to about 95% while cutting agent turns from 155 to 53 and output tokens from about 81K to 29K. Previously such multistage flows could only check whether individual tool calls succeeded, and an early setup error could propagate silently downstream; now a composite reward combining coverage, validity, relative validation, integrity and the in-tolerance fraction scores the final artifact of each rollout, and those findings guide rewrites of skills and MCP tool descriptions.
Interpretation
Reliability evaluation for chip design agents moves from checking whether individual tool calls succeeded to checking the final engineering artifact produced by a multistage flow. In multistage characterization flows, an early setup error could propagate silently through downstream stages, and intermediate checks did not see those errors. Final Liberty verification exposed missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance that intermediate checks missed.
The evaluation loop starts each attempt from a clean state and runs baseline rollouts with the same task, tools, state and verifier while recording thinking traces, input and output messages, and tool actions. The team inspected trajectories, failed tool calls and verifier diagnostics to identify ambiguous instructions, unproductive actions and incorrect tool selection. The composite reward multiplies four gates, coverage, validity, relative validation and integrity, by the fraction of output values within tolerance, and is zero if any gate fails.
On the open source GF180MCU PDK, the benchmark dataset covers 229 standard cells across 36 PVT corners with Solido AFS for SPICE simulation. Each run was restricted to two cells across 36 PVT corners, because this smaller subset was sufficient for tuning the agentic workflow. Each run took about 1.5 to 2 hours, and the team conducted about 25 rollouts using Claude Opus 4.8 in Claude Code as the fixed model and agent harness.
Across evaluation rounds, the team refined the agent's characterization skills and MCP tool descriptions, simplified overlapping file and shell interfaces, and defined tool inputs with explicit JSON schemas. These changes made the characterization tools easier for the agent to call and reduced incorrect tool selection. The mean composite verification score across evaluated rollouts rose from about 55% to 95%, turns fell from 155 to 53, about 66%, and output tokens fell from about 81K to 29K, about 64%.
Perspective
The evaluation targets standard-cell characterization: Liberty file generation on the open source GF180MCU PDK with Solido AFS for SPICE simulation, on a dataset of 229 standard cells across 36 PVT corners, while tuning runs used two cells across 36 PVT corners. The intended readers are chip design teams and EDA developers, who can run an initial evaluation and then apply the outlined steps to one representative task: define a valid final artifact, build a domain-specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent conditions.
Each tuning run covered only two cells across 36 PVT corners, which the source states was sufficient for tuning the agentic workflow; the about 55% to about 95% composite score is a mean across the evaluated rollouts, and the turn and output-token reductions come from the same evaluations.