Skip to main content
Back to timeline
arXivSource publication:

Multi-agent teams stop adding agents by default: estimate the backbone's competence first, and a strong model saves 36% of tokens

Lead

Multi-agent collaboration is now sized by how competent the backbone is on the current query: 93.81% average across six benchmarks, and on higher-competence backbones average accuracy rises from 87.22 to 89.97 while token cost per query drops 36.26%.

Source-provided article image: Competence-Aware Orchestration for LLM-Based Multi-Agent Systems
Figure 1 · arXiv

Story

A multi-agent system no longer applies one collaboration structure to every query; it first estimates how competent the backbone is on the current query, then decides which roles take part and how those roles communicate. Earlier automatic-structure methods organized collaboration mainly around task semantics or runtime context, with limited explicit modeling of the backbone's competence on the current query. On MMLU, across a group of higher-competence backbones, average accuracy rises from 87.22 for G-Designer to 89.97, while token usage per query after deployment falls by 36.26%.

Competence is first written as a reusable backbone capability fingerprint, then combined with the current query into a task-conditioned competence representation that serves as the control signal for orchestration. The fingerprint is measured once from a fixed 48-item probe set, laid out as four capability domains by twelve probes ordered easy to hard, so different backbones can be compared directly. Supervision comes from empirical success rates of 14 backbones on 192 non-probe queries, giving 1060 training instances; in ablation, removing the capability fingerprint lowers the average score by 3.02 and raises token cost by 154.64%.

Both role activation and communication-graph generation read this competence representation: high competence lowers the average activation scale, while low competence opens up more roles and denser communication. Role activation has no direct labels, so training uses task-role similarity as a semantic anchor and reuses the query-level success rate as a soft budget prior that regulates the overall activation scale. On a group of efficiency-oriented backbones, the average score rises from 74.07 for G-Designer to 78.65; the same trend holds on GLM 5.1 and Gemini 3.1 Pro, which were used neither for probe construction nor for supervision collection.

What to watch

The reported evaluation covers six benchmarks: MMLU, GSM8K, MultiArith, SVAMP, AQuA and HumanEval, with accuracy for the first five and Pass@1 for HumanEval. Cross-backbone results are given on MMLU, with supplementary evaluation on GPQA Diamond and BBEH-mini using the unseen backbone Gemma4. It is aimed at researchers and engineers building or tuning multi-agent reasoning pipelines.

Competence supervision comes from empirical success rates estimated over repeated runs, so it is a proxy signal rather than a direct label. Role activation has no direct supervision and relies on two weak signals, a semantic anchor and a soft budget prior. The efficiency comparison counts only deployment-time token usage, kept separate from offline supervision collection and model training; the one-time fingerprint registration cost is amortized as an average over three performance-oriented backbones, recovered after about 28 queries. Conclusions on unseen backbones are currently given mainly on MMLU.

Sources