Skip to main content
Back to timeline
arXivSource publication:

Agents inflate confidence when it buys compute: audited pivot scoring prices the allocation, cutting cutoff crossings

Synopsis

In multi-agent systems, confidence often decides who receives scarce downstream computation and decision weight, which gives agents an incentive to overstate it. The Audited Pivot Scoring Mechanism (APSM) offsets platform-controlled allocation benefits in the transfer, verifies outcomes through inverse-propensity-weighted sparse audits, charges the Clarke pivot externality, and adds a strictly proper score, making truthful reporting the unique expected-utility best response. Calculator-free experiments show agents inflate confidence under Top-K incentives; with an agent-local utility interface, APSM yields lower Brier loss and fewer cutoff crossings than Compensated Proper Scoring (CPS), while CPS achieves a higher exact allocation match.

Source-provided article image: Strategic Calibration in LLM Multi-Agent Systems: Audited Pivot Scoring for Truthful Elicitation
arXiv

Interpretation

In multi-agent coordination, confidence is no longer only a description of uncertainty but an allocative signal that determines how limited downstream attention, computation and decision weight are distributed, so reporting becomes a strategic action. Earlier calibration methods largely treat a reported probability as a passive prediction, improving confidence quality through post-hoc calibration, prompted elicitation or multi-agent deliberation, and do not address the reporting incentives created when confidence controls selection. In calculator-free experiments, holding candidates and initial confidence fixed, a Selection-only condition that keeps the allocation benefit raises confidence relative to a neutral re-report, and a substantial share of eligible agents cross the Top-K cutoff; adding a scale-matched Brier score shrinks the shift but some agents still cross.

APSM offsets the platform-controlled allocation benefit in its transfer, so an agent cannot gain a discrete selection benefit by inflating its report. A strictly proper score alone is insufficient: crossing a hard allocation threshold yields a discrete gain, while the Brier loss from a small exaggeration is quadratic, so exaggeration near the threshold can remain profitable. Under fixed candidate content, risk-neutral agents that maximize current-round utility, report-independent audits and accurately modeled platform benefits, the mechanism yields truthful reporting as the unique expected-utility best response; in controlled replay on two six-agent workflows, AgentWebBench and MultiAgentBench database diagnosis, APSM drives reference deviation to zero and exactly reproduces the reference allocation.

APSM verifies outcomes through inverse-propensity-weighted sparse audits and charges the Clarke pivot externality, making an agent bear the opportunity cost its occupation of a limited coordination slot imposes on other agents. Compensated Proper Scoring (CPS) keeps benefit compensation and proper scoring but omits the audited Groves allocation term, so it lacks the belief-space loss incurred when a deviation induces an inefficient allocation. Under exact execution CPS and APSM both reach zero reference deviation and exact allocation match; the difference appears under approximate execution, where at utility noise of 0.05 APSM reduces allocation-changing errors from CPS's 17.2% to 11.2% and raises NDCG@5 from 0.304 to 0.356, and the advantage persists after matching mean utility range.

Perspective

The demonstrated scope is fixed candidate content, risk-neutral agents that maximize only current-round expected utility, report-independent sparse audits, and allocation benefits the platform can accurately model and offset; evaluation covers 200 web-retrieval queries in AgentWebBench and 50 PostgreSQL incidents in a MultiAgentBench database extension, each with two Qwen3-8B, two Qwen3-4B and two DeepSeek-V4-Pro agents and three paired seeds.

The truthfulness guarantee relies on the platform accurately modeling the allocation benefit it controls; under deterministic Top-K, even a small residual uncompensated benefit can reintroduce threshold manipulation near the cutoff. Sparse auditing preserves expected incentives but lower audit rates amplify transfer variance and tail magnitude. The reference belief is an agent-local proxy rather than observable truth, and APSM is not uniformly easier to execute than CPS.