AI teams underperformed single agents in simulated service
Under a fixed simulated user, teams scored a weighted average of 28 percentage points below single agents.
Edited by Boris Djordjevic · Paperfoot
The study compared a centrally coordinated team of language-model agents with a single agent on τ²-bench, a benchmark of simulated user conversations governed by written policies. It covered airline, retail and telecom tasks, using six models from three families across about 7,950 conversations. Each team used the same model as its single-agent counterpart.
Under a fixed user simulator, teams performed worse in 12 of 13 valid model-and-domain comparisons and tied in one. In the two comparisons where an earlier rule predicted that collaboration would help weaker agents, teams instead lost by 13 and 22 percentage points. The only gain appeared with a second user simulator. Whether collaboration’s benefit declined as single agents grew stronger remained unresolved. Teams did not detectably reduce wrongful database changes.
Why it matters
These benchmark results challenged a rule that predicted benefits from collaboration when single agents were weak. The authors recommended single agents by default unless teams demonstrate gains in the intended setting with matched computing budgets.
Caveats
The evaluation used simulated conversations and one team design whose planner never received descriptions of the available tools, leaving one validity requirement unmet. The paper is a preprint, not yet peer-reviewed.
From the paper
- Limitation
Its orchestrator plans without the tool schemas that only the database agent holds, so under the stricter audit of §4.1 only 25% of its write orders were executed as issued, against the 90% the plan required; the pre-registered fidelity bar fails in every cell and the result is a labelled deviation.
Quoted word for word from the full text on arXiv.
In this paper
multi-agent collaboration in dual-control settings
- Methods
- centralized MAS (this paper)M2 (this paper)stock SAS (baseline)S2 (baseline)qwen3.5-4b (evaluated)qwen3.5-9b (evaluated)gpt-5-mini (evaluated)gpt-5-nano (evaluated)gemini-3-flash (evaluated)gemini-3.1-flash-lite (evaluated)gpt-4.1
- Datasets
- τ2-bench (evaluated)
- Links
- Code
Results
- S2, pass rate, telecom: 36.0%
Telecom × gemini-3.1-flash-lite (150 episodes, no infrastructure errors, $14.04): S2 36.0% [28.8,43.9] against 34.0% for the SAS and 12.0% for the MAS at 4×; per task against the SAS, better on 15, equal on 23 and worse on 12.
Results.
- S2, pass rate, airline: 45.3%
Airline × gpt-5-nano, the straddling cell (150 episodes, no infrastructure errors, $11.30): S2 45.3% [37.6,53.3] against 42.7% for the SAS and 45.3% for the MAS at 4×; per task against the SAS, better on 12, equal on 27 and worse on 11.
Results.
- S2, pass rate, telecom: 27.3%
Telecom × gpt-5-nano, the other cell below 45% (150 episodes, no infrastructure errors, $20.14): S2 27.3% [20.8,35.0] against 32.7% for the SAS and 19.3% for the MAS at 4×; per task against the SAS, better on 7, equal on 29 and worse on 14.
Results.
- qwen3.5-4b, score, telecom, served at 4-bit: 94.7%; qwen3.5-9b, score, telecom: 95.3%
The weakest open model, qwen3.5-4b served at 4-bit, scores 94.7% on telecom, above every closed model (qwen3.5-9b scores 95.3%), and adds no low cell.
Cells below 45%.
- gpt-5-mini, accuracy, telecom: 77.3%
Telecom separates the models most: the two low-tier closed models sit at 32.7 and 34.0%, gpt-5-mini at 77.3%, and gemini-3-flash and both open models at 91.3 to 95.3%.
Cells below 45%.
- qwen3.5-9b, pass1, excluding runaway calls, telecom: 34.2%
The route returned more than max_tokens on 17 (telecom) and 12 (retail) calls, of which 4 and 6 were runaway calls of about 131k reasoning tokens that exhausted the episode in one call, while the collaborator’s retail replication had 6 such calls and no runaway, so the route’s handling of the reasoning budget varies by run; pass1 excluding those episodes is 34.2% [27.0, 42.3] on telecom and 51.4% [43.3, 59.4] on retail …
The hosted route of the 9-billion model.
What it shows
- Under a fixed simulated user, the multi-agent system shows no benefit over the single-agent system across 13 valid cells, losing where the capability-saturation rule predicts a benefit.
Under a fixed simulated user no cell shows a MAS benefit: the MAS is worse in 12 of the 13 valid cells and tied in the thirteenth, and the fitted gain is negative across the observed capability range (weighted mean -28 points, 95% CI [-44, -12]).
Abstract
- Whether the collaboration gain declines with single-agent capability remains unresolved because model-by-domain effects dominate.
Whether the gain declines with capability is unresolved (slope -0.41, 95% CI [-1.51, +0.68]), because model-by-domain effects dominate.
Abstract
- The multi-agent system does not detectably reduce wrongful database writes, instead converting other failures into budget exhaustion and inaction.
The MAS does not detectably reduce wrongful database writes; it converts other failures into budget exhaustion and inaction.
Abstract
- Without a budget ceiling, the multi-agent system at best draws level with the single-agent system at 5.5 times the token spend.
On the two cells where the rule predicts a benefit, the MAS loses by 13 and 22 points, and without a budget ceiling it at best draws level at 5.5 times the SAS's token spend.
Abstract
- The strong single-agent system does not differ detectably from the stock single-agent system across three cells, while the multi-agent system loses to it on both cells below 45%.
Across the three cells, S2 does not differ detectably from the SAS, and the MAS loses to S2 on both cells below 45%.
Results.
Quoted word for word from the full text on arXiv.
The paper
arXiv, Oct 20, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.27655
arXiv:2610.27655v1 in cs.AI, also cs.MA
Cite as BibTeX
@misc{yang2026does,
title = {Does Multi-Agent Collaboration Pay When Agents Serve Users? Testing Agent Scaling in Dual-Control Settings},
author = {Yang, Ethan and Shi, Xiangcheng},
year = {2026},
eprint = {2610.27655},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arxiv.2610.27655},
url = {https://arxiv.org/abs/2610.27655}
}