PaperfootJoin free
arXiv

AI agents' tool calls can change before they run

IntAct, a repair system deployed in a commercial product, recovered 79.2% of failures involving altered tool calls.

Paper by Boyang Yang, Zhenhao Li, Ziyao Yang, 3 more authorsHaoye Tian
Preprint: system

Edited by Boris Djordjevic · Paperfoot

Paper

The researchers built a protocol to locate changes to AI agents' tool calls and IntAct, a repair system that recovered failures involving altered calls. These agents send commands through software layers that can change them before execution. The protocol checked what each layer received without running the command. IntAct delivered the call in a form the offending layer could not alter, or refused it.

The researchers examined 47,828 shell calls—commands sent to a computer's command-line interface—in production sessions. Claude Code's Bash tool changed 12.0% of calls carrying code, long text or escape sequences, which represent special characters. Among calls whose backslashes changed, 80.7% ran the wrong action without reporting an error. All 10 tested harnesses—the software that handles agents' tool calls—changed a call.

Why it matters

Judgments based on agents' recorded action sequences attributed 95.1% of production failures to the language model, although the execution path caused more than half. The authors called for testing each software layer so that correct calls execute as intended or are refused.

Caveats

The production-session analysis concerned shell calls. The paper is a preprint that has not yet been peer-reviewed.

In this paper

intent-execution correspondence in LLM agents

Methods
IntAct (this paper)Claude Code's Bash tool (evaluated)
Datasets
IEC-Bench (this paper)

Results

Claude Code's Bash tool, percentage of calls changed, calls that carry code, escape sequences, or long text in 47,828 shell calls within production sessions: 12.0%

In 47,828 shell calls within production sessions, Claude Code's Bash tool changes 12.0% of the calls that carry code, escape sequences, or long text.

Abstract

What it shows

IntAct recovers 79.2% of failures involving a changed call when deployed in a commercial product.

IntAct, deployed in a commercial product, recovers 79.2% of the failures with a changed call.

Abstract

All 10 measured harnesses alter tool calls during execution.

All 10 measured harnesses change a call.

Abstract

Trajectory-based judgment attributes 95.1% of production failures to the LLM, despite execution paths causing more than half of them.

Trajectory-based judgment attributes 95.1% of the production failures to the LLM, although the path caused more than half of them.

Abstract

A hop changing a call conceals subsequent changes, causing 55.1% of failures on one path to manifest only after repairing the initial hop.

A hop that changes a call also hides the changes after it, so 55.1% of the failures on one path appear only after its first hop is repaired.

Abstract

For 80.7% of calls with altered backslashes, incorrect actions execute without reporting an error.

For 80.7% of the calls whose backslashes are changed, the wrong action runs without any reported error.

Abstract

Quoted word for word from the abstract on arXiv.

The paper

Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agents

Boyang Yang, Zhenhao Li, Ziyao Yang, 3 more authorsHaoye Tian

arXiv, Oct 3, 2026, Open access, Preprint, not peer-reviewed

doi.org/10.48550/arxiv.2610.04375

arXiv:2610.04375v1 in cs.AI, also cs.SE

AbstractPDFHTML

Cite as BibTeX
@misc{yang2026tool,
  title = {Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agents},
  author = {Yang, Boyang and Li, Zhenhao and Yang, Ziyao and Jia, Kanghui and Yin, Xin and Liu, Mingmou and Tian, Haoye},
  year = {2026},
  eprint = {2610.04375},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  doi = {10.48550/arxiv.2610.04375},
  url = {https://arxiv.org/abs/2610.04375}
}

Related