On two impossible game levels, the framework produced checked proofs in all six runs; two models with coding tools produced none in twelve comparison runs.On two impossible game levels, the framework produced checked proofs in all six runs; two models with coding tools produced none in twelve comparison runs.
Ruoyu Wu, Zengji Tu, Aolong Sun et al.
Before AI can be trusted to discover new science, it should be able to reproduce what science already knows.Before AI can be trusted to discover new science, it should be able to reproduce what science already knows.
Paired views of one scene needed 42 times fewer training samples than predicting the next piece of text to match alignment on the same text collection.Paired views of one scene needed 42 times fewer training samples than predicting the next piece of text to match alignment on the same text collection.
Justin Clarke, Isha Karn, Patrick Murray et al.
Large language models (LLMs) appear to represent and reason about a large range of phenomena. In particular, recent studies have examined whether LLMs can accurately…Large language models (LLMs) appear to represent and reason about a large range of phenomena. In particular, recent studies have examined whether LLMs can accurately reason about causation.
Abhinav Goel, Agostino Capponi, Alfio Gliozzo et al.
AI agents can now write software from a plain-English description, but code that runs is not the same as code that does what was asked.AI agents can now write software from a plain-English description, but code that runs is not the same as code that does what was asked. This gap matters most for smart contracts, the programs behind Web3 applications that move…
In simulated populations of 10,000 language-model agents, a measure of network connectivity explained almost none of the variation in final outcomes.In simulated populations of 10,000 language-model agents, a measure of network connectivity explained almost none of the variation in final outcomes.
Sinuo Wang, Wentao Zhao, Jiacheng Lu et al.
In finance, the answer that matters is usually the current one: prices, rates, and disclosures hold only as of a given day, so retrieving up-to-date facts is a core…In finance, the answer that matters is usually the current one: prices, rates, and disclosures hold only as of a given day, so retrieving up-to-date facts is a core capability of a financial agent.
Shiyi Cao, Ziming Mao, Dacheng Li et al.
The rapid progress of software engineering agents drives interest in applying them to open-ended problems such as GPU kernel optimization.The rapid progress of software engineering agents drives interest in applying them to open-ended problems such as GPU kernel optimization. However, we find that existing coding agents converge to local optima on real-world…
Yangshen Deng, Jiaqi Ruan, Yeqi Huang et al.
Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or…Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that…
Jane Pan, You Lyu, Abe Bohan Hou et al.
LLM agents increasingly interact with other agents on behalf of users, yet how users in the loop affect their agents' behavior–for better or worse–remains poorly…LLM agents increasingly interact with other agents on behalf of users, yet how users in the loop affect their agents' behavior–for better or worse–remains poorly understood.
Minh Pham Van, Cong Tran
A retrieval-augmented language model answers from two sources, the retrieved context and its own knowledge, and the two disagree whenever the context is incomplete or…A retrieval-augmented language model answers from two sources, the retrieved context and its own knowledge, and the two disagree whenever the context is incomplete or wrong.
Per Kristian Lehre
LLMs now prove hard theorems in Lean. Can they also design algorithms and prove their correctness and efficiency?LLMs now prove hard theorems in Lean. Can they also design algorithms and prove their correctness and efficiency? LLM-guided design of optimisation heuristics (such as AlphaEvolve) is purely empirical.
Chao Wan, Paul Jünger, Beixi Du et al.
While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces.While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces.
Tingzhu Bi, Xun Zhang, Jinglin Li et al.
Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages…Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents.
Harish Santhanalakshmi Ganesan
We present the Infinity Machine, a self-improving AI agent that we use for research on problems whose answers aprogram or data can check.We present the Infinity Machine, a self-improving AI agent that we use for research on problems whose answers aprogram or data can check. Each candidate solution is produced by a full tool-using agent that works for up to an hour…
Yinsong Yan, Ling Wang, Yuxuan Wang et al.
Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams.Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect…
Amirhossein Basareh, Rishabh Sanjay, Daniel de Mello et al.
How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of…How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems.
Matthias Busch, Marius Tacke, Roland C. Aydin et al.
Agent harnesses keep long sessions inside the context window by compacting the history when a token threshold is crossed.Agent harnesses keep long sessions inside the context window by compacting the history when a token threshold is crossed. We study an alternative in which the model manages its own context: it decides when to compact, what to keep…
Yihua Qin, Jiajie Jerry Zhao
LLM-based sales agents are often evaluated on static policy questions or task success, but deployment asks a sharper question: when a model can state a rule, will it…LLM-based sales agents are often evaluated on static policy questions or task success, but deployment asks a sharper question: when a model can state a rule, will it choose compliant, tool-grounded actions under commercial…
Jiaju Huang, Shaobin Chen, Xinglong Liang et al.
Tool-using language model agents are usually evaluated by whether they finish the user's task.Tool-using language model agents are usually evaluated by whether they finish the user's task.
Changlian Ma, Limin Wang
Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people?Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden…
Qifan Zhang, Yuhan Li, Nuo Chen et al.
Training environments remain a major bottleneck for agents in data-driven scientific discovery. Existing collections of executable and verifiable training environments…Training environments remain a major bottleneck for agents in data-driven scientific discovery. Existing collections of executable and verifiable training environments are small, and their construction remains difficult to scale.
Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin et al.
Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution.Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still…
Michael Yew Chuan Lee, Manya Sriram, N. Benjamin Erichson
AI-for-Science agents are increasingly being used for formulating scientific hypotheses, running autonomous experiments, and interpreting evidence.AI-for-Science agents are increasingly being used for formulating scientific hypotheses, running autonomous experiments, and interpreting evidence.
Yuqing Huang, Jack Ji, Jithin Pradeep et al.
Modern Large Language Models (LLMs) remain vulnerable to carefully orchestrated multi-turn interactions that progressively bypass safety mechanisms.Modern Large Language Models (LLMs) remain vulnerable to carefully orchestrated multi-turn interactions that progressively bypass safety mechanisms.
Yuchen Wu, Jerry Wei
Safety probes trained on a model's activations are a standard tool for monitoring language models in real time.Safety probes trained on a model's activations are a standard tool for monitoring language models in real time. Each time a new model is released or a new harm area needs to be monitored, the probe must be retrained, which…
Gabriel Manalu, Saad Wazir, Thao Do et al.
Driving vision-language-action (VLA) models bring an appealing form of explainability to a safety-critical domain, since a system that can state the reasons for its…Driving vision-language-action (VLA) models bring an appealing form of explainability to a safety-critical domain, since a system that can state the reasons for its maneuvers offers the transparency that deployment and public…
Shidong Cao, Jing Ma, Wenhao Chai et al.
Diffusion language models (DLMs) decode multiple tokens in parallel, so individually valid tokens can be incompatible together under a schema.Diffusion language models (DLMs) decode multiple tokens in parallel, so individually valid tokens can be incompatible together under a schema.
Ahmed M. Mahran
Replacing the judge can move a benchmark score, and models can be trained to hide capabilities. Both put the evaluator in question: what can it certify?Replacing the judge can move a benchmark score, and models can be trained to hide capabilities. Both put the evaluator in question: what can it certify?
Zhengye Bao, Shengtao Zhang, Haokai Zhang et al.
LLM memory is widely used in long-term conversation understanding. Agentic memory systems often use mutation, structural organization, learned retrieval strategies, or…LLM memory is widely used in long-term conversation understanding. Agentic memory systems often use mutation, structural organization, learned retrieval strategies, or other complex operations. Is this complexity always necessary?
Abhishek Mishra, Sagnik Chatterjee, Suyash Maniyar
Large language models have been reported to encode the harmfulness of a request separately from their decision to refuse it, and latent safety monitors such as Latent…Large language models have been reported to encode the harmfulness of a request separately from their decision to refuse it, and latent safety monitors such as Latent Guard build on this report by classifying prompts as harmful…
Zhiming Meng
The claim that speculative-decoding acceptance rate decays as a chain-of-thought grows longer has become a de facto industry narrative, motivating draft-model co-design…The claim that speculative-decoding acceptance rate decays as a chain-of-thought grows longer has become a de facto industry narrative, motivating draft-model co-design and position-aware training.
Rujun Han, Yufan Zhuang, Arijit Ray et al.
Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work.Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work.
Kaiwen Zhou, Udari Madhushani Sehwag
Recursive self-improvement, in which frontier AI models work on training the next generation, is quickly becoming a reality.Recursive self-improvement, in which frontier AI models work on training the next generation, is quickly becoming a reality. However, the safety of this process has received far less attention than it should.
Haoyue Liu, Xiaoying Tang
World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model.World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model. However, these models are judged mainly by how well they render frames, even though a visually faithful…
Rui Sun, Peizhao Mei, Xiewei Cheng et al.
Multilingual large language models are increasingly deployed in settings where visible reasoning traces support verification, education, and safety auditing, yet standard…Multilingual large language models are increasingly deployed in settings where visible reasoning traces support verification, education, and safety auditing, yet standard evaluations typically measure only final-answer agreement…
Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang et al.
While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests…While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests in text.
Ming Zhong, Stacy Li, Nicholas Carlini et al.
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific…Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML…
Zhiping Zhang, Franklin Mingzhe Li, Zihao Huang et al.
When an LLM agent acts on behalf of a user, it is expected to act within the authorization scope the user grants.When an LLM agent acts on behalf of a user, it is expected to act within the authorization scope the user grants. Existing agent evaluations largely measure task completion or harmful consequences and can therefore overlook…
Soyoung Yoon, Peiran Xu, Han Wang et al.
While LLM agents are increasingly capable of gathering and synthesizing evidence from a single homogeneous source, such as structured tables or unstructured data…While LLM agents are increasingly capable of gathering and synthesizing evidence from a single homogeneous source, such as structured tables or unstructured data, real-world analytical questions frequently span heterogeneous…