Woobin Park, Jaeyeon Kim, Yilun Du et al.
Diffusion and flow matching are now dominant frameworks for generative modeling in continuous domains, but they typically require multiple network evaluations at…Diffusion and flow matching are now dominant frameworks for generative modeling in continuous domains, but they typically require multiple network evaluations at inference.
Peilong Zhou, Hongtao Zhang, Yulong Huang et al.
Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with.Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known to overthink: iterating past the trained depth can turn a correct answer into…
Joshua Cheung, Zhang He
Latent world models are routinely selected, checkpointed and ablated on one-step prediction loss, on the assumption that a model which predicts the next latent better is…Latent world models are routinely selected, checkpointed and ablated on one-step prediction loss, on the assumption that a model which predicts the next latent better is a better model.
George Ma, Yuhao Zhang, Bing He et al.
Reinforcement learning for code usually rewards passing tests, which a model can do while remaining wrong on the inputs the tests do not cover.Reinforcement learning for code usually rewards passing tests, which a model can do while remaining wrong on the inputs the tests do not cover.
Behrooz Tahmasebi, Melanie Weber
Modern machine learning models increasingly operate on variable-size structured data, such as sets and graphs.Modern machine learning models increasingly operate on variable-size structured data, such as sets and graphs. A central requirement is size generalization: the ability of a model trained on inputs of one size to generalize to…
Leif Sieben, Yoel Zimmermann
Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks.Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks.
Andis Draguns, Sumeet Ramesh Motwani, Raymond Douglas et al.
Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be…Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited.
Minchan Kwon, Seunghee Koh, Sunghyun Baek et al.
LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters.LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL).
Guoxia Wang, Qianyue He, Siming Wu et al.
Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with…Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant…
Guanghan Wang, Bing He, Zhan Shi et al.
Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to train than…Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to train than autoregressive language models (ARMs): a widely cited study reports…
Yingzi Ma, Yulong Cao, Nanxi Li et al.
Scientific discovery with language models requires finding the best valid solution within a finite computational budget.Scientific discovery with language models requires finding the best valid solution within a finite computational budget. We study how feedback from this search can improve the procedure used to conduct it.
Roey Magen, Nirmit Joshi, Gal Vardi
The success of autoregressive language models trained with Chain-of-Thought~(CoT) supervision raises a fundamental question: How simple can a next-token predictor be…The success of autoregressive language models trained with Chain-of-Thought~(CoT) supervision raises a fundamental question: How simple can a next-token predictor be while still enabling the efficient simulation of general…
Xiaoyang Yuan, Ji Zhao, Xingyu Hu et al.
Reliable agentic tool use requires training data that pair diverse multi-step workflows with executable evidence that the requested outcomes were achieved.Reliable agentic tool use requires training data that pair diverse multi-step workflows with executable evidence that the requested outcomes were achieved.
Jaewon Choi, Jihye Kang, Namhyuk Ahn
A release decision about a fine-tuned model is an operational commitment about the next rerun of its training recipe.A release decision about a fine-tuned model is an operational commitment about the next rerun of its training recipe. Yet standard benchmark reporting gives only the mean accuracy of past seeds.
Yifeng Guo, Yanran Li, Xiang Chen
Sparse autoencoder (SAE) features are typically discovered from how their activations co-vary with behaviour, yet they are often used through interventions such as…Sparse autoencoder (SAE) features are typically discovered from how their activations co-vary with behaviour, yet they are often used through interventions such as steering or ablation.
Muhammad Usama, Yeoreum Jung
Existing depth-compression methods for large language models (LLMs) score layers individually or group only adjacent or contiguous layers, implicitly assuming that…Existing depth-compression methods for large language models (LLMs) score layers individually or group only adjacent or contiguous layers, implicitly assuming that functional redundancy is local.
Abdulrahman Alabdulkareem, Irene Tenison, Yang Zhang et al.
LLM multi-agent systems split work across agents that communicate through text. An agent summarizes its findings, another agent receives the text.LLM multi-agent systems split work across agents that communicate through text. An agent summarizes its findings, another agent receives the text.
Amin Najafgholizadeh, Mohsen Afsharchi
PATTERN, a synthetic node-classification benchmark still used to rank graph networks, plants a random subgraph in a random host graph and asks which nodes belong to it.PATTERN, a synthetic node-classification benchmark still used to rank graph networks, plants a random subgraph in a random host graph and asks which nodes belong to it.
Sheng Zhao, Yuhao Zhu
An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training.An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training.
Henry Bourne, Robert F. Allison
What can an adversary recover from a model trained with DP-SGD? Its (ε,δ) guarantee does not answer this question directly.What can an adversary recover from a model trained with DP-SGD? Its (ε,δ) guarantee does not answer this question directly.
Irene Cannistraci, Julia E. Vogt
In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without…In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training.
Zhicheng Cai, Xinyuan Guo, Hanlin Wu et al.
Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs.Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that…
Tomo Oga, Devesh Shah, Cailum M. K. Stienstra et al.
Molecular machine learning evaluates out-of-distribution performance with structure disjoint splits, a construction rule fixed once at the scale its library happened to…Molecular machine learning evaluates out-of-distribution performance with structure disjoint splits, a construction rule fixed once at the scale its library happened to have: threshold a pairwise distance, then assign whole…
Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh et al.
Efficient autoregressive generation is increasingly limited by the memory footprint of the key-value (KV) cache, which grows linearly with sequence length, batch size…Efficient autoregressive generation is increasingly limited by the memory footprint of the key-value (KV) cache, which grows linearly with sequence length, batch size, and model depth.
Accounting for realistic delays left published adaptation methods improving by at most half a percent over frozen models across four of five benchmarks.Accounting for realistic delays left published adaptation methods improving by at most half a percent over frozen models across four of five benchmarks.
Nikhil Verma, Siddharthan Dileep, Anoop Singh et al.
Diffusion models generalize early in training and later reproduce individual training samples. Standard tests detect memorization only once one-shot generation produces…Diffusion models generalize early in training and later reproduce individual training samples. Standard tests detect memorization only once one-shot generation produces near-copies, leaving a released model unaudited until its…
Matěj Kripner, Milan Straka
We introduce NanoProof, to our knowledge the first factorized execution-guided theorem prover in Lean 4 whose training data, extraction tooling, training pipeline, and…We introduce NanoProof, to our knowledge the first factorized execution-guided theorem prover in Lean 4 whose training data, extraction tooling, training pipeline, and weights are all released, making it end-to-end reproducible…
Ali Alavi, Donald S. Williamson
Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws.Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws.
Yixin Tan, Jiayang Liu, Lu Sun et al.
Mixture-of-Experts (MoE) language models produce routing information during inference that may be logged or exposed for monitoring, debugging, load analysis, and safety…Mixture-of-Experts (MoE) language models produce routing information during inference that may be logged or exposed for monitoring, debugging, load analysis, and safety auditing.
Amit Nautiyal
When reinforcement learning teaches a language model a new behavior, can we find the training rollouts that taught it?When reinforcement learning teaches a language model a new behavior, can we find the training rollouts that taught it? And when an attribution method says it can, how do we know the answer is real?
Fanqing Meng, Lingxiao Du, Haocheng Lu et al.
Recursive self-improvement requires carrying accepted changes into later improvement cycles, while studying agent-proposed changes also requires substantial research…Recursive self-improvement requires carrying accepted changes into later improvement cycles, while studying agent-proposed changes also requires substantial research infrastructure.
Nicholas Tan Jerome, Fangnian Wang
A natural strategy for inverse problems with scarce labelled data is to transfer relational structure learned from abundant forward-simulation data.A natural strategy for inverse problems with scarce labelled data is to transfer relational structure learned from abundant forward-simulation data.
Yujia Zheng, David Klindt, Randall Balestriero et al.
Methods that recover individual latent variables of the world, from nonlinear ICA to dictionary learning and causal representation learning, anchor the latents to…Methods that recover individual latent variables of the world, from nonlinear ICA to dictionary learning and causal representation learning, anchor the latents to observations through reconstruction, auxiliary supervision, or…
Shikhar Srivastava, Christopher Kanan
As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly…As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed.
Mojtaba Nafez, Aref Mousavi, Mohammad Ebrahim Mahdavi et al.
Many organizations fine-tune publicly available pretrained Automatic Speech Recognition (ASR) models and deploy them in black-box settings, assuming limited access…Many organizations fine-tune publicly available pretrained Automatic Speech Recognition (ASR) models and deploy them in black-box settings, assuming limited access provides protection.
Guilherme Afonso Galindo Padilha, Paulo Salgado Gomes de Mattos Neto, Rafael Menelau Oliveira e Cruz
While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current research.While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current research.
Domenic Rosati, Alessa Carbo, Ali Dadsetan et al.
Does removing harmful information make open-weight models resistant to fine-tuning attacks? We show that mutual information at release alone cannot universally certify…Does removing harmful information make open-weight models resistant to fine-tuning attacks? We show that mutual information at release alone cannot universally certify slow recovery.
Ziyun Chen, Jerry Li, Kevin Tian et al.
The Fr'echet Inception Distance (FID) is widely used to evaluate generative models, but its empirical plug-in estimator suffers from finite-sample bias [BSAG18, CF20].The Fr'echet Inception Distance (FID) is widely used to evaluate generative models, but its empirical plug-in estimator suffers from finite-sample bias [BSAG18, CF20].
Robin Young
Machine learning classifiers for remote sensing imagery are typically evaluated as though every pixel were an independent sample.Machine learning classifiers for remote sensing imagery are typically evaluated as though every pixel were an independent sample. Spatial autocorrelation violates this assumption, since neighboring pixels carry redundant…
Yiming Xu, Hongyue Yu, Beihua Yang et al.
As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment.As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment.