Pratuat Amatya, Vinay Setty, V Venktesh et al.
Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps.Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps. However, standard RL alignment steps such as RLHF do not focus on factuality of LLMs, hence they may generate…
Apu Chandraw Shill, Rabindra Nath Nandi
We investigate the robustness of widely used AI text detectors under attacks applied to both human and machine text.We investigate the robustness of widely used AI text detectors under attacks applied to both human and machine text. Prior evaluations typically attack only machine text, overlooking that human text is routinely modified through…
Alexander Metzger, Aruna Srivastava, Ruslan Mukhamedvaleev et al.
Automatic Speech Recognition (ASR) system evaluations have consistently revealed performance disparities along lines of race, gender, socioeconomic status, and language…Automatic Speech Recognition (ASR) system evaluations have consistently revealed performance disparities along lines of race, gender, socioeconomic status, and language variety.
Shangbin Feng, Yulia Tsvetkov, Jan Kautz et al.
Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained.Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained.
Shubham Agarwal, Akshat Gupta, Mike Heddes et al.
Evaluation has been central to advances in AI, and it is equally critical to autonomous discovery. Yet scaling autonomous discovery to real-world engineering challenges…Evaluation has been central to advances in AI, and it is equally critical to autonomous discovery. Yet scaling autonomous discovery to real-world engineering challenges faces a fundamental bottleneck: the cost of evaluating…
Alex Martí Guiu, Nicolas Zumarraga, Kevin O'Sullivan et al.
Emergent misalignment (EM) occurs when fine-tuning a language model to a narrow task makes it misaligned on unrelated ones.Emergent misalignment (EM) occurs when fine-tuning a language model to a narrow task makes it misaligned on unrelated ones. Because organizations now routinely fine-tune models on their own data, it matters which kinds of training…
Project Apertus, :, Yixuan Xu et al.
Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few…Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale.
Wenxuan Zhu, Shuming Liu, Bang An et al.
Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens.Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens. A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports…
Roy Eisenstadt, Harel Lidar, Tal Gonen et al.
Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens that are subsequently verified in parallel by…Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens that are subsequently verified in parallel by the target model.
Shubh Garg, Chen Shani
Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured…Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning.
Re-wrapping a machine-written abstract without changing a word reversed 20 to 27 percent of a public detector's correct decisions.Re-wrapping a machine-written abstract without changing a word reversed 20 to 27 percent of a public detector's correct decisions. In an empirical study published as a preprint, researchers analyzed ten scientific-document subsets…
Emily Öhman, Anna Koufakou
Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across…Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across resources.
Arya Bastani, Iman Modarressi
A conversational speech model has to decide, at every pause, whether to speak. Existing evaluations test boundary detection, policy adherence, or aggregate statistics…A conversational speech model has to decide, at every pause, whether to speak. Existing evaluations test boundary detection, policy adherence, or aggregate statistics, not this decision.
Yueying Li, Jingxuan Fan, Kianté Brantley et al.
Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy.Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy.
Elias Lumer, Sahil Sen, Vamse Kumar Subbiah et al.
Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts.Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts.
Zhiyun Shi
Many key-value (KV) caches are compressed before anyone knows what will be asked of them: a document cached for retrieval, a prompt prefix shared across requests, the…Many key-value (KV) caches are compressed before anyone knows what will be asked of them: a document cached for retrieval, a prompt prefix shared across requests, the memory of a long conversation.
An audit tested six frontier models across four benchmarks and found the most consistent judge agreed with ground truth only 51% of the time.An audit tested six frontier models across four benchmarks and found the most consistent judge agreed with ground truth only 51% of the time.
Nicolás Vera Zúñiga
Byte-level language models such as the Byte Latent Transformer (BLT) group bytes into patches and run their large global model once per patch.Byte-level language models such as the Byte Latent Transformer (BLT) group bytes into patches and run their large global model once per patch.
Radhika Gaonkar
Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in…Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation.
Kazuki Nakajima, Takayuki Mizuno
Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs).Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs). The reliability of this screening rests on benchmark evaluations against a fixed set of LLM versions, while…
Lucas Florin, Amelie Knecht, Ulysse Schaller et al.
Large language models (LLMs) increasingly act as agents with little human oversight, so potential mistakes they make can go unnoticed.Large language models (LLMs) increasingly act as agents with little human oversight, so potential mistakes they make can go unnoticed. Users then depend on the model to report what went wrong.
Shunyuan Zhou, Hao Chen, Tianyu Wang et al.
Following retrieved evidence does not guarantee factual correctness: misleading evidence can induce a model to replace an answer it previously gave correctly.Following retrieved evidence does not guarantee factual correctness: misleading evidence can induce a model to replace an answer it previously gave correctly.
Obada Kraishan
Reasoning models allocate extra computation at inference time and present their answers as the product of deliberate thought.Reasoning models allocate extra computation at inference time and present their answers as the product of deliberate thought. If this deliberation works the way dual-process accounts of human cognition suggest, longer thinking…
Jonghyun Han, Younghoon Song, Jongyoul Park
Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using often-inaccessible…Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using often-inaccessible original data.
Yixuan Tang, Yi Yang
On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning…On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown.
Santhosh Kumar Kasa, Siva Rajesh Kasa, Sumit Negi
Deterministic inference is essential for reliable and trustworthy machine learning. Prior studies of text generation have shown that changing factors such as batch size…Deterministic inference is essential for reliable and trustworthy machine learning. Prior studies of text generation have shown that changing factors such as batch size, batch composition, hardware, or inference engine can alter…
Rui Liu, Tong Zheng, Jindong Gu et al.
While vision-language-action (VLA) models have advanced rapidly, running them at every control step remains expensive.While vision-language-action (VLA) models have advanced rapidly, running them at every control step remains expensive. Prior work accelerates VLA inference using techniques like action chunking and visual-token pruning, typically…
Stephanie Buttigieg, Maeve Madigan, Parameswaran Kamalaruban et al.
Activation steering has gained popularity as a lightweight inference-time debiasing technique for large language models.Activation steering has gained popularity as a lightweight inference-time debiasing technique for large language models. However, prior work reports that steering vectors generalise poorly, with unintended effects on model…
Pranjal Garg
Probes are the workhorse of interpretability. If a model's hidden states predict a variable, the model is said to represent it. But a probe score has no fixed meaning.Probes are the workhorse of interpretability. If a model's hidden states predict a variable, the model is said to represent it. But a probe score has no fixed meaning.
Edan Kinderman, Elad Hoffer, Yochai Blau et al.
Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus.Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus.
G. L. John Salvin, Swapnil Hingmire
COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts.COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts.
Chufan Shi, Cheng Yang, Tiannuo Yang et al.
Unified multimodal models (UMMs) integrate understanding and generation, yet their generative behavior is rarely governed by what they understand about the task.Unified multimodal models (UMMs) integrate understanding and generation, yet their generative behavior is rarely governed by what they understand about the task.
Haotian Yang, Huikang Jiang, Yucheng Wu et al.
Activation steering provides a lightweight and flexible way to control large language model (LLM) behavior.Activation steering provides a lightweight and flexible way to control large language model (LLM) behavior. However, effective steering requires more than inducing the intended behavior: it should also limit unintended changes and…
Rishanth Rajendhran, Minjoon Choi, Jenna Russell et al.
While modern AI detectors identify who wrote the words, emerging policies on AI use increasingly hinge on a different question: who came up with the ideas?While modern AI detectors identify who wrote the words, emerging policies on AI use increasingly hinge on a different question: who came up with the ideas?
Haonan Huang
A model under pushback can correct itself, capitulate, or hold, and one flip rate counts a correction and a capitulation alike.A model under pushback can correct itself, capitulate, or hold, and one flip rate counts a correction and a capitulation alike. Using SycoLens, a modular replay protocol, we test how user pressure and evaluation settings shape…
Jakub Macina, Manu Kapur, Mrinmaya Sachan
Large language models (LLMs) trained to answer questions are natively poor at teaching. Reinforcement Learning (RL) against a simulated student is a promising approach to…Large language models (LLMs) trained to answer questions are natively poor at teaching. Reinforcement Learning (RL) against a simulated student is a promising approach to improve their pedagogy, but existing RL-trained tutors…
Lennart Carstens-Behrens, Holger Fröhlich
We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior.We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior.
Yekun Chai, Haoyi Xiong
As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything…As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the epoch count matters.
Noor Islam S. Mohammad, Md. Basim Al Zabir Shammo, Hasan Siddiki et al.
Chain-of-Thought (CoT) faithfulness detectors are widely used to audit reasoning models, yet a detector is itself a predictor whose verdicts are treated as stable…Chain-of-Thought (CoT) faithfulness detectors are widely used to audit reasoning models, yet a detector is itself a predictor whose verdicts are treated as stable properties.
Xueping Gao
Activation probes can predict safety-relevant properties of language models with high area under the receiver-operating-characteristic curve (AUROC), but deployed agent…Activation probes can predict safety-relevant properties of language models with high area under the receiver-operating-characteristic curve (AUROC), but deployed agent monitors make thresholded alarm decisions under tight…