Marcus Armstrong, Navid Ayoobi, Pradham Mummaleti et al.
Answer correctness is encoded as a recoverable geometric direction in the hidden states of language models.Answer correctness is encoded as a recoverable geometric direction in the hidden states of language models.
Saurabh Kumar Singh, Yogeshwar Singh Dadwhal, Malhar Vedak
Post-training quantization to the GGUF format's mixed-precision K-quants is commonly how open-weight language models reach consumer hardware, yet its effect on…Post-training quantization to the GGUF format's mixed-precision K-quants is commonly how open-weight language models reach consumer hardware, yet its effect on fine-grained lexical competence is uncharacterized.
Adithya Bhaskar, Jeffrey Cheng, Danqi Chen
Modern chess engines are silent experts: they play at a superhuman level, but do not offer explanations for their play.Modern chess engines are silent experts: they play at a superhuman level, but do not offer explanations for their play. On the other hand, language models (LMs) can generate plausible-sounding explanations, but their weak playing…
Tao Shi, Chaoyi Xiang, Qiongkai Xu et al.
Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training…Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus.
Lin Tian, Marian-Andrei Rizoiu
Language models learn from web documents, some of them false, and false content can reach a model's answer to a factual question and the summaries and decisions that use…Language models learn from web documents, some of them false, and false content can reach a model's answer to a factual question and the summaries and decisions that use it.
Omar Farouk Zouak, Houssam Eddine Boukhalfa, Soumaya Lakehal et al.
Large language models often solve a theorem forward yet fail to disprove a closely related false one: a falsification gap that supervised fine-tuning does not close and…Large language models often solve a theorem forward yet fail to disprove a closely related false one: a falsification gap that supervised fine-tuning does not close and can actively worsen.
Yekun Chai, Qiwei Peng, Haoyi Xiong
Static evaluations credit a language model for naming the right move, but an agent must carry a plan through to a verified outcome while an opponent responds.Static evaluations credit a language model for naming the right move, but an agent must carry a plan through to a verified outcome while an opponent responds.
Olivia Macmillan-Scott, Michael Jacobs, Nils Metternich et al.
Large language models (LLMs) are increasingly used to answer questions about politically contentious issues, yet evaluations typically treat a model's stance as a…Large language models (LLMs) are increasingly used to answer questions about politically contentious issues, yet evaluations typically treat a model's stance as a relatively stable property.
Pengfei Li, Naufal Suryanto, Sicheng Zhang et al.
LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations.LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existing evaluations focus on knowledge-based assessments or end-to-end agentic tasks…
Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing et al.
Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results.Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results.
Mojtaba Nafez, James Henderson
Uniform-state diffusion models (USDMs) can revise any token at any denoising step, which lets them correct their own mistakes, a key advantage over masked diffusion.Uniform-state diffusion models (USDMs) can revise any token at any denoising step, which lets them correct their own mistakes, a key advantage over masked diffusion.
Chao Li, Shigeng Wang, Anbang Yao
Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with…Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit quantized at each stage.
Beatriz Almeida Felicio
Compression reports summarize how far a compressed language model moved from the dense one, usually by a KL divergence; a deployment that relies on the dense model's…Compression reports summarize how far a compressed language model moved from the dense one, usually by a KL divergence; a deployment that relies on the dense model's outputs needs to know how many of its decisions changed.
Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin et al.
Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over…Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain…
Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur et al.
Large language models are characterized by three key properties: capability, alignment, and faithfulness.Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third…
Zhanyu Chen, Jaap Jumelet
Claims about the grammatical competence of multilingual language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm…Claims about the grammatical competence of multilingual language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm, post-training, and language resource availability has not been…
Jungseob Lee, Dongyub Jude Lee, Sugyeong Eo et al.
Fine-tuning adapts aligned large language models (LLMs) to downstream tasks, but a few dozen harmful examples can remove their refusal of harmful requests.Fine-tuning adapts aligned large language models (LLMs) to downstream tasks, but a few dozen harmful examples can remove their refusal of harmful requests.
Krishna Chytanya Ayyagari
Modern LLM evaluation assumes that pinning a judge to a fixed model version and decoding at temperature zero yields reproducible verdicts.Modern LLM evaluation assumes that pinning a judge to a fixed model version and decoding at temperature zero yields reproducible verdicts. We show this assumption fails as a property of how LLM-as-Judge is operationalized on cloud…
Jenna Russell, Ben Glickenhaus, Katherine Thai et al.
Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026…Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to…
Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus et al.
Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT).Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned…
Haohan Yuan, Simin Chen, Xi Niu et al.
Model-attribution classifiers can often identify which language model produced a text, making model-specific writing patterns a signal of provenance.Model-attribution classifiers can often identify which language model produced a text, making model-specific writing patterns a signal of provenance.
Nusrat Jahan Lia, Shubhashis Roy Dipta
Agent benchmarks test agents in worlds that stay still. Deployed agents work in worlds that other people also change.Agent benchmarks test agents in worlds that stay still. Deployed agents work in worlds that other people also change. Someone texts the agent to send the money elsewhere or an order confirmation asks it to reply with a door code.