Youqing Fang, Zhouqi Hua, Yinhao Tang et al.
Mathematical supervision for language models is typically flattened into isolated problem-solution pairs, discarding the curricular organization and evidential structure…Mathematical supervision for language models is typically flattened into isolated problem-solution pairs, discarding the curricular organization and evidential structure of the original sources.
Peng Xie, Amr Alanwar
To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an…To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its…
Alexander Stern, Tim Rocktäschel, Acyr Locatelli et al.
We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments.We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments.
Esther Xin
Monitoring a model's chain of thought (CoT) is useful for safety only if the final answer depends on the written steps.Monitoring a model's chain of thought (CoT) is useful for safety only if the final answer depends on the written steps. Prior measurements use natural tasks where true reasoning is unknown.
EasyProver resolved 597 of 600 code-verification checks and all 189 tasks in a second test set, by proof or refutation, without additional model training.EasyProver resolved 597 of 600 code-verification checks and all 189 tasks in a second test set, by proof or refutation, without additional model training.
Nvidia, :, Aastha Jhunjhunwala et al.
We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context…We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads.
Ilija Subasic, Brett Levert, Andrew Rabinovich et al.
LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable…LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument.
Mingqian Feng, Xiaodong Liu, Weiwei Yang et al.
Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the…Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting…
Rong Huang, Yi Xiong, Yuan-Yuan Cheng et al.
Parameter-efficient fine-tuning (PEFT) methods adapt large models by introducing a small number of trainable parameters.Parameter-efficient fine-tuning (PEFT) methods adapt large models by introducing a small number of trainable parameters. However, we find that these parameters still contain redundancy and can be further compressed while…
Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan et al.
Constrained decoding guarantees that Large Language Models (LLMs) produce outputs conforming to predefined formats, but is often assumed to inherently degrade reasoning.Constrained decoding guarantees that Large Language Models (LLMs) produce outputs conforming to predefined formats, but is often assumed to inherently degrade reasoning.
Yujia Zheng, Tianhao Li, Shangzhi Xu et al.
Agent Skills allow LLM agents to combine reusable capabilities, but checking each Skill in isolation cannot rule out policy violations when they are composed.Agent Skills allow LLM agents to combine reusable capabilities, but checking each Skill in isolation cannot rule out policy violations when they are composed.
Shan Yang
Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one.Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one.
A different pretraining seed lowered the correlation of per-prompt non-refusal rates to 0.811 on average, compared with 0.984 for copies from the same run.A different pretraining seed lowered the correlation of per-prompt non-refusal rates to 0.811 on average, compared with 0.984 for copies from the same run.
Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva et al.
LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood.LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood.
Anish Laddha, Nitesh Pradhan, Gaurav Srivastava
LLM agents take instructions from the operator, repository configuration, user, tool results and saved memory, which often conflict.LLM agents take instructions from the operator, repository configuration, user, tool results and saved memory, which often conflict. Instruction-hierarchy benchmarks ask whether the higher-ranked source wins, but a source sets…
Yeonju Kim, Bella Godiva, Se Jin Park et al.
Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation.Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes?
Michael Dinzinger, Kanishka Ghosh Dastidar, Michael Granitzer
System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text.System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers.
Mahdi Naser Moghadasi
We report a systematic dissociation in modern text embedders: sentiment is almost perfectly decodable from their embeddings, yet almost entirely absent from their metric.We report a systematic dissociation in modern text embedders: sentiment is almost perfectly decodable from their embeddings, yet almost entirely absent from their metric.
Kirill Afendulev, Alexey Dontsov, Alla Chepurova et al.
Language models can answer a question from a supplied document or from their parametric knowledge. When later asked where the answer came from, what determines their…Language models can answer a question from a supplied document or from their parametric knowledge. When later asked where the answer came from, what determines their source report?
Shujin Wu, Heng Ji
Reliable confidence estimation is essential for trustworthy deployment of large language models (LLMs), but dedicated confidence training stage requires additional data…Reliable confidence estimation is essential for trustworthy deployment of large language models (LLMs), but dedicated confidence training stage requires additional data and compute.
Ethan Ludwick, Yu-Chung Hsiao
LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to…LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion.
Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu et al.
Multilingual language models can give different identity judgments when the same people's names are written in different scripts.Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with attested Devanagari and Latin forms…
Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro
LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based…LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks.
Gabriel Lars Sabadin, Nicola Harding, Sarah Kingston
Content warning: this paper quotes slurs and hateful statements from published test suites; the racial slur among them is masked.Content warning: this paper quotes slurs and hateful statements from published test suites; the racial slur among them is masked. Automated systems decide at scale which messages are harmful, and they decide unevenly.
Zhaolu Kang, Yidi Wang, Yile Li et al.
Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered.Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered.
Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez et al.
Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses.Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt.
Fei Ding
Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project.Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project. Passing those tests can require information the description never states: a constant's name, a module path…
Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF…Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF into tokens - and that layer is where an agent deployed in…
Three retrieval techniques predicted a 47% combined improvement but delivered only 17% when used together across eight benchmarks.Three retrieval techniques predicted a 47% combined improvement but delivered only 17% when used together across eight benchmarks. The authors introduced LASAGNA, an open-source evaluation framework for retrieval-augmented…
Auss Abbood, Fernando Martínez-Plumed, José Hernández-Orallo et al.
Temporal reasoning is widely assumed to decompose into two modes: temporal arithmetic (date and time calculations), and temporal semantics (understanding temporal and…Temporal reasoning is widely assumed to decompose into two modes: temporal arithmetic (date and time calculations), and temporal semantics (understanding temporal and causal relations).
Xuan Yang, Chunxiao Liu, Hongsheng Xin et al.
Large language model (LLM) agents increasingly use tools to complete tasks in stateful environments.Large language model (LLM) agents increasingly use tools to complete tasks in stateful environments.
Yun Li, Ehsan Javanmardi, Simon Thompson et al.
Preference optimization can improve every ranking metric while building a hidden, shared representation shift that the relative loss barely sees, one that can eventually…Preference optimization can improve every ranking metric while building a hidden, shared representation shift that the relative loss barely sees, one that can eventually destroy generation.
Ziqing Qiao, Xiaoyue Xu, Sikui Zhang et al.
Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains…Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear.
Jiaxuan Guo, Xingyu Ren, Kejia Zhang et al.
Vision-language models are now routinely tested on theory of mind: whether they can work out what another person sees, knows, and means.Vision-language models are now routinely tested on theory of mind: whether they can work out what another person sees, knows, and means. A common design asks the model to report that perspective and then to act for the person, and…
Hong Ting Tsang, Boyi Zhang, Haoyu Huang et al.
Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report…Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-and-image or report a single end-to-end score, so they cannot locate whether…
Haoning Yang, Lili Pan
Continuous diffusion language models generate a whole sequence in parallel, never drawing a single word at any intermediate step; only the final step turns embeddings…Continuous diffusion language models generate a whole sequence in parallel, never drawing a single word at any intermediate step; only the final step turns embeddings into text.
Jungseob Lee, Hogun Seong, Euijun Choi et al.
Speculative decoding accelerates large language models by verifying drafted tokens in parallel. New draft-training objectives are usually credited with the acceptance…Speculative decoding accelerates large language models by verifying drafted tokens in parallel. New draft-training objectives are usually credited with the acceptance they add over cross-entropy on externally written responses…
Shenglai Zeng, Xiao Yang, Tiffany Ma et al.
As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and…As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all.
Arnav Chavan, Chanhyuk Lee, Udbhav Bamba et al.
Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques…Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less…
Tapan Parikh
A model behind an API can be identified from its one-word answers to everyday questions such as "Name a sound" or "Name a poet".A model behind an API can be identified from its one-word answers to everyday questions such as "Name a sound" or "Name a poet". A model's profile records how often it gives each answer to each question.