Database queries using language models change between runs
At temperature 0, between 0.07% and 10.2% of rows changed their operator output over five executions, depending on the model.
Edited by Boris Djordjevic · Paperfoot
A study of existing systems, reported in a preprint, measured how much semantic query engines vary between runs across text and tabular workloads from SemBench, LRO-Bench, and UDA-Bench. Semantic query engines ask a language model to judge individual database rows. The authors re-executed workloads through each benchmark's harness, logged 1.2 million model calls, and reanalyzed published SemBench executions.
At temperature 0, a setting intended to prevent randomness, between 0.07% and 10.2% of rows changed operator output over five executions, depending on the model, and up to 24.0% on one task. Query plans amplified these shifts: on UDA-Bench, a few percent of extracted values changed, but about two thirds of answers did. In SemBench's published results, 21% of text results changed quality between executions, including on engines with deterministic plans. Additionally, 16 of 28 implementation recommendation tests on LRO-Bench remained unsettled by benchmark runs.
Why it matters
Across three public benchmarks, the evidence shows that small variations in language model outputs multiply through query plans to alter final answers and leave system comparisons unsettled. This demonstrates that current benchmark executions cannot reliably rank query engines, prompting the authors to derive new engine design and benchmark reporting rules.
Caveats
The evaluation was limited to three benchmarks over five executions, and the abstract does not report the specific language models or hardware used. Furthermore, the paper is a preprint that has not yet undergone peer review.
From the paper
- Result
… gpt-oss-120B’s stack returns no log-probabilities, but its reasoning trace serves the same purpose: the length of the first execution’s trace predicts which items change in the other nine with AUC 0.84 (95% CI [0.83, 0.86], 1,057 changing items).
Results, A single execution’s margin predicts later changes.
- Events by group
In a separate control, five executions of gpt-oss-120B on thirteen filter and map tasks of our own over 100 synthetic product reviews, copying a city never changed, whereas a six-way sentiment score changed on 11% of reviews and a free-text summary on 82%.
Quoted word for word from the full text on arXiv.
In this paper
- Methods
- BigQuery (evaluated)ThalamusDB (evaluated)Palimpzest (evaluated)LOTUS (evaluated)Mistral-24B (evaluated)gpt-oss-120B (evaluated)Granite-8B (evaluated)Qwen-72B (evaluated)Llama-70B (V) (evaluated)Llama-70B (M) (evaluated)GPT-4o (evaluated)Claude Sonnet 5 (evaluated)
- Datasets
- SemBench (evaluated)LRO-Bench (evaluated)UDA-Bench (evaluated)
- Links
- Code
Results
- Mistral-24B, flip rate, on SemBench, five executions, pooled over ten filter and map queries: 0.07%; gpt-oss-120B, flip rate, on SemBench, five executions, pooled over ten filter and map queries: 10.2%
Pooled over the ten SemBench filter and map queries, the flip rate over five executions ranges from 0.07% for Mistral-24B to 10.2% (95% CI [9.55%, 10.85%]) for gpt-oss-120B (Table 5).
The model sets the order of magnitude.
- gpt-oss-120B, pairwise disagreement: 5.19%; Granite-8B, pairwise disagreement: 1.38%
The pairwise disagreement, which does not depend on the number of executions, is 5.19% for gpt-oss-120B and 1.38% for Granite-8B.
The model sets the order of magnitude.
- GPT-4o, share of items changed, over five executions on filters and maps: 7.5%; Claude Sonnet 5, share of items changed, over five executions on filters and maps: 4.1%
On the same filters and maps, GPT-4o changes 7.5% of items over five executions and Claude Sonnet 5 4.1% (Table 6).
Commercial models vary too.
- gpt-oss-120B, flip rate, Boolean filters: 3.4%; gpt-oss-120B, flip rate, two-way label: 3.3%
For gpt-oss-120B it is 3.4% on Boolean filters and 3.3% on the two-way label.
The task sets the rest.
- gpt-oss-120B, flip rate, 1–5 scores: 17.5%; gpt-oss-120B, flip rate, 24-way diagnosis: 24.0%
On 1–5 scores it is 17.5%, and on the 24-way diagnosis 24.0%.
The task sets the rest.
- gpt-oss-120B, share of extracted values changed, on UDA-Bench: 7.5%; Llama-70B (M), share of extracted values changed, on UDA-Bench: 4.0%
On UDA-Bench (Table 9), gpt-oss-120B changes 7.5% of 2,208 extracted values and Llama-70B (M) 4.0%.
The model sets the order of magnitude.
What it shows
- At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task.
At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task.
Abstract
- Answer reproduction decays exponentially in the number of items the answer exposes when the answer determines every item's output.
Reproduction decays exponentially in the number of items the answer exposes: with \bar{\delta}=0.1%, a join over n=1,560 pairs reproduces with probability about 0.21.
3. A Possible-Worlds Model and Measures
- About one published cell in five changes quality across executions across engines, including when the engine's plan is deterministic.
About one published cell in five changes quality across executions, on every engine and also when the engine’s plan is deterministic.
System comparisons.
- Run-to-run changes concentrate on items that models disagree about, and identical requests within one execution disagree as often as across executions.
Changes are not uniform noise, and they are not a property of the execution: identical requests within one execution disagree as often as across executions.
Client settings have no detectable effect.
- Repetition reduces variance but cannot eliminate it, while recording operator outputs makes an answer reproducible at the cost of tens of bytes per output.
An answer is reproducible only if the operator outputs that produced it are recorded, which costs tens of bytes per output.
Recording outputs.
Quoted word for word from the full text on arXiv.
The paper
Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
arXiv, Oct 9, 2026, Open access, CC BY-NC-ND, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.13823
arXiv:2610.13823v1 in cs.DB, also cs.AI
Comments: Code and data: https://github.com/semviews/sqp-variance-study
Cite as BibTeX
@misc{hassanzadeh2026same,
title = {Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing},
author = {Hassanzadeh, Oktie and Subramanian, Dharmashankar},
year = {2026},
eprint = {2610.13823},
archivePrefix = {arXiv},
primaryClass = {cs.DB},
doi = {10.48550/arxiv.2610.13823},
url = {https://arxiv.org/abs/2610.13823}
}