Detectors spotted machine text by its line breaks
Re-wrapping a machine-written abstract without changing a word reversed 20 to 27 percent of a public detector's correct decisions.
Edited by Boris Djordjevic · Paperfoot
In an empirical study published as a preprint, researchers analyzed ten scientific-document subsets from six popular benchmarks to see how automated tools detect machine-generated writing. The team found that near-perfect detector scores often stem from formatting quirks created during document processing rather than actual text content. Human-authored papers extracted from digital files retained their original line breaks, whereas machine-generated samples appeared as continuous blocks of text. A basic statistical model using 27 superficial layout features reached macro-F1 accuracy scores of 0.833 to 0.997, beating every evaluated detector on three arXiv subsets. Re-wrapping a machine-written abstract without altering any words reversed 20 to 27 percent of a public detector's correct decisions. Standardizing formatting before training reduced the performance gap on unseen data from 0.32 to 0.22, and raised scores on clean benchmarks from 0.33 to 0.65.
Why it matters
The findings show that benchmark evaluations can mistake shallow file-formatting quirks for genuine detection of computer-generated prose. Removing these artifacts helps researchers train models that rely on actual text characteristics rather than layout accidents.
Caveats
The work is a preprint that has not yet been peer-reviewed. The study focused on scientific papers and ten specific subsets, leaving open how far other text detectors and domain benchmarks rely on similar formatting artifacts.
From the paper
- Limitation
We analyzed only ten subsets of six resources, all English and all of scientific domain.
Quoted word for word from the full text on arXiv.
In this paper
machine-generated text detection
- Methods
- logistic regression (this paper)format counterfactuals (this paper)Artifact Reliance Score (this paper)DeBERTa-v3-base (evaluated)Binoculars (evaluated)Fast-DetectGPT (evaluated)chatgpt-detector-roberta (evaluated)roberta-base-openai-detector (evaluated)RADAR (evaluated)roberta-base (evaluated)
- Datasets
- M4 (evaluated)PeerRead (evaluated)SemEval-2024 Task 8 (evaluated)COLING-2025 GenAI Task 1 (evaluated)IDMGSP (evaluated)GPABench2 (evaluated)CHEAT (evaluated)
Results
- logistic regression, macro-F1, 27 superficial features: 0.833-0.997; logistic regression, F1-scores, on the three arXiv subsets: 0.951-0.997
A logistic regression using 27 superficial features reached 0.833-0.997 macro-F1, and on the three arXiv subsets it outperformed every detector we evaluate with F1-scores 0.951-0.997.
Abstract
- format counterfactuals, fraction of correct decisions reversed: 20-27%
To distinguish benchmarks that contain such a signal from detectors that rely on one, we introduce format counterfactuals: just re-wrapping a machine-written abstract without changing a single word reverses 20-27% of a public detector's correct decisions.
Abstract
- format counterfactuals, generalization gap, normalizing formatting before training: 0.22
Normalizing formatting before training reduces the generalization gap from 0.32 to 0.22.
Abstract
- format counterfactuals, out-of-domain macro-F1, on format-clean benchmarks, for detectors trained on artifact-containing subsets: 0.65
On format-clean benchmarks, it also raises out-of-domain macro-F1 from 0.33 to 0.65 for detectors trained on artifact-containing subsets.
Abstract
- RADAR, ARS net, on the arXiv subsets: -0.004
The three detectors rank as following: the HC3 detector relies most on formatting, OpenAI’s GPT-2 detector less, and the adversarially trained RADAR does not rely at all ( -0.004 on the arXiv subsets).
7.2 Artifact Reliance
- chatgpt-detector-roberta, unconditioned margin, on IDMGSP, on IDMGSP, without conditioning: -0.171
Instead, chatgpt-detector-roberta reverses sign on IDMGSP, significantly and without conditioning ( -0.171, CI [-0.206,-0.138] ).
Is this a selection artifact?
What it shows
- A logistic regression using 27 content-free surface features matches or exceeds shared-task winners on scientific-document subsets, reaching 0.833-0.997 macro-F1.
A logistic regression using 27 superficial features reached 0.833-0.997 macro-F1, and on the three arXiv subsets it outperformed every detector we evaluate with F1-scores 0.951-0.997.
Abstract
- Re-wrapping a machine-written abstract without altering wording reverses 20-27% of a public detector's correct decisions.
To distinguish benchmarks that contain such a signal from detectors that rely on one, we introduce format counterfactuals: just re-wrapping a machine-written abstract without changing a single word reverses 20-27% of a public detector's correct decisions.
Abstract
- Whether formatting artifacts are exploitable depends on the detector's tokenizer, with SentencePiece models unable to represent whitespace cues.
DeBERTa-v3-base uses SentencePiece, which collapses whitespace runs and discards newlines: raw and normalized documents yield identical token sequences, so the model cannot represent the cue that dominates these benchmarks.
Exploitability depends on the tokenizer.
Quoted word for word from the full text on arXiv.
The paper
Measuring Artifact Reliance in Machine-Generated Text Detectors
arXiv, Oct 9, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.13583
arXiv:2610.13583v1 in cs.CL
Comments: Accepted to DocInsights workshop at EMNLP 2026
Cite as BibTeX
@misc{voznyuk2026measuring,
title = {Measuring Artifact Reliance in Machine-Generated Text Detectors},
author = {Voznyuk, Anastasia and Grabovoy, Andrey},
year = {2026},
eprint = {2610.13583},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
doi = {10.48550/arxiv.2610.13583},
url = {https://arxiv.org/abs/2610.13583}
}