Scores of human input in AI writing can be easily manipulated
In tests across news and scientific abstracts, the same articles averaged 85% human or 15% human depending only on which prompt was on record.
Edited by Boris Djordjevic · Paperfoot
A preprint study evaluated methods that measure human contribution in writing by how predictive a prompt is of the final text, testing news articles and scientific abstracts across models from 3.8 billion to 753 billion parameters. The researchers showed that these measures can be easily manipulated. They introduced specificity padding, asking a model to copy random strings from a prompt. In an audit holding each text fixed while showing or hiding the strings, scores swung by a third to half their scale. Scores rose for every tested document, and the same articles averaged 85% human or 15% human depending only on which prompt was recorded.
An exact mathematical identity explained that rare strings made copyable by context collect credit regardless of meaning. The authors also proved that when prompts could be machine-written, transcript-only methods must in the worst case miss by half the contribution at stake.
Why it matters
The empirical tests and mathematical proof show that likelihood-based scores alone cannot reliably establish human contribution without evidence of what a person added and how the text was produced. This shows that auditing systems relying solely on writing transcripts face fundamental limits.
Caveats
The abstract tests news articles and scientific abstracts but does not report results on other writing genres or real-world user workflows. The work is also a preprint that has not yet been peer-reviewed.
From the paper
- Limitation
We study English news with seven models and scientific abstracts with three, using one clean and one attacked generation per item.
Quoted word for word from the full text on arXiv.
In this paper
measuring human contribution in AI-assisted writing
- Methods
- SPECIFICITY PADDING (this paper)context-swap audit (this paper)credit ledger (this paper)Llama-3.1-8BDeepSeek-V4-FlashQwen3-8BGemma-3-4B
- Datasets
- news benchmark released by (evaluated)arXiv abstracts (evaluated)
Results
- SPECIFICITY PADDING, score, 300-item run: 0.75
The 300-item run agrees (0.75 versus 0.55).
4.1 A few copied IDs outrank the article’s key ideas
- SPECIFICITY PADDING, score, eight IDs appended to title request: 13–27 points above separately generated clean articles
Models asked to use eight IDs reproduce 65–99% of them verbatim, and their articles score 13–27 points above separately generated clean articles, with gains in 2,040 of 2,100 pairs.
4.3 Whoever holds the prompt holds the score
- DeepSeek-V4-Flash, score, full prompt versus visible request: 0.85
For DeepSeek-V4-Flash, the same articles average 0.85 with the full prompt and 0.15 with the visible request.
4.3 Whoever holds the prompt holds the score
- SPECIFICITY PADDING, score, eight IDs on 30 items reproducing ladder: 20 points over the clean title request
On the 30 items where we reproduce the ladder, the eight IDs are worth 20 points over the clean title request, more than the entire step from a title to a summary (17 points).
4.1 A few copied IDs outrank the article’s key ideas
- context-swap audit, fixed-output score gain, showing matching IDs on fixed output: 32–51 points
Showing the matching IDs raises the fixed-output score by 32–51 points on average, and it raises the score of every one of the 2,100 model–document pairs.
4.2 One text, many authors
- context-swap audit, score gain, matched IDs versus wrong IDs in fixed-output audit: 16–29 points
Matched IDs still beat wrong ones by 16–29 points, again in all 2,100 pairs.
Exact values carry the credit.
- Llama-3.1-8B, estimate of minimal human contribution, appending eight IDs: 0.68
Appending eight IDs raises Llama-3.1-8B’s estimate of minimal human contribution from 0.36 to 0.68, and raises it for all 300 articles (Appendix Figure 6).
4.4 Without any prompt, rarity itself earns credit
- context-swap audit, fixed-output score gain, on arXiv abstracts, showing matching IDs on fixed output: 43–47 points; context-swap audit, score gain, on arXiv abstracts, matched IDs versus wrong IDs on fixed output: 25–28 points
On 200 arXiv abstracts, showing matching IDs raises fixed-output scores by 43–47 points and beats wrong IDs by 25–28 points, in all 600 pairs.
4.5 Findings extend to other datasets
What it shows
- A prompt-conditioned likelihood score computes a surprisal-weighted average of local scores across spans of text, so rare copyable strings collect disproportionate credit regardless of meaning.
The score is a surprisal-weighted average over the text, so rare strings that the context makes copyable collect the credit, whatever they mean.
Abstract
- Showing matching random IDs in the context raises fixed-output scores by 32 to 51 points across all tested model-document pairs.
Showing the matching IDs raises the fixed-output score by 32–51 points on average, and it raises the score of every one of the 2,100 model–document pairs.
4.2 One text, many authors
- If a prompt could have been machine-written, every transcript-only estimator must in the worst case miss true contribution by at least half the contribution at stake.
In particular, if o is compatible with a history in which the person authored the recorded input (contribution c_{1} ) and with a laundered history in which that input was machine-generated (contribution 0 ), then every transcript-only estimator errs by at least c_{1}/2 on one of them.
Proposition 2 (Identification boundary).
- The output-only estimate of minimal human contribution is strictly increasing in mean token surprisal and approaches 1 as mean surprisal tends to infinity.
For fixed 0<\tau<1, \widehat{\phi}(y) is strictly increasing in the mean surprisal \bar{\mathcal{I}}(y) and tends to 1 as \bar{\mathcal{I}}(y)\to\infty.
Corollary 2 (Output-only estimate).
- Likelihood-based scores alone cannot reliably establish human contribution without evidence of what the person added and how the text was produced.
Likelihood-based scores alone cannot reliably establish human contribution without evidence of what the person added and how the text was produced.
Abstract
Quoted word for word from the full text on arXiv.
The paper
Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing?
arXiv, Oct 14, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.19767
arXiv:2610.19767v1 in cs.CL
Cite as BibTeX
@misc{amin2026likelihood,
title = {Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing?},
author = {Amin, Hasan and Lu, Zhuoran and Yin, Ming},
year = {2026},
eprint = {2610.19767},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
doi = {10.48550/arxiv.2610.19767},
url = {https://arxiv.org/abs/2610.19767}
}
