Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models — Paper

Abstract

When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each. Those counts make three questions answerable directly. The answers form a pincer, closing from two sides. At the duplication levels ordinary text actually has, five models from 1B to 13B parameters carry at most a faint trace of their own exposure. We measure that trace with a design that reads the same sentence through two models, which cancels fluency and quality by construction, and it comes to a rank correlation near -0.08, where -1 would be a perfect relation and 0 none. Where the trace does become strong, above roughly a thousand copies, the two corpora agree on which sentences those are, because they are the famous ones, so exposure can no longer be told apart from fame. Two further measurements show how apparent membership signal gets manufactured. A common way to build a non-member is to change one word of a member. The model does prefer the original, but the gap is the same whether the original appeared once or a hundred times, so what the model is rewarding is the author's word choice, not memory. Above a thousand copies the gap grows with model size on the twelve sentences we can test there, at the same boundary where the pincer closes. And swapping the controls for sentences that differ from the members in register moves a detector from 0.83 to 0.94 AUC, on a scale where 0.5 is a coin flip and 1.0 is perfect separation. We release the sentence banks, counts, and code.

Authors 1

Arman Nik Khah

Specification

Official page: arxiv.org/abs/2609.10830
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh
Arxiv announce type: new
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh
arXiv id: 2609.10830
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh
Categories: cs.CL, cs.CR, cs.LG
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh
PDF: arxiv.org/pdf/2609.10830
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh
Primary category: cs.CL
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 1 h agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

1 h ago

Conflicts

None

No models linked to this paper yet.

Authors: Arman Nik Khah

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · PDF

1 claims · 1 propertiesShow all properties

PDFpdf_url1

Claim history for PDF
Value	Valid from → to	Status	Source	Confidence	Extractor
https://arxiv.org/pdf/2609.10830	11 Sept 2026 21:02 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
Property changedPaperDetectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models: arxiv announce type changed from cross to new
Arxiv announce typecross→newarxiv
rss.arxiv.org
11 Sept 2026
New paperPaperDetectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
New paper: Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.CL	feed	T1· Official	1 h ago	1
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.LG	feed	T1· Official	1 h ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.