Distance generalization in transformers: why bother with positional encoding? — Paper

Abstract

Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.

Authors 2

Daniel Henrik Nevermann, Claudius Gros

Specification

Official page: arxiv.org/abs/2609.11913
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh
Arxiv announce type: new
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh
arXiv id: 2609.11913
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh
Categories: cs.CL
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh
PDF: arxiv.org/pdf/2609.11913
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh
Primary category: cs.CL
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 7 h agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

7 h ago

Conflicts

None

No models linked to this paper yet.

Authors: Daniel Henrik Nevermann, Claudius Gros

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Published

1 claims · 1 propertiesShow all properties

Publishedpublished_at1

Claim history for Published
Value	Valid from → to	Status	Source	Confidence	Extractor
11 Sept 2026	11 Sept 2026 21:03 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
New paperPaperDistance generalization in transformers: why bother with positional encoding?
New paper: Distance generalization in transformers: why bother with positional encoding?
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.CL	feed	T1· Official	5 h ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.