Structural priors for data-efficient language learning — Paper

Abstract

Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types - notably music, probabilistic grammars, and cellular automata - yield lower language-modeling loss than random initialization. These gains coincide with smaller weight shifts during subsequent language training, suggesting that structural transfer positions models in a more favorable region of the parameter space. However, a lower loss does not translate consistently into better downstream linguistic performance, and transfer from non-language data is less efficient than additional language data. We conclude that non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization.

Authors 4

Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager, Lisa Beinborn

Specification

Official page: arxiv.org/abs/2609.11505
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh
Arxiv announce type: new
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh
arXiv id: 2609.11505
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh
Categories: cs.CL, cs.AI, cs.LG
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh
PDF: arxiv.org/pdf/2609.11505
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh
Primary category: cs.CL
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 53 min agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

53 min ago

Conflicts

None

No models linked to this paper yet.

Authors: Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · arXiv id

1 claims · 1 propertiesShow all properties

arXiv idarxiv_id1

Claim history for arXiv id
Value	Valid from → to	Status	Source	Confidence	Extractor
2609.11505	11 Sept 2026 21:02 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
Property changedPaperStructural priors for data-efficient language learning
Structural priors for data-efficient language learning: arxiv announce type changed from cross to new
Arxiv announce typecross→newarxiv
rss.arxiv.org
11 Sept 2026
New paperPaperStructural priors for data-efficient language learning
New paper: Structural priors for data-efficient language learning
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.CL	feed	T1· Official	53 min ago	1
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.LG	feed	T1· Official	53 min ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.