Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web — Paper

Abstract

-cross Abstract: Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.

Authors 10

Gon\c{c}alo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simpl\'icio, Diogo Tavares, David Semedo, Daniel Gomes, Jo\~ao Magalh\~aes

Specification

Official page: arxiv.org/abs/2609.07699
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh
Arxiv announce type: replace
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh
arXiv id: 2609.07699
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh
Categories: cs.CL, cs.AI
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh
PDF: arxiv.org/pdf/2609.07699
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh
Primary category: cs.CL
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 3 h agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

3 h ago

Conflicts

None

No models linked to this paper yet.

Authors: Gon\c{c}alo Vinagre, Rui Pedro Guerra, Pedro Gomes

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Abstract

1 claims · 1 propertiesShow all properties

Abstractabstract1

Claim history for Abstract
Value	Valid from → to	Status	Source	Confidence	Extractor
-cross Abstract: Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.	11 Sept 2026 21:03 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
New paperPaperFine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
New paper: Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.AI	feed	T1· Official	3 h ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.