How AI Atlas records facts
The dataset is the product. These are the rules every connector, extractor and page follows — and the vocabularies the API exposes.
AI Atlas is built from first-party connectors that read public documents directly: official documentation, pricing pages, model cards, release notes, papers, feeds, sitemaps and repositories. Nothing depends on a third-party data API. Every document is snapshotted and archived; every fact points back to a snapshot.
Facts are temporal claims: a property, a value, a source, a tier, a confidence, an extractor and a validity interval. When a better-or-equal source states a new value, the old claim is superseded (its interval closes) and the new one becomes current. When a worse source disagrees, the value is stored as conflicting and flagged for review — it is never averaged or silently overwritten. Missing means missing: the site shows “Unavailable” rather than a guess.
Source tiers
Primary sources first
A tier 1 source is the entity's own publisher (a lab's documentation, a provider's pricing page, an arXiv listing for a paper). Tier 2 are quality secondary sources, tier 3 community sources, tier 4 unverified. A higher tier can supersede a lower one; the reverse produces a flagged conflict.
Confidence
Confidence levels
Default confidence follows the tier (tier 1 → high, tier 2 → medium, tiers 3–4 → low). A claim becomes conflicted when a current value is contradicted by another source. Extractors can raise confidence to verified when a value is confirmed by several independent tier 1–2 sources.
Data quality
The quality score measures our knowledge, not the entity
quality.score = 100 × (0.25 completeness + 0.25 primary-source ratio + 0.20 freshness + 0.15 agreement + 0.15 source diversity). It says how well AI Atlas knows an entity — how many expected attributes are filled, how much comes from tier 1, how recently it was re-observed, how few conflicts remain and how many independent sources agree. It says nothing about whether a model is good.
History
Change events
Material properties — context length, status, license, openness, parameters, release date, deprecation and retirement dates, versions, prices — emit events when they change. Noisy metrics (downloads, likes, stars) are stored as time series and never generate events. Descriptions and other soft text follow their own source without events. Each event has a category and an importance from 0 (minor) to 3 (major), and links to the source document that triggered it.
Extraction
Deterministic before LLM
Stage 1 runs on every document: DOM selectors, JSON-LD, tables, Markdown, feeds. Only documents flagged as needing it go to the local LLM factory (MacLustr, OpenAI-compatible), through versioned schemas with full token accounting. LLM-extracted values are marked as such in provenance. Improving a parser bumps its version and reprocesses archived snapshots — never a re-crawl.
Benchmarks
Results are never compared blindly
Benchmark results are append-only and carry their configuration (harness, prompting, number of shots, judge). Leaderboards rank current rows under the benchmark's own direction; results with different configurations are shown with their config so the reader can judge comparability. We do not compute composite indices.
Estimates
Hardware fit is an estimate
The only derived figures on AI Atlas are hardware-fit estimates: memory need ≈ parameters × bytes per parameter (4-bit 0.5 × 1.15 overhead, 8-bit 1.0, fp16 2.0) plus a KV-cache allowance for the chosen context. They are labelled Estimated everywhere and never mixed with observed facts.
Crawling
Respectful by design
Connectors honour robots.txt, use per-domain rate limits and conditional requests, identify as AIAtlasBot, never bypass access controls and never collect private data. The list of sources and connector health is public on /sources.