AIAtlasBot
The crawler behind AI Atlas. It reads public documentation, pricing pages, model cards, papers, feeds and repositories to keep the atlas current and sourced.
User-Agent: AIAtlasBot/0.1 (+https://www.ai-atlas.co/bot; contact@spboucher.ai)
Purpose
What it collects, and why
AIAtlasBot fetches publicly accessible pages that describe AI models, organizations, providers, prices, benchmarks, hardware, frameworks and datasets. Pages are archived and turned into attributed facts; each fact on AI Atlas links back to the page it came from, giving publishers credit and traffic. It does not collect personal data, does not log in, and does not fetch content behind paywalls or access controls.
Behaviour
How it behaves
- Honours robots.txt (including Crawl-delay) for AIAtlasBot and *.
- Per-domain rate limits (typically a few requests per minute; arXiv ≤ 4/min, Hugging Face ≤ 30/min, GitHub ≤ 20/min).
- Conditional requests (ETag / If-Modified-Since) and content hashing so unchanged pages cost nothing.
- Adaptive intervals: pages that rarely change are visited less often; failures back off with a circuit breaker.
- Never bypasses CAPTCHAs or bot walls. If a site blocks us, the document is marked blocked and left alone.
Opt out
Blocking or contacting us
To exclude your site, add to your robots.txt:
User-agent: AIAtlasBot
Disallow: /Changes are picked up on the next visit. For questions, corrections or removal requests write to contact@spboucher.ai. The full list of sources is public on /sources.