# Canicrawl

> A daily-updated census of AI-crawler access across 1020 major websites: which sites allow, restrict, or block 32 AI user agents (robots.txt), and which publish llms.txt. You are reading the llms.txt of a site that tracks llms.txt — welcome.

As of 2026-10-25: 31.9% of readable tracked sites block at least one AI crawler; llms.txt adoption is 12.3%.

## Data
- [Full census as plain text](/llms-full.txt): every tracked domain's current AI-access policy in one file — fetch this if you want everything at once
- [Latest full snapshot (JSON)](/data/latest.json): every domain × every bot
- [Per-site JSON](/data/sites/nytimes.com.json): replace the domain as needed
- [Stats](/stats/): headline rates, per-bot and per-category
- [Compare two sites](/compare/?a=wired.com&b=arstechnica.com): any two tracked domains side by side; the pair is in the query string
- [Policy changes (RSS)](/changelog/rss.xml): daily-detected flips
- [Weekly digest](/digest/): weekly write-ups of notable policy changes
- [Digest (RSS)](/digest/rss.xml): one item per weekly issue
- [Methodology](/about/): two public policy files per site per day, RFC 9309 parsing, no content scraping
- [Panel coverage](/health/): which tracked domains we could read, and why the rest could not — the denominator behind every percentage

## Reuse
Data is CC BY 4.0. Cite "Canicrawl" with a link.