canicrawl.com valid
Canicrawl
A daily-updated census of AI-crawler access across 1020 major websites: which sites allow, restrict, or block 32 AI user agents (robots.txt), and which publish llms.txt. You are reading the llms.txt of a site that tracks llms.txt — welcome.
- Authority
- 45 reach 0 · trust 5 · quality 25 · freshness 15
- Tier
- listed
- llms.txt
- https://canicrawl.com/llms.txt
- llms-full.txt
- https://canicrawl.com/llms-full.txt
- Sections
- 2
- Links
- 10
- Size
- 1384 bytes
- Country
- United States Hosting inference
- Hosting country
- United States Hosting inference
- Country evidence
- No published address evidence collected yet
- For agents
- Documentation: developer or product docs worth reading, nothing to call
- First seen
- 2026-10-26T00:26:22.287Z
- Last checked
- 2026-10-26T00:26:23.052Z
- Badge
Their llms.txt
The beginning of the file, from our validated copy saved 2026-10-26T00:26:22.293Z. Cached copy · Source on canicrawl.com
# Canicrawl
> A daily-updated census of AI-crawler access across 1020 major websites: which sites allow, restrict, or block 32 AI user agents (robots.txt), and which publish llms.txt. You are reading the llms.txt of a site that tracks llms.txt — welcome.
As of 2026-10-25: 31.9% of readable tracked sites block at least one AI crawler; llms.txt adoption is 12.3%.
## Data
- [Full census as plain text](/llms-full.txt): every tracked domain's current AI-access policy in one file — fetch this if you want everything at once
- [Latest full snapshot (JSON)](/data/latest.json): every domain × every bot
- [Per-site JSON](/data/sites/nytimes.com.json): replace the domain as needed
- [Stats](/stats/): headline rates, per-bot and per-category
- [Compare two sites](/compare/?a=wired.com&b=arstechnica.com): any two tracked domains side by side; the pair is in the query string
- [Policy changes (RSS)](/changelog/rss.xml): daily-detected flips
- [Weekly digest](/digest/): weekly write-ups of notable policy changes
- [Digest (RSS)](/digest/rss.xml): one item per weekly issue
- [Methodology](/about/): two public policy files per site per day, RFC 9309 parsing, no content scraping
- [Panel coverage](/health/): which tracked domains we could read, and why the rest could not — the denominator behind every percentage
## Reuse
Data is CC BY 4.0. Cite "Canicrawl" with a link.
Machine-readable: GET /v1/resolve?q=canicrawl.com · GET /v1/domain/canicrawl.com · this page as Accept: application/json or text/markdown · MCP lookup_domain
Automatically discovered from public files. Inclusion does not imply endorsement or publisher participation.
Own or manage this website?
If you own or manage canicrawl.com, you can update its listing and review file health, discovery activity and score history.
Manage this listing — optional
What verification unlocks
See how your site is discovered—and control what agents see.
- See your website in the map
Review your indexed topics, nearby publishers sharing headings, measured listing exposure and map snapshots in a private report for your website.
- Control what agents see
Edit your listing description, tags, public contact details and additional llms.txt paths. Publish your terms for automated access, attribution and pricing.
- Make your profile eligible for search
A valid, listed, unblocked, verified profile can appear in our publisher sitemap and become eligible for search-engine visibility. Search engines decide what to index and rank.
- Appear in our llms-full.txt
Eligible verified publishers are listed by name, with their file links and summaries, in the service overview agents can read.
Free. No account registration, backlink or payment required. Public discovery remains available without verification.
Let your coding agent verify it
If your coding agent is authorized to manage this website, paste these instructions into it.
Claim my listing on Agent Web Map for canicrawl.com:
1. POST https://agentwebmap.com/v1/sites/canicrawl.com/claims and keep claim_id and claim_secret private.
2. Publish the verification line "AgentWebMap verification: <token>" (token from step 1) in https://canicrawl.com/llms.txt, or use the DNS, meta-tag or well-known method the response describes. This only verifies the listing; your website and domain remain yours.
3. POST https://agentwebmap.com/v1/sites/canicrawl.com/verify with {"claim_id": "...", "claim_secret": "..."}.
4. Save the api_key from the answer; it manages the listing (PATCH https://agentwebmap.com/v1/sites/canicrawl.com).
MCP equivalent: tools begin_ownership_claim, then verify_domain, at https://agentwebmap.com/mcp. Docs: https://agentwebmap.com/docs#verifyCorrections or removal requests: contact the directory.