crawlcrawl.com valid
crawlcrawl
The crawler API built for RAG ingestion. Markdown, structured signals, scheduled crawls, dataset storage, and change-detection diff all included at every paid tier. Start free at 1,000 credits per month, scale to enterprise. The Firecrawl alternative with more credits at every tier.
- Authority
- 46 reach 0 · trust 6 · quality 25 · freshness 15
- Tier
- listed
- llms.txt
- https://crawlcrawl.com/llms.txt
- llms-full.txt
- https://crawlcrawl.com/llms-full.txt
- Sections
- 10
- Links
- 47
- Size
- 9907 bytes
- Registered
- 2026-05-08
- Country
- — Publisher location unknown; CDN address excluded
- Country evidence
- No published address evidence collected yet
- For agents
- Documentation: developer or product docs worth reading, nothing to call
- First seen
- 2026-10-27T12:03:39.023Z
- Last checked
- 2026-10-27T12:03:40.837Z
- Badge
Their llms.txt
The beginning of the file, from our validated copy saved 2026-10-27T12:03:39.026Z. Cached copy · Source on crawlcrawl.com
# crawlcrawl
> The crawler API built for RAG ingestion. Markdown, structured signals, scheduled crawls, dataset storage, and change-detection diff all included at every paid tier. Start free at 1,000 credits per month, scale to enterprise. The Firecrawl alternative with more credits at every tier.
Base URL: https://api.crawlcrawl.com
Auth: `Authorization: Bearer crk_...`
Spec: https://crawlcrawl.com/openapi.json
Signup: https://crawlcrawl.com/signup (no card required)
## What crawlcrawl does
crawlcrawl is a single REST API that turns any URL into clean, LLM-ready markdown with structured-data signals (schema.org, Open Graph, JSON-LD, hreflang, canonical) in the same response. Scheduled crawls, change-detection diffs, dataset storage, and HMAC-signed webhooks are included at every paid tier. Global routing across 190+ countries.
## Products
- [Crawler](https://crawlcrawl.com/products/crawler): Multi-page crawl with link discovery and orphan detection. Returns markdown for every page. `POST /v1/crawls`.
- [Monitor](https://crawlcrawl.com/products/monitor): Cron-scheduled recurring crawls. HMAC-SHA256 signed webhooks fire only on real changes.
- [Scrape](https://crawlcrawl.com/products/scrape): Single-URL fetch with markdown and structured signals.
- [Search](https://crawlcrawl.com/products/search): Live web search returning structured results. `POST /v1/cloud/search`.
- [Transform](https://crawlcrawl.com/products/transform): HTML or PDF to clean markdown.
- [Render](https://crawlcrawl.com/products/render): Browser-rendered HTML for sites that need JavaScript execution.
- [Unblock](https://crawlcrawl.com/products/unblock): One API call bypasses Cloudflare, Akamai, PerimeterX, Datadome, reCAPTCHA. Drop-in for any existing scraper. `POST /v1/cloud/unblock`.
- [LLMs.txt builder](https://crawlcrawl.com/products/llms-txt): Generate /llms.txt for any site.
- [AI-bot audit](https://crawlcrawl.com/products/ai-bot-audit): Resolve which AI crawlers a site allows.
- [Per-site scrapers](https://crawlcrawl.com/products/per-site-scrapers): Pre-built configs for popular sites.
- [Proxy fetch](https://crawlcrawl.com/products/proxy-fetch): Bytes-billed HTML fetch via spider.cloud proxy. ~13x cheaper than chrome render. 4 pools, 199 countries. No JS execution. `POST /v1/cloud/proxy-fetch`.
## Actors
- [audit-onpage](https://crawlcrawl.com/products/audit-onpage): ~30 on-page SEO rules per call. Errors, warnings, info. `POST /v1/actors/audit-onpage`.
- [extract-article](https://crawlcrawl.com/products/extract-article): Trafilatura body extraction with author + date metadata. `POST /v1/actors/extract-article`.
- [check-links](https://crawlcrawl.com/products/check-links): Lychee-based broken-link validation with optional chrome or proxy retry. `POST /v1/actors/check-links`.
- [structured-data](https://crawlcrawl.com/products/structured-data): JSON-LD, Microdata, RDFa, OpenGraph, Dublin Core, Microformats in one response. `POST /v1/actors/structured-data`.
- [rend
…Related sites
Similar publisher descriptions and documentation. Similarity does not imply affiliation or equivalent capabilities.
| Domain | Title | Category | Country | Authority | llms.txt | llms-full.txt | Save |
|---|---|---|---|---|---|---|---|
| crawlbase.com | Crawlbase | interface APIs and SDKs Data and analytics +1 | — | 60 | ✓ | ✓ | |
| scraperapi.com | ScraperAPI | interface AI and machine learning APIs and SDKs +2 | US | 58 | ✓ | ||
| webclaw.io | webclaw Documentation | interface AI and machine learning AI agents and MCP +1 | DE | 50 | ✓ | ✓ | |
| scrapegraphai.com | llms.txt | interface AI agents and MCP APIs and SDKs | US | 53 | ✓ | ✓ | |
| scrapeops.io | ScrapeOps | interface Data and analytics APIs and SDKs | GB | 53 | ✓ | ✓ | |
| crawl4ai.com | Crawl4AI Cloud API | interface AI and machine learning APIs and SDKs | DE | 50 | ✓ | ||
| mrscraper.com | MrScraper | interface APIs and SDKs AI agents and MCP +1 | US | 55 | ✓ | ||
| scrape.do | Scrape.do | interface APIs and SDKs | DO | 51 | ✓ |
Machine-readable: GET /v1/resolve?q=crawlcrawl.com · GET /v1/domain/crawlcrawl.com · this page as Accept: application/json or text/markdown · MCP lookup_domain
Automatically discovered from public files. Inclusion does not imply endorsement or publisher participation.
Own or manage this website?
If you own or manage crawlcrawl.com, you can update its listing and review file health, discovery activity and score history.
Manage this listing — optional
What verification unlocks
See how your site is discovered—and control what agents see.
- See your website in the map
Review your indexed topics, nearby publishers sharing headings, measured listing exposure and map snapshots in a private report for your website.
- Control what agents see
Edit your listing description, tags, public contact details and additional llms.txt paths. Publish your terms for automated access, attribution and pricing.
- Make your profile eligible for search
A valid, listed, unblocked, verified profile can appear in our publisher sitemap and become eligible for search-engine visibility. Search engines decide what to index and rank.
- Appear in our llms-full.txt
Eligible verified publishers are listed by name, with their file links and summaries, in the service overview agents can read.
Free. No account registration, backlink or payment required. Public discovery remains available without verification.
Let your coding agent verify it
If your coding agent is authorized to manage this website, paste these instructions into it.
Claim my listing on Agent Web Map for crawlcrawl.com:
1. POST https://agentwebmap.com/v1/sites/crawlcrawl.com/claims and keep claim_id and claim_secret private.
2. Publish the verification line "AgentWebMap verification: <token>" (token from step 1) in https://crawlcrawl.com/llms.txt, or use the DNS, meta-tag or well-known method the response describes. This only verifies the listing; your website and domain remain yours.
3. POST https://agentwebmap.com/v1/sites/crawlcrawl.com/verify with {"claim_id": "...", "claim_secret": "..."}.
4. Save the api_key from the answer; it manages the listing (PATCH https://agentwebmap.com/v1/sites/crawlcrawl.com).
MCP equivalent: tools begin_ownership_claim, then verify_domain, at https://agentwebmap.com/mcp. Docs: https://agentwebmap.com/docs#verifyCorrections or removal requests: contact the directory.