spider.cloud valid

Spider

Spider is a research lab for web data quality.

For agents: Spider: web data API offering crawling, scraping, headless browser sessions, proxies and an MCP server for agents. Documented API with OpenAPI spec and per-cred

web scrapingweb crawlingheadless browserproxy apimcp server

Their llms.txt

The beginning of the file, from our validated copy saved 2026-10-21T15:41:41.829Z. Cached copy · Source on spider.cloud

# Spider

> Spider is a research lab for web data quality.
> Web crawling, scraping, and search API for AI agents and RAG pipelines.
> Any site to clean markdown or structured JSON. Rust engine, a headless
> Browser when a page needs it, proxy rotation, streaming.

- Website: https://spider.cloud
- API base: https://api.spider.cloud
- Browser: https://spider.cloud/browser/ (wss://browser.spider.cloud, up to 100 concurrent sessions)
- Proxy: https://spider.cloud/proxy-locations/ (http://proxy.spider.cloud, residential and ISP exits, rotated per request)
- Agent setup skill: https://spider.cloud/agent-skill/SKILL.md
- Full API reference: https://spider.cloud/llms-full.txt
- Docs: https://spider.cloud/docs/overview
- OpenAPI: https://spider.cloud/openapi.yaml
- MCP server: https://mcp.spider.cloud/mcp (setup: https://spider.cloud/mcp/connect)
- Pricing: https://spider.cloud/pricing ($1 per 10,000 credits; bandwidth at $1 per GB plus compute at $0.0001 per minute)
- Open source: https://github.com/spider-rs/spider (MIT)
- Support: support@spider.cloud, https://discord.spider.cloud

If you are an agent setting Spider up for a user, read
https://spider.cloud/agent-skill/SKILL.md. For the complete parameter
reference, read https://spider.cloud/llms-full.txt.

## Links

- [Website](https://spider.cloud)
- [API base](https://api.spider.cloud)
- [Browser](https://spider.cloud/browser/): wss://browser.spider.cloud, up to 100 concurrent sessions
- [Proxy](https://spider.cloud/proxy-locations/): http://proxy.spider.cloud, residential and ISP exits, rotated per request
- [Agent setup skill](https://spider.cloud/agent-skill/SKILL.md)
- [Full API reference](https://spider.cloud/llms-full.txt)
- [Docs](https://spider.cloud/docs/overview)
- [OpenAPI](https://spider.cloud/openapi.yaml)
- [MCP server](https://mcp.spider.cloud/mcp)
- [MCP setup](https://spider.cloud/mcp/connect)
- [Pricing](https://spider.cloud/pricing): $1 per 10,000 credits; bandwidth at $1 per GB plus compute at $0.0001 per minute
- [Open source](https://github.com/spider-rs/spider): MIT
- [Support email](mailto:support@spider.cloud)
- [Support on Discord](https://discord.spider.cloud)
- [FAQ](https://spider.cloud/faq/): billing, rate limits, output formats, robots.txt, blocked pages, and what happens when a crawl fails
- [Why Spider](https://spider.cloud/why-spider/): Spider against ScrapingBee, Firecrawl, Bright Data, Apify and Crawl4AI
- [Compare](https://spider.cloud/compare/): Spider vs Firecrawl, Crawl4AI, ScrapingBee, and Bright Data on success rate, latency, and cost per page

## Try it with no key

`POST /scrape` works with no API key and no signup. Nothing else does.

```bash
curl -X POST https://api.spider.cloud/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "return_format": "markdown"}'
```

It is rate limited and covers `/scrape` only. Every other route returns `402`
without a key. On a `429`, honor `Retry-After`. Get a key for anything beyond
tr
…

Related sites

Other listed sites in APIs and SDKs. See the whole category →

Machine-readable: GET /v1/resolve?q=spider.cloud · GET /v1/domain/spider.cloud · this page as Accept: application/json or text/markdown · MCP lookup_domain

Automatically discovered from public files. Inclusion does not imply endorsement or publisher participation.

Claim my listing on Agent Web Map for spider.cloud:
1. POST https://agentwebmap.com/v1/sites/spider.cloud/claims and keep claim_id and claim_secret private.
2. Publish the proof: add the line "AgentWebMap ownership: <token>" (token from step 1) to https://spider.cloud/llms.txt, or use the DNS, meta-tag or well-known method the response describes.
3. POST https://agentwebmap.com/v1/sites/spider.cloud/verify with {"claim_id": "...", "claim_secret": "..."}.
4. Save the api_key from the answer; it manages the listing (PATCH https://agentwebmap.com/v1/sites/spider.cloud).
MCP equivalent: tools begin_ownership_claim, then verify_domain, at https://agentwebmap.com/mcp. Docs: https://agentwebmap.com/docs#verify

Corrections or removal requests: contact the directory.