scrapy.org valid
Scrapy
Scrapy is a fast high-level web crawling and web scraping framework, used
For agents: Scrapy: open-source Python framework for fast web crawling and scraping, with full docs covering tutorial, API, architecture, extensions and use with coding age
web scrapingweb crawlingpython frameworkspiderscoding agents
- Authority
- 60 reach 15 · trust 10 · quality 20 · freshness 15
- Tier
- listed
- llms.txt
- https://docs.scrapy.org/llms.txt
- llms-full.txt
- https://docs.scrapy.org/llms-full.txt
- Sections
- 1
- Links
- 55
- Size
- 3111 bytes
- Popularity rank
- #26870
- Registered
- 2007-09-13
- Country
- — Publisher location unknown; CDN address excluded
- Country evidence
- No published address evidence collected yet
- Categories
- Developer tools · Open source projects
- For agents
- Documentation: developer or product docs worth reading, nothing to call
- Inferred actions
- Read documentation About these labels
- Audience
- developers · agents can: read docs · en
- First seen
- 2026-10-18T15:08:54.779Z
- Last checked
- 2026-10-25T11:35:34.990Z
- Badge
Their llms.txt
The beginning of the file, from our validated copy saved 2026-10-25T11:35:34.468Z. Cached copy · Source on scrapy.org
# Scrapy
> Scrapy is a fast high-level web crawling and web scraping framework, used
> to crawl websites and extract structured data from their pages. It can be used
> for a wide range of purposes, from data mining to monitoring and automated
> testing.
## Docs
- [Scrapy 2.19 documentation](en/latest/index.md)
- [Scrapy at a glance](en/latest/intro/overview.md)
- [Examples](en/latest/intro/examples.md)
- [Installation guide](en/latest/intro/install.md)
- [Scrapy Tutorial](en/latest/intro/tutorial.md)
- [Scrapy’s building blocks](en/latest/topics/concepts.md)
- [Add-ons](en/latest/topics/addons.md)
- [Using Scrapy with coding agents](en/latest/topics/agents.md)
- [Core API](en/latest/topics/api.md)
- [Architecture overview](en/latest/topics/architecture.md)
- [asyncio](en/latest/topics/asyncio.md)
- [AutoThrottle extension](en/latest/topics/autothrottle.md)
- [Benchmarking](en/latest/topics/benchmarking.md)
- [Command line tool](en/latest/topics/commands.md)
- [Components](en/latest/topics/components.md)
- [Spiders Contracts](en/latest/topics/contracts.md)
- [Cookies](en/latest/topics/cookies.md)
- [Coroutines](en/latest/topics/coroutines.md)
- [Debugging Spiders](en/latest/topics/debug.md)
- [Deploying Spiders](en/latest/topics/deploy.md)
- [Using your browser’s Developer Tools for scraping](en/latest/topics/developer-tools.md)
- [DjangoItem](en/latest/topics/djangoitem.md)
- [Download handlers](en/latest/topics/download-handlers.md)
- [Downloader Middleware](en/latest/topics/downloader-middleware.md)
- [Selecting dynamically-loaded content](en/latest/topics/dynamic-content.md)
- [Exceptions](en/latest/topics/exceptions.md)
- [Item Exporters](en/latest/topics/exporters.md)
- [Extensions](en/latest/topics/extensions.md)
- [Feed exports](en/latest/topics/feed-exports.md)
- [Item Pipeline](en/latest/topics/item-pipeline.md)
- [Items](en/latest/topics/items.md)
- [Jobs: pausing and resuming crawls](en/latest/topics/jobs.md)
- [Debugging memory leaks](en/latest/topics/leaks.md)
- [Link Extractors](en/latest/topics/link-extractors.md)
- [Item Loaders](en/latest/topics/loaders.md)
- [Logging](en/latest/topics/logging.md)
- [Downloading and processing files and images](en/latest/topics/media-pipeline.md)
- [Optimization](en/latest/topics/optimize.md)
- [Common Practices](en/latest/topics/practices.md)
- [Requests and Responses](en/latest/topics/request-response.md)
- [Scheduler](en/latest/topics/scheduler.md)
- [Scrapyd](en/latest/topics/scrapyd.md)
- [Security](en/latest/topics/security.md)
- [Selectors](en/latest/topics/selectors.md)
- [Settings](en/latest/topics/settings.md)
- [Scrapy shell](en/latest/topics/shell.md)
- [Signals](en/latest/topics/signals.md)
- [Spider Middleware](en/latest/topics/spider-middleware.md)
- [Spiders](en/latest/topics/spiders.md)
- [Stats Collection](en/latest/topics/stats.md)
- [Telnet Console](en/latest/topics/telnetconsole.md)
…Related sites
Similar publisher descriptions and documentation. Similarity does not imply affiliation or equivalent capabilities.
| Domain | Title | Category | Country | Authority | llms.txt | llms-full.txt | Save |
|---|---|---|---|---|---|---|---|
| webscraper.io | Web Scraper | interface Data and analytics APIs and SDKs | LV | 49 | ✓ | ||
| crawlee.dev | Crawlee for Python · Fast, reliable Python web crawlers. | documentation Open source projects Developer tools | — | 55 | ✓ | ✓ | |
| fetchfox.ai | FetchFox | interface AI agents and MCP APIs and SDKs +1 | US | 46 | ✓ | ✓ | |
| firecrawl.dev | Firecrawl | interface AI and machine learning APIs and SDKs +2 | US | 65 | ✓ | ✓ | |
| scrapegraphai.com | llms.txt | interface AI agents and MCP APIs and SDKs | US | 53 | ✓ | ✓ | |
| webscraping.ai | WebScraping.AI | interface AI and machine learning APIs and SDKs +1 | — | 56 | ✓ | ✓ | |
| octoparse.com | Web Scraping Tool & Free Web Crawlers | Octoparse | documentation Data and analytics No-code and automation | US | 59 | ✓ | ||
| scrapeops.io | ScrapeOps | interface Data and analytics APIs and SDKs | GB | 53 | ✓ | ✓ |
Machine-readable: GET /v1/resolve?q=scrapy.org · GET /v1/domain/scrapy.org · this page as Accept: application/json or text/markdown · MCP lookup_domain
Automatically discovered from public files. Inclusion does not imply endorsement or publisher participation.
Own or manage this website?
If you own or manage scrapy.org, you can update its listing and review file health, discovery activity and score history.
Manage this listing — optional
What verification unlocks
See how your site is discovered—and control what agents see.
- See your website in the map
Review your indexed topics, nearby publishers sharing headings, measured listing exposure and map snapshots in a private report for your website.
- Control what agents see
Edit your listing description, tags, public contact details and additional llms.txt paths. Publish your terms for automated access, attribution and pricing.
- Make your profile eligible for search
A valid, listed, unblocked, verified profile can appear in our publisher sitemap and become eligible for search-engine visibility. Search engines decide what to index and rank.
- Appear in our llms-full.txt
Eligible verified publishers are listed by name, with their file links and summaries, in the service overview agents can read.
Free. No account registration, backlink or payment required. Public discovery remains available without verification.
Let your coding agent verify it
If your coding agent is authorized to manage this website, paste these instructions into it.
Claim my listing on Agent Web Map for scrapy.org:
1. POST https://agentwebmap.com/v1/sites/scrapy.org/claims and keep claim_id and claim_secret private.
2. Publish the verification line "AgentWebMap verification: <token>" (token from step 1) in https://scrapy.org/llms.txt, or use the DNS, meta-tag or well-known method the response describes. This only verifies the listing; your website and domain remain yours.
3. POST https://agentwebmap.com/v1/sites/scrapy.org/verify with {"claim_id": "...", "claim_secret": "..."}.
4. Save the api_key from the answer; it manages the listing (PATCH https://agentwebmap.com/v1/sites/scrapy.org).
MCP equivalent: tools begin_ownership_claim, then verify_domain, at https://agentwebmap.com/mcp. Docs: https://agentwebmap.com/docs#verifyCorrections or removal requests: contact the directory.