crawlee.dev valid
Crawlee for Python · Fast, reliable Python web crawlers.
For agents: Crawlee for Python: documentation for a web crawling and scraping library, covering crawlers, Playwright integration, and sitemap parsers.
python web crawlerweb scrapingplaywrightsitemap parsercrawlee
- Authority
- 50 reach 9 · trust 5 · quality 21 · freshness 15
- Tier
- listed
- llms.txt
- https://crawlee.dev/llms.txt
- llms-full.txt
- https://crawlee.dev/llms-full.txt
- Sections
- 1
- Links
- 365
- Size
- 30822 bytes
- Popularity rank
- #359267
- Categories
- Open source projects · Developer tools
- For agents
- Documentation: developer or product docs worth reading, nothing to call
- Audience
- developers · agents can: read docs · en
- First seen
- 2026-10-19T10:41:18.550Z
- Last checked
- 2026-10-20T22:39:37.865Z
- Badge
Their llms.txt
The beginning of the file, from our validated copy saved 2026-10-20T22:39:33.653Z. Cached copy · Source on crawlee.dev
# Crawlee for Python · Fast, reliable Python web crawlers.
## python
- [Search the documentation](https://crawlee.dev/python/search.md)
- [crawlee](https://crawlee.dev/python/api.md)
- [_AsyncSession](https://crawlee.dev/python/api/class/_AsyncSession.md)
- [_AutoscaledPoolRun](https://crawlee.dev/python/api/class/_AutoscaledPoolRun.md)
- [_BasicCrawlerOptions](https://crawlee.dev/python/api/class/_BasicCrawlerOptions.md)
- [_BasicCrawlerOptionsGeneric](https://crawlee.dev/python/api/class/_BasicCrawlerOptionsGeneric.md)
- [_CurlImpersonateResponse](https://crawlee.dev/python/api/class/_CurlImpersonateResponse.md)
- [_DatasetMetadataUpdateParams](https://crawlee.dev/python/api/class/_DatasetMetadataUpdateParams.md)
- [_DatasetMetadataUpdateParams](https://crawlee.dev/python/api/class/_DatasetMetadataUpdateParams.md)
- [_DomainState](https://crawlee.dev/python/api/class/_DomainState.md)
- [_EmptyCookies](https://crawlee.dev/python/api/class/_EmptyCookies.md)
- [_Field](https://crawlee.dev/python/api/class/_Field.md)
- [_HttpxResponse](https://crawlee.dev/python/api/class/_HttpxResponse.md)
- [_HttpxTransport](https://crawlee.dev/python/api/class/_HttpxTransport.md)
- [_ImpitResponse](https://crawlee.dev/python/api/class/_ImpitResponse.md)
- [_Middleware](https://crawlee.dev/python/api/class/_Middleware.md)
- [_NewUrlFunction](https://crawlee.dev/python/api/class/_NewUrlFunction.md)
- [_NonPersistentStatistics](https://crawlee.dev/python/api/class/_NonPersistentStatistics.md)
- [_PlaywrightCrawlerAdditionalOptions](https://crawlee.dev/python/api/class/_PlaywrightCrawlerAdditionalOptions.md)
- [_ProxyTierTracker](https://crawlee.dev/python/api/class/_ProxyTierTracker.md)
- [_PssAvailability](https://crawlee.dev/python/api/class/_PssAvailability.md)
- [_QueueMetadataUpdateParams](https://crawlee.dev/python/api/class/_QueueMetadataUpdateParams.md)
- [_QueueMetadataUpdateParams](https://crawlee.dev/python/api/class/_QueueMetadataUpdateParams.md)
- [_RequestManagerOpener](https://crawlee.dev/python/api/class/_RequestManagerOpener.md)
- [_ResourceLimits](https://crawlee.dev/python/api/class/_ResourceLimits.md)
- [_RetryableSitemapStatusError](https://crawlee.dev/python/api/class/_RetryableSitemapStatusError.md)
- [_Rules](https://crawlee.dev/python/api/class/_Rules.md)
- [_SetCookieResponse](https://crawlee.dev/python/api/class/_SetCookieResponse.md)
- [_SitemapItem](https://crawlee.dev/python/api/class/_SitemapItem.md)
- [_StorageCache](https://crawlee.dev/python/api/class/_StorageCache.md)
- [_TxtSitemapParser](https://crawlee.dev/python/api/class/_TxtSitemapParser.md)
- [_XMLSaxSitemapHandler](https://crawlee.dev/python/api/class/_XMLSaxSitemapHandler.md)
- [_XmlSitemapParser](https://crawlee.dev/python/api/class/_XmlSitemapParser.md)
- [AbortError](https://crawlee.dev/python/api/class/AbortError.md)
- [AbstractHttpCrawler](https://crawlee.dev/python/api/class/AbstractHttpCrawler.md)
- [AbstractHttpParser](https://crawlee.dev/python/api/class/Abstr
…Related sites
Other listed sites in Open source projects. See the whole category →
| Domain | Title | Category | Country | Authority | llms.txt | llms-full.txt | Save |
|---|---|---|---|---|---|---|---|
| runx.ai | runx | interface AI agents and MCP Open source projects | 65 | ✓ | ✓ | ||
| 0state.com | 0state | documentation Open source projects AI agents and MCP +1 | 61 | ✓ | |||
| ubuntu.com | Ubuntu | documentation Open source projects Infrastructure +1 | GB | 76 | ✓ | ✓ | |
| getbootstrap.com | Bootstrap | documentation Open source projects Developer tools | 74 | ✓ | ✓ | ||
| sourceforge.net | SourceForge | documentation Business software Open source projects | 73 | ✓ | |||
| nodejs.org | Node.js Documentation | documentation Developer tools Documentation and reference +1 | 69 | ✓ | |||
| ruby-lang.org | Ruby Programming Language | documentation Developer tools Open source projects +1 | US | 69 | ✓ | ✓ | |
| vite.dev | Vite | documentation Developer tools Open source projects +1 | CA | 68 | ✓ | ✓ |
Machine-readable: GET /v1/resolve?q=crawlee.dev · GET /v1/domain/crawlee.dev · this page as Accept: application/json or text/markdown · MCP lookup_domain
Automatically discovered from public files. Inclusion does not imply endorsement or publisher participation.
Own or manage this website?
If you own or manage crawlee.dev, you can update its listing and review file health, discovery activity and score history.
Manage this listing — optional
What verification unlocks
See how your site is discovered—and control what agents see.
- Control what agents see
Edit your listing description, tags, public contact details and additional llms.txt paths. Publish your terms for automated access, attribution and pricing.
- Make your profile eligible for search
A valid, listed, unblocked, verified profile can appear in our publisher sitemap and become eligible for search-engine visibility. Search engines decide what to index and rank.
- Appear in our llms-full.txt
Eligible verified publishers are listed by name, with their file links and summaries, in the service overview agents can read.
Free. No account registration, backlink or payment required. Public discovery remains available without verification.
Let your coding agent verify it
If your coding agent is authorized to manage this website, paste these instructions into it.
Claim my listing on Agent Web Map for crawlee.dev:
1. POST https://agentwebmap.com/v1/sites/crawlee.dev/claims and keep claim_id and claim_secret private.
2. Publish the proof: add the line "AgentWebMap ownership: <token>" (token from step 1) to https://crawlee.dev/llms.txt, or use the DNS, meta-tag or well-known method the response describes.
3. POST https://agentwebmap.com/v1/sites/crawlee.dev/verify with {"claim_id": "...", "claim_secret": "..."}.
4. Save the api_key from the answer; it manages the listing (PATCH https://agentwebmap.com/v1/sites/crawlee.dev).
MCP equivalent: tools begin_ownership_claim, then verify_domain, at https://agentwebmap.com/mcp. Docs: https://agentwebmap.com/docs#verifyCorrections or removal requests: contact the directory.