crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools 50 ✓ ✓
crawlbase.com Crawlbase
interface APIs and SDKs Data and analytics 60 ✓ ✓
zenrows.com Zenrows
interface AI and machine learning APIs and SDKs +1 61 ✓ ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 56 ✓ ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
mupdf.com MuPDF
documentation Developer tools APIs and SDKs +1 US 55 ✓ ✓
weaviate.io Weaviate
interface Databases AI and machine learning US 55 ✓
supermemory.ai supermemory
interface AI and machine learning AI agents and MCP +1 55 ✓ ✓
mkennedy.codes Michael Kennedy's Blog
documentation Personal sites and blogs Developer tools US 47 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
litestar.dev Litestar
documentation Developer tools Open source projects US 51 ✓ ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
centinelanalytica.com Centinel Analytica
documentation 58 ✓ ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 51 ✓ ✓
scrapegraphai.com llms.txt
interface AI agents and MCP APIs and SDKs US 48 ✓ ✓
apify.com Apify
interface Developer tools AI agents and MCP +1 57 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
alpaca.markets Alpaca Main Page
interface Developer tools Finance and payments +1 52 ✓ ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
helpcenter.io HelpCenter.io
interface Business software APIs and SDKs DE 53 ✓ ✓
plotly.com Plotly
documentation AI and machine learning Data and analytics +1 US 64 ✓ ✓
zadroweb.com Zadro Web
documentation Developer tools Infrastructure +1 US 46 ✓
firecrawl.dev Firecrawl
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
actowizsolutions.com Actowiz Solutions
documentation Data and analytics US 48 ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 50 ✓
rdp.monster rdp.monster
transactional Infrastructure 50 ✓ ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 54 ✓
arduino.cc Arduino homepage
transactional Consumer electronics Education 72 ✓ ✓
octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
interface Data and analytics US 59 ✓
trigger.dev Trigger.dev
documentation AI and machine learning AI agents and MCP +2 61 ✓ ✓
gaiaonline.com Gaia Online - AI Crawler Guidance
US 50 ✓
prosopo.io Prosopo
documentation Cybersecurity NL 55 ✓ ✓
chinapp.com 抓取间隔
informational CN 49 ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs 49 ✓
cloudflare.com Cloudflare
interface AI and machine learning Cybersecurity +1 85 ✓ ✓
armature.tech Armature
transactional AI agents and MCP Marketing US 48 ✓
sen7ry.com Sentry
46 ✓ ✓
errortrace.dev Sentry
US 57 ✓ ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
visitcalgary.com llms.txt for visitcalgary.com
45 ✓
scrapfly.io Scrapfly
interface AI agents and MCP APIs and SDKs +1 US 47 ✓
ceos3c.com Ceos3c: Complicated Things \- Made Easy
documentation AI and machine learning Developer tools +1 US 47 ✓
scraperapi.com ScraperAPI
interface AI and machine learning APIs and SDKs +2 US 53 ✓
cdnetworks.com 规则组 1: 针对所有爬虫的通用规则
documentation FR 48 ✓
trainocate.co.jp トレノケート株式会社 LLMs.txtポリシー / Trainocate Corporation AI Content Policy (LLMs.txt) | This file defines AI crawling and training permissions for LLMs. Please read carefully. | LLMs.txt Policy for AI Crawler
informational JP 47 ✓
scrape.do Scrape.do
interface APIs and SDKs DO 51 ✓
convert.com Convert.com
interface Ecommerce Marketing +2 US 59 ✓
cloro.dev cloro
interface APIs and SDKs 50 ✓
clinicorp.com llms.txt for www.clinicorp.com
45 ✓
agent-ready.dev Agent Ready
interface AI and machine learning AI agents and MCP US 45 ✓ ✓
openeye.net llms.txt — OpenEye
CA 45 ✓
scrapeops.io ScrapeOps
interface Data and analytics APIs and SDKs GB 53 ✓ ✓
latitude.so Latitude
interface AI and machine learning AI agents and MCP +2 US 51 ✓
menafn.com Directories
documentation US 50 ✓
viseer.co Viseer.co Developer Utility Platform
documentation Developer tools TR 47 ✓
zyte.com Zyte – The complete toolbox for web scraping, data extraction, and proxy infrastructure at scale. Trusted by thousands of companies worldwide.
interface APIs and SDKs Data and analytics 55 ✓
qa.tech QA.tech
interface Developer tools AI and machine learning +1 US 48 ✓
avtoelon.uz Instructions for AI models crawling avtoelon.uz
informational AI and machine learning UZ 54 ✓ ✓
gomechanic.in llms.txt for gomechanic.in
documentation IN 47 ✓
findip.net FindIP
interface Infrastructure Cybersecurity +1 59 ✓ ✓
leboat.com llm.txt for Le Boat LLM Crawling Guide
US 50 ✓
arcade.dev Arcade
interface AI and machine learning AI agents and MCP 56 ✓ ✓
howthelightgetsin.org howthelightgetsin.org
documentation GB 46 ✓
getbring.com llms.txt for getbring.com
US 46 ✓
codersera.com Codersera
interface Recruiting and careers DE 56 ✓ ✓
host-tracker.com HostTracker
transactional DevOps and observability 60 ✓
otis.edu llms.txt — AI-friendly site summary for LLM crawlers
US 45 ✓
nevermined.ai Nevermined
interface AI agents and MCP Finance and payments +1 US 47 ✓
hbzhan.com llms.txt - 专为AI模型提供的环保在线网站内容导航
CN 46 ✓
ipgeolocation.io IPGeolocation.io
interface Cybersecurity Infrastructure +2 DE 59 ✓ ✓
kernel-error.de Kernel-Error — llms.txt
documentation Cybersecurity Personal sites and blogs DE 51 ✓ ✓
mcpplaygroundonline.com MCP Playground
documentation AI and machine learning Developer tools +1 US 51 ✓ ✓
bitculator.com Bitculator
interface Finance and payments APIs and SDKs +1 46 ✓
keboola.com Keboola
interface Data and analytics Business software US 56 ✓ ✓
mbrenndoerfer.com Michael Brenndoerfer
documentation AI and machine learning Personal sites and blogs +1 47 ✓