crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools 50 ✓ ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 50 ✓
octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
interface Data and analytics US 59 ✓
fandangoseo.com FandangoSEO: Website crawler \& SEO Log Analyzer
documentation Marketing IE 46 ✓
spotibo.com Spotibo is not a web spider. It is a web dog! drumbigoo
transactional Marketing Developer tools 46 ✓
crawlbase.com Crawlbase
interface APIs and SDKs Data and analytics 60 ✓ ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
cdnetworks.com 规则组 1: 针对所有爬虫的通用规则
documentation FR 48 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
hankookilbo.com Hankook Ilbo — AI Agent & Web Crawler Guidelines
documentation AI and machine learning Media KR 66 ✓ ✓
prescriptionaestheticandwellnessspa.com Prescription Aesthetic and Wellness
transactional Marketing Healthcare FR 46 ✓
theplusaddons.com The Plus Addons for Elementor
documentation 56 ✓ ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 56 ✓ ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
aerosend.io Aerosend
transactional Marketing Communications 50 ✓ ✓
pr-cy.io pr-cy.io — SEO Tools & Website Analysis
informational Marketing AI and machine learning +1 RU 51 ✓
supermemory.ai supermemory
interface AI and machine learning AI agents and MCP +1 55 ✓ ✓
spd.tech SPD Technology
documentation AI and machine learning Developer tools +1 51 ✓
piperic.com Piperic
documentation Ecommerce Data and analytics FI 48 ✓
sen7ry.com Sentry
46 ✓ ✓
errortrace.dev Sentry
US 57 ✓ ✓
promptcloud.com PromptCloud
documentation AI and machine learning Ecommerce +2 US 48 ✓
juiceddigital.com Digital PR Expert | Digital Marketing & Search Leaders
documentation Marketing US 46 ✓
scrapfly.io Scrapfly
interface AI agents and MCP APIs and SDKs +1 US 47 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
skyfire.xyz Skyfire Developer Portal Documentation
interface Developer tools AI and machine learning +3 47 ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 51 ✓ ✓
konabayev.com Konabayev — B2B Marketing, Browser Extensions, Apify Actors
transactional Marketing AI and machine learning 51 ✓ ✓
evisio.co Evisio
documentation Marketing Productivity +1 US 47 ✓
prisync.com Prisync
documentation Ecommerce Data and analytics 53 ✓
seotoolsforexcel.com SeoTools for Excel
documentation Marketing DE 51 ✓ ✓
hasdata.com HasData
interface APIs and SDKs Data and analytics 51 ✓
modpagespeed.com mod_pagespeed 2.1
documentation Infrastructure Developer tools US 53 ✓ ✓
gaiaonline.com Gaia Online - AI Crawler Guidance
US 50 ✓
chinapp.com 抓取间隔
informational CN 49 ✓
janbaskdigitaldesign.com JanBask Digital Design — Web Design, Web Development & Digital Marketing Agency
documentation Developer tools Marketing +1 52 ✓ ✓
cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
nohacks.co No Hacks - AI Agent Guidelines
informational AI and machine learning Media +1 51 ✓ ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
openclaw.ai OpenClaw
documentation AI and machine learning AI agents and MCP US 47 ✓
avtoelon.uz Instructions for AI models crawling avtoelon.uz
informational AI and machine learning UZ 54 ✓ ✓
firecrawl.dev Firecrawl
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
scrapegraphai.com llms.txt
interface AI agents and MCP APIs and SDKs US 48 ✓ ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs 49 ✓
thriveweb.com.au Thrive Digital Web Design Gold Coast: Websites that grow with your business ⋆ Gold Coast Web Design ⋆ Wordpress Support \& Hosting ⋆ Coding \& Development ⋆ Based in Burleigh Heads, Gold Coast, Austra
informational Design Local business AU 48 ✓
hashtag.org hashtag.org
documentation AI and machine learning 53 ✓ ✓
niural.com Niural
interface Finance and payments Productivity +1 US 47 ✓
itagroup.com ITA Group
informational Data and analytics Marketing US 51 ✓
core.app Core
transactional Crypto and Web3 53 ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 54 ✓
activesearchresults.com Active Search Results
documentation Marketing US 51 ✓
jump.bg llms.txt for Jump.BG
transactional Infrastructure BG 50 ✓ ✓
zenrows.com Zenrows
interface AI and machine learning APIs and SDKs +1 61 ✓ ✓
ocaml.org OCaml
documentation Developer tools Open source projects FR 61 ✓
clinicorp.com llms.txt for www.clinicorp.com
45 ✓
aerones.com Robotic Wind Turbine Care Systems | Aerones
informational Industry and manufacturing Energy and utilities 48 ✓
maxburst.com Web Design Company & SEO Marketing Agency in NYC & Long Island
informational Marketing Design US 55 ✓ ✓
tomba.io Tomba Email Finder
interface Marketing 51 ✓
otis.edu llms.txt — AI-friendly site summary for LLM crawlers
US 45 ✓
tech-now.io TechNow (tech-now.io)
documentation 47 ✓
gomechanic.in llms.txt for gomechanic.in
documentation IN 47 ✓
truenorthsocial.com True North Social - Full-Service Digital Marketing Agency
informational Marketing Media 52 ✓
honour.app Honour.app
informational Recruiting and careers Education IN 46 ✓
scrape.do Scrape.do
interface APIs and SDKs DO 51 ✓
webless.ai Webless
documentation AI and machine learning Business software US 46 ✓
scraperapi.com ScraperAPI
interface AI and machine learning APIs and SDKs +2 US 53 ✓
mupdf.com MuPDF
documentation Developer tools APIs and SDKs +1 US 55 ✓ ✓
vistancenetworks.com Vistance Networks
informational SE 51 ✓
leboat.com llm.txt for Le Boat LLM Crawling Guide
US 50 ✓
angeo.dev angeo.dev
documentation Ecommerce AI agents and MCP FR 45 ✓ ✓
overgrow.com Overgrow.com
informational Community CZ 57 ✓
techenclave.com TechEnclave
informational Community 56 ✓
sorumatik.co Sorumatik
documentation Education AI and machine learning 51 ✓
specification.website The Website Specification
interface Documentation and reference 52 ✓ ✓
cchcpelink.com llms.txt - Guidelines for Large Language Models and AI Crawlers
informational AI and machine learning US 48 ✓
potensic.com Potensic Official Website
transactional Consumer electronics US 50 ✓
unicyclist.com Unicyclist.com
informational Community FI 50 ✓
ziggit.dev Ziggit
documentation Community Developer tools US 50 ✓
igears.net iGears Technology Limited
documentation AI and machine learning Cybersecurity 47 ✓
ifin.network IFIN
documentation US 49 ✓
isharkfly.com iSharkFly
documentation Community 49 ✓
sat4all.com Sat4all
documentation Communications Community NL 49 ✓
forospyware.com ForoSpyware
documentation Cybersecurity Community 48 ✓
getbring.com llms.txt for getbring.com
US 46 ✓
clojureverse.org ClojureVerse
documentation Community Developer tools FR 47 ✓
cometforums.com CometForums
documentation Community 47 ✓
londonbikers.com londonbikers.com
informational Community 47 ✓
newshosting.com llms.txt for https://www.newshosting.com
informational US 55 ✓
s4talk.in S4Talk
documentation Community IN 47 ✓
tiexuedanxin.net 铁血丹心
informational Community HK 47 ✓
sendegate.de Sendegate
documentation Community DE 46 ✓
tstprep.com TST Prep
transactional Education 51 ✓ ✓
routerich.ru Routerich AX3000
informational Consumer electronics RU 45 ✓
hbzhan.com llms.txt - 专为AI模型提供的环保在线网站内容导航
CN 46 ✓
y2kwebs.com Y2K Webs
documentation Marketing Ecommerce 48 ✓
farbentour.de Farbentour
documentation Marketing DE 46 ✓
krita-artists.org Krita Artists
informational Arts and culture Community DE 51 ✓
penpot.app Penpot
documentation Design 64 ✓ ✓
k8slens.dev Lens Forums
documentation GB 57 ✓
yourgpt.ai YourGPT - The Complete AI Agent Platform
interface AI and machine learning 57 ✓ ✓
elitefourum.com Elite Fourum
informational Community US 51 ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
dastereo.ru DA Stereo
informational Community RU 49 ✓
finsweet.com Finsweet
informational Marketing 49 ✓
troypointinsider.com TROYPOINT Insider
informational Community US 49 ✓
bcharts.com.br BCharts Fórum
informational Community Communications BR 48 ✓
openeye.net llms.txt — OpenEye
CA 45 ✓
hitmanforum.com Hitman Forum
informational Community 47 ✓
erlangforums.com Erlang Forums
documentation Community DE 46 ✓
idmforums.com IDM Forums
documentation Community US 46 ✓
smytten.com Smytten
transactional Ecommerce Fashion and beauty IN 57 ✓
okhten.com Artem Okhten Official Website
documentation AI and machine learning US 52 ✓ ✓
sanyglobal.com SANY Group | Global Website
documentation Industry and manufacturing Construction and architecture FR 58 ✓
freehost.com.ua Разрешаем OpenAI GPTBot (ChatGPT)
documentation UA 50 ✓
ud.hk UD (UDomain) — ud.hk
documentation Cybersecurity Infrastructure +1 HK 46 ✓
onwebchat.com onWebChat
documentation AI and machine learning Communications DE 49 ✓
natu.care Natu.Care
informational Ecommerce US 50 ✓
powerthesaurus.org Power Thesaurus
documentation Education Documentation and reference 54 ✓
socure.com Socure
documentation Cybersecurity Authentication and identity US 59 ✓
easymusic.ai EasyMusic.AI
transactional AI and machine learning Entertainment and streaming 49 ✓
kontron-ais.com kontron-ais.com - llms.txt (for AI crawlers and generative engines)
documentation Industry and manufacturing Business software DE 49 ✓
webotit.ai Webotit.ai
documentation AI and machine learning US 53 ✓ ✓
verifieddr.com VerifiedDR
interface AI and machine learning Marketing +1 50 ✓ ✓
svenkersten.nl Sven Kersten
documentation Marketing NL 46 ✓
zenmedia.com Zen Media: B2B Marketing and PR Agency for B2B Brands
documentation Marketing Media 48 ✓
laserliner.com Laserliner - AI Crawler Index
documentation AI and machine learning Consumer electronics DE 48 ✓ ✓
teenager365.to llms.txt
47 ✓
casrai.org CASRAI — AI Citation Guide
documentation Documentation and reference Science and research 53 ✓
mailbutler.io Mailbutler
documentation Productivity DE 53 ✓
tryghed.dk TryghedsGruppen - AI Agent Information
informational AI and machine learning DK 46 ✓
image.sc Image.sc Forum
informational Community US 45 ✓
kenyatalk.com Kenya Talk
documentation Community 45 ✓
hostedscan.com HostedScan
interface Cybersecurity Infrastructure +1 DE 47 ✓
levamo.com Levamo
documentation Education Infrastructure 51 ✓ ✓
bbcincorp.com BBCIncorp
documentation Business software 49 ✓
cuny.com C.U.N.Y. Digital
documentation Marketing Nonprofit and advocacy 52 ✓
onehack.st OneHack a.k.a 1Hack
informational Education Community 48 ✓ ✓
wowlazymacros.com WoW Lazy Macros
informational Gaming Community 48 ✓
colorlibsupport.com Colorlib Support Forum
informational Community 47 ✓
failheap-challenge.com Failheap-Challenge
informational Community 46 ✓
thephilosophyforum.com The Philosophy Forum
informational Community 46 ✓
timetravelinstitute.com Time Travel Institute
documentation Travel Community 46 ✓
musicpusic.com Music Pusic - LLM and AI Agent Guide
documentation AI and machine learning Media IR 45 ✓
ukr.media UkrMedia — Instructions for AI Agents and LLMs
documentation AI and machine learning Media DE 50 ✓
coldstonecreamery.com Cold Stone Creamery
transactional Food and dining US 60 ✓
beyondblue.org.au Beyond Blue (Australia)
informational Healthcare AU 58 ✓
wazuh.com Wazuh
documentation Cybersecurity 62 ✓
computechstore.in Computech Store
transactional Ecommerce Consumer electronics IN 59 ✓ ✓
interlochen.org Interlochen Center For the Arts
documentation Education Arts and culture US 59 ✓
smartdatacollective.com SmartDataCollective
informational AI and machine learning Data and analytics +1 US 57 ✓
sociallykeeda.com SociallyKeeda
informational Media 53 ✓ ✓
rhinorank.io Rhino Rank
interface Marketing AI agents and MCP 52 ✓ ✓
bonobolabs.com Bonobo
documentation Productivity US 51 ✓ ✓
mirovina.hr mirovina.hr
informational Media HR 51 ✓
e-jaib.com JAIB - Digital Financial Solutions
documentation Finance and payments US 48 ✓
fuelcycle.com Fuel Cycle: Insights that Ignite Action
documentation AI and machine learning Data and analytics 48 ✓
clearlyacquired.com ClearlyAcquired
informational Finance and payments Business software 46 ✓
pikabooshop.com Pikabooshop
transactional Ecommerce Home and garden DE 46 ✓
tramadolusa.com Tramadol USA
transactional AI and machine learning Healthcare +1 US 46 ✓
vouchercloud.com Vouchercloud
transactional Ecommerce 46 ✓
seba.health Seba
interface Documentation and reference Science and research 45 ✓ ✓
getpublii.com Publii Static CMS
documentation Developer tools Open source projects PL 52 ✓
cloudflare.com Cloudflare
interface AI and machine learning Cybersecurity +1 85 ✓ ✓
adkernel.com AdKernel
documentation Marketing Data and analytics GB 64 ✓
thegww.com TheGWW\.com: News, Reviews, and Updates for Cards, Comics, Collectibles, Games, Entertainment \& Technology
documentation Gaming Media 51 ✓
doetsreizen.nl Doets Reizen
transactional Travel NL 50 ✓
armature.tech Armature
transactional AI agents and MCP Marketing US 48 ✓
lpuonline.com LPU Online – llms.txt
US 47 ✓
greythr.com greytHR
documentation Finance and payments Recruiting and careers +1 IN 59 ✓
rocketpool.net Rocket Pool
interface Crypto and Web3 53 ✓
names.co.uk llms.txt file for https://www.names.co.uk/
transactional Infrastructure GB 62 ✓ ✓
adalo.com Adalo
transactional No-code and automation 64 ✓ ✓
mxp.de Managed Service Provider & Digitalagentur Augsburg
documentation Infrastructure Business software DE 47 ✓
casio-mea.com LLMS.txt for https://casio-mea.com/
transactional Ecommerce 45 ✓
noloco.io Noloco
documentation Business software No-code and automation 47 ✓
searchenginejournal.com Search Engine Journal: The AI Search \& SEO News That Matters
informational Marketing AI and machine learning +1 GB 68 ✓
l2beat.com L2BEAT
documentation Crypto and Web3 Science and research 59 ✓
duplicati.com Duplicati
documentation Cybersecurity Infrastructure CA 60 ✓
particle.io Particle
interface Infrastructure APIs and SDKs 55 ✓
stork.ai Stork.ai
documentation AI and machine learning Media US 55 ✓ ✓
paradyz.com Ceramika Paradyz
informational Industry and manufacturing Construction and architecture PL 51 ✓
built2winweb.com BuiltToWinWeb
interface Design Marketing +5 52 ✓ ✓
rankai.ai RankAI
documentation Marketing Business software US 49 ✓
smartbear.com SmartBear Software
documentation Developer tools DevOps and observability 63 ✓
rdp.monster rdp.monster
transactional Infrastructure 50 ✓ ✓
near.org NEAR Protocol
documentation AI and machine learning Crypto and Web3 US 61 ✓
goniyo.com Niyo
transactional Travel Finance and payments US 51 ✓
modx.com MODX
documentation Developer tools Media +1 55 ✓
legiit.com Legiit
transactional Business software Recruiting and careers 59 ✓
thunkable.com Thunkable
documentation No-code and automation AI and machine learning US 58 ✓
obkio.com Obkio Network Observability & Network Monitoring SaaS Solution
documentation DevOps and observability Infrastructure 56 ✓
arduino.cc Arduino homepage
transactional Consumer electronics Education 72 ✓ ✓
ispazio.net iSpazio
documentation Media DE 57 ✓
angelcam.com Angelcam
interface Media APIs and SDKs US 58 ✓
cambiumnetworks.com Cambium Networks
interface Telecom APIs and SDKs US 62 ✓
localseoguide.com Local SEO Guide
informational Marketing Ecommerce US 52 ✓
vodafone.al Vodafone Albania
transactional Telecom AL 51 ✓
georgianholidays.com Georgian Holidays
transactional Travel 47 ✓
eliterooter.com Elite Rooter
transactional Local business US 46 ✓
epiusers.help Epicor User Help Forum
informational Community US 52 ✓ ✓
textkit.com Textkit Greek and Latin
informational Education Community US 51 ✓
xpeng-fahrer.de Xpeng-Fahrer Community
documentation Community Automotive DE 47 ✓
denverwebsitedesigns.com Denver Website Designs
documentation Ecommerce Marketing +3 CA 46 ✓
whisperweb.dev Whisper Web
transactional Media AI and machine learning 50 ✓ ✓
sunbeltrentals.com Sunbelt Rentals LLMs.txt
documentation 58 ✓
againstdata.com AgainstData
documentation AI and machine learning US 50 ✓ ✓
weaviate.io Weaviate
interface Databases AI and machine learning US 55 ✓
piwik.pro Piwik PRO
documentation SE 54 ✓
bitwarden.com Bitwarden
documentation Cybersecurity Authentication and identity 65 ✓
centrifuge.io Centrifuge
documentation Finance and payments Crypto and Web3 47 ✓
alpaca.markets Alpaca Main Page
interface Developer tools Finance and payments +1 52 ✓ ✓
plotly.com Plotly
documentation AI and machine learning Data and analytics +1 US 64 ✓ ✓
cosmos.network Cosmos
documentation Crypto and Web3 Developer tools 61 ✓
launchx.com LaunchX
documentation Education US 47 ✓
fansgain.com FansGain
informational Marketing FR 45 ✓
hardwareforums.com Hardware Forums - Computer Support Forums
documentation Community 52 ✓
kanoplay.com Kanoplay
informational Gaming US 60 ✓
xovi.de Xovi.de
informational Marketing DE 50 ✓
youcanbook.me YouCanBookMe — Online Scheduling Software
transactional Productivity Finance and payments +2 US 55 ✓
homefirstindia.com Home First Finance Company India Limited
documentation Finance and payments US 50 ✓
koreatimes.co.kr The Korea Times
documentation Industry and manufacturing Media KR 61 ✓
outspot.be Outspot
transactional Ecommerce Travel BE 51 ✓
parsers.vc Parsers VC
interface AI and machine learning Data and analytics +2 50 ✓
interlinkbooks.com Interlink Publishing
transactional Media Arts and culture 49 ✓
outspot.de Outspot
transactional Ecommerce DE 49 ✓
evertune.ai Evertune
documentation Marketing AI and machine learning 48 ✓
igears.com.hk iGears Technology Limited
documentation Marketing Business software +1 HK 46 ✓
clawhub.ai ClawHub
interface AI agents and MCP APIs and SDKs US 56 ✓
appbasis.com App Basis Inc
documentation Developer tools Ecommerce +2 SG 52 ✓ ✓
knownagents.com Known Agents
interface AI and machine learning AI agents and MCP +1 US 52 ✓
incrementors.com Incrementors Web Solutions
documentation Marketing PL 52 ✓
elmaouid.dz جريدة الموعد اليومي (Elmaouid Daily Newspaper)
informational Media DZ 46 ✓
pdq.com PDQ
documentation Cybersecurity DevOps and observability +1 61 ✓
thetechnovate.com Technovate
documentation Developer tools Business software 46 ✓
seomator.com SEOmator
documentation Marketing Data and analytics US 57 ✓ ✓
nuxt.com Nuxt Docs
documentation Developer tools Open source projects US 67 ✓ ✓
parkingo.com ParkinGO
transactional Travel 57 ✓
scalperscompany.mx Scalpers MX
informational Ecommerce MX 45 ✓ ✓
siteor.com Siteor — website builder and CMS
transactional Design Ecommerce +1 IE 49 ✓
anarlog.so Anarlog
interface US 47 ✓
builtwith.com BuiltWith
informational Marketing Data and analytics CA 63 ✓ ✓
arzdigital.com Arzdigital
documentation Finance and payments Media IR 60 ✓
365i.co.uk 365i, llms.txt
transactional Infrastructure GB 48 ✓
a-premium.com A-Premium
transactional Automotive Ecommerce 59 ✓
as-pl.com AS-PL
documentation Automotive 58 ✓
fonteynspas.com Fonteyn Spas
transactional Home and garden CA 54 ✓ ✓
meteozentrale.de Meteozentrale
informational Media DE 47 ✓
writesonic.com Writesonic
documentation Marketing AI and machine learning 64 ✓
wbtc.network WBTC
interface Crypto and Web3 APIs and SDKs 57 ✓ ✓
lsretail.com LS Retail
documentation Ecommerce Business software 52 ✓