crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools — 55 ✓ ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 55 ✓
octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
interface Data and analytics US 59 ✓
fandangoseo.com FandangoSEO: Website crawler \& SEO Log Analyzer
documentation Marketing IE 51 ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
crawlbase.com Crawlbase
interface APIs and SDKs Data and analytics — 60 ✓ ✓
cdnetworks.com 规则组 1: 针对所有爬虫的通用规则
documentation FR 48 ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 — 54 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
hankookilbo.com Hankook Ilbo — AI Agent & Web Crawler Guidelines
documentation AI and machine learning Media KR 66 ✓ ✓
spotibo.com Spotibo is not a web spider. It is a web dog! drumbigoo
transactional Marketing Developer tools — 51 ✓
coozmoo.com LLMs.txt for https://coozmoo.com/
documentation Marketing NL 47 ✓
prescriptionaestheticandwellnessspa.com Prescription Aesthetic and Wellness
transactional Marketing Healthcare FR 51 ✓
gauntletai.com Gauntlet AI — AI Crawler Policy & Site Description
US 49 ✓ ✓
theplusaddons.com The Plus Addons for Elementor
documentation — 61 ✓ ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 61 ✓ ✓
pr-cy.io pr-cy.io — SEO Tools & Website Analysis
informational Marketing AI and machine learning +1 RU 51 ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
aerosend.io Aerosend
transactional Marketing Communications — 50 ✓ ✓
supermemory.ai supermemory
interface AI and machine learning AI agents and MCP +1 — 60 ✓ ✓
piperic.com Piperic
documentation Ecommerce Data and analytics FI 53 ✓
spd.tech SPD Technology
documentation AI and machine learning Developer tools +1 — 56 ✓
sen7ry.com Sentry
— 46 ✓ ✓
errortrace.dev Sentry
US 57 ✓ ✓
promptcloud.com PromptCloud
documentation AI and machine learning Ecommerce +2 US 53 ✓
juiceddigital.com Digital PR Expert | Digital Marketing & Search Leaders
documentation Marketing US 51 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
skyfire.xyz Skyfire Developer Portal Documentation
interface Developer tools AI and machine learning +3 — 52 ✓
scrapfly.io Scrapfly
interface AI agents and MCP APIs and SDKs +1 US 47 ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 56 ✓ ✓
konabayev.com Konabayev — B2B Marketing, Browser Extensions, Apify Actors
transactional Marketing AI and machine learning — 54 ✓ ✓
evisio.co Evisio
documentation Marketing Productivity +1 US 47 ✓
jetoctopus.com Technical SEO Platform for Google & AI Search
documentation Marketing — 49 ✓
seotoolsforexcel.com SeoTools for Excel
documentation Marketing DE 56 ✓ ✓
hasdata.com HasData
interface APIs and SDKs Data and analytics — 56 ✓
prisync.com Prisync
documentation Ecommerce Data and analytics — 58 ✓
icade.fr Specific rules for AI crawlers
informational FR 47 ✓
modpagespeed.com mod_pagespeed 2.1
documentation Infrastructure Developer tools US 58 ✓ ✓
gaiaonline.com Gaia Online - AI Crawler Guidance
US 50 ✓
chinapp.com 抓取间隔
informational CN 49 ✓
janbaskdigitaldesign.com JanBask Digital Design — Web Design, Web Development & Digital Marketing Agency
documentation Developer tools Marketing +1 — 57 ✓ ✓
cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
findglocal.com FindGlocal agent search
— 46 ✓
firecrawl.dev Firecrawl
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
nohacks.co No Hacks - AI Agent Guidelines
informational AI and machine learning Media +1 — 51 ✓ ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
openclaw.ai OpenClaw
documentation AI and machine learning AI agents and MCP US 47 ✓
bangaloreashram.org LLMs URL Index
— 47 ✓
scrapegraphai.com llms.txt
interface AI agents and MCP APIs and SDKs US 53 ✓ ✓
avtoelon.uz Instructions for AI models crawling avtoelon.uz
informational AI and machine learning UZ 54 ✓ ✓
ottuhr.com OTTUHR
informational FR 45 ✓
thriveweb.com.au Thrive Digital Web Design Gold Coast: Websites that grow with your business ⋆ Gold Coast Web Design ⋆ Wordpress Support \& Hosting ⋆ Coding \& Development ⋆ Based in Burleigh Heads, Gold Coast, Austra
informational Design Local business AU 48 ✓
hashtag.org hashtag.org
documentation AI and machine learning — 58 ✓ ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs LV 49 ✓
niural.com Niural
interface Finance and payments Productivity +1 US 52 ✓
itagroup.com ITA Group
informational Data and analytics Marketing US 56 ✓
activesearchresults.com Active Search Results
documentation Marketing US 58 ✓
gteek.com Gteek.com - Roadmap and Context for AI Agents and Large Language Models (LLMs)
documentation AI and machine learning BG 47 ✓
jump.bg llms.txt for Jump.BG
transactional Infrastructure BG 50 ✓ ✓
ocaml.org OCaml
documentation Developer tools Open source projects FR 61 ✓
snapser.com Snapser (snapser.com)
US 45 ✓
maxburst.com Web Design Company & SEO Marketing Agency in NYC & Long Island
informational Marketing Design US 60 ✓ ✓
tomba.io Tomba Email Finder
interface Marketing US 51 ✓
aerones.com Robotic Wind Turbine Care Systems | Aerones
informational Industry and manufacturing Energy and utilities — 53 ✓
dotcom-monitor.com Dotcom-Monitor
documentation DevOps and observability US 57 ✓
tech-now.io TechNow (tech-now.io)
documentation — 47 ✓
truenorthsocial.com True North Social - Full-Service Digital Marketing Agency
informational Marketing Media US 59 ✓
sitespeed.io sitespeed.io and Browsertime
documentation Developer tools DevOps and observability +1 US 51 ✓
honour.app Honour.app
informational Recruiting and careers Education IN 49 ✓
stallionesearch.com StallionESearch
informational AI and machine learning US 46 ✓
webless.ai Webless
documentation AI and machine learning Business software US 51 ✓
mupdf.com MuPDF
documentation Developer tools APIs and SDKs +1 US 62 ✓ ✓
d3corp.com D3
informational US 48 ✓
overgrow.com Overgrow.com
informational Community CZ 57 ✓
techenclave.com TechEnclave
informational Community — 56 ✓
angeo.dev angeo.dev
documentation Ecommerce AI agents and MCP FR 46 ✓ ✓
unicyclist.com Unicyclist.com
informational Community FI 55 ✓
ziggit.dev Ziggit
documentation Community Developer tools US 55 ✓
isharkfly.com iSharkFly
documentation Community — 54 ✓
sat4all.com Sat4all
documentation Communications Community NL 54 ✓
potensic.com Potensic Official Website
transactional Consumer electronics US 55 ✓
forospyware.com ForoSpyware
documentation Cybersecurity Community — 53 ✓
clojureverse.org ClojureVerse
documentation Community Developer tools FR 52 ✓
specification.website The Website Specification
interface Documentation and reference — 53 ✓ ✓
cometforums.com CometForums
documentation Community — 52 ✓
ifin.network IFIN
documentation US 52 ✓
londonbikers.com londonbikers.com
informational Community — 52 ✓
igears.net iGears Technology Limited
documentation AI and machine learning Cybersecurity — 52 ✓
s4talk.in S4Talk
documentation Community IN 52 ✓
tiexuedanxin.net 铁血丹心
informational Community HK 52 ✓
sorumatik.co Sorumatik
documentation Education AI and machine learning — 51 ✓
6do.world 六度世界
documentation Community Media — 48 ✓
turborenault.co.uk TurboRenault.co.uk
informational Community Automotive GB 47 ✓
sendegate.de Sendegate
documentation Community DE 46 ✓
routerich.ru Routerich AX3000
informational Consumer electronics RU 45 ✓
y2kwebs.com Y2K Webs
documentation Marketing Ecommerce CR 53 ✓
krita-artists.org Krita Artists
informational Arts and culture Community DE 56 ✓
penpot.app Penpot
documentation Design — 64 ✓ ✓
k8slens.dev Lens Forums
documentation Community GB 57 ✓
tstprep.com TST Prep
transactional Education — 56 ✓ ✓
farbentour.de Farbentour
documentation Marketing DE 46 ✓
yourgpt.ai YourGPT - The Complete AI Agent Platform
interface AI and machine learning — 57 ✓ ✓
smytten.com Smytten
transactional Ecommerce Fashion and beauty IN 57 ✓
pushleads.com PushLeads | Asheville SEO Services: We’re Asheville SEO Experts at growing your business. Our SEO services boost your bottom line with powerful onsite and offsite SEO. Be visible and grow!
documentation Marketing — 49 ✓
okhten.com Artem Okhten Official Website
documentation AI and machine learning US 57 ✓ ✓
sanyglobal.com SANY Group | Global Website
documentation Industry and manufacturing Construction and architecture FR 58 ✓
freehost.com.ua Разрешаем OpenAI GPTBot (ChatGPT)
documentation UA 50 ✓
ud.hk UD (UDomain) — ud.hk
documentation Cybersecurity Infrastructure +1 HK 46 ✓
onwebchat.com onWebChat
documentation AI and machine learning Communications DE 54 ✓
natu.care Natu.Care
informational Ecommerce US 55 ✓
powerthesaurus.org Power Thesaurus
documentation Education Documentation and reference — 54 ✓
socure.com Socure
documentation Cybersecurity Authentication and identity US 59 ✓
igears.com.hk iGears Technology Limited
documentation Marketing Business software +1 HK 46 ✓
easymusic.ai EasyMusic.AI
transactional AI and machine learning Entertainment and streaming GB 54 ✓
kontron-ais.com kontron-ais.com - llms.txt (for AI crawlers and generative engines)
documentation Industry and manufacturing Business software DE 54 ✓
askeygeek.com ASKEYGEEK.com
documentation AI and machine learning — 48 ✓
webotit.ai Webotit.ai
documentation AI and machine learning US 58 ✓ ✓
verifieddr.com VerifiedDR
interface AI and machine learning Marketing +1 — 51 ✓ ✓
svenkersten.nl Sven Kersten
documentation Marketing NL 46 ✓
zenmedia.com Zen Media: B2B Marketing and PR Agency for B2B Brands
documentation Marketing Media — 53 ✓
laserliner.com Laserliner - AI Crawler Index
documentation AI and machine learning Consumer electronics DE 53 ✓ ✓
vaperanger.com Allow all known major AI and LLM bots to crawl the entire site
informational US 45 ✓
teenager365.to llms.txt
— 47 ✓
casrai.org CASRAI — AI Citation Guide
documentation Documentation and reference Science and research — 58 ✓
mailbutler.io Mailbutler
documentation Productivity DE 53 ✓
tryghed.dk TryghedsGruppen - AI Agent Information
informational AI and machine learning DK 46 ✓
elitefourum.com Elite Fourum
informational Community US 56 ✓
troypointinsider.com TROYPOINT Insider
informational Community US 54 ✓
bcharts.com.br BCharts Fórum
informational Community Communications BR 53 ✓
openenergymonitor.org OpenEnergyMonitor Community
documentation Community — 53 ✓
hitmanforum.com Hitman Forum
informational Community — 52 ✓
erlangforums.com Erlang Forums
documentation Community DE 51 ✓
idmforums.com IDM Forums
documentation Community US 51 ✓
image.sc Image.sc Forum
informational Community US 50 ✓
kenyatalk.com Kenya Talk
documentation Community — 50 ✓
dastereo.ru DA Stereo
informational Community RU 49 ✓
finsweet.com Finsweet
informational Marketing — 49 ✓
thinkgeo.com ThinkGeo
interface Developer tools AI agents and MCP +1 US 54 ✓
hostedscan.com HostedScan
interface Cybersecurity Infrastructure +1 DE 52 ✓
levamo.com Levamo
documentation Education Infrastructure — 56 ✓ ✓
bbcincorp.com BBCIncorp
documentation Business software — 54 ✓
cuny.com C.U.N.Y. Digital
documentation Marketing Nonprofit and advocacy — 57 ✓
wowlazymacros.com WoW Lazy Macros
informational Gaming Community — 53 ✓
colorlibsupport.com Colorlib Support Forum
informational Community — 52 ✓
euler.finance Euler
interface Finance and payments — 52 ✓
failheap-challenge.com Failheap-Challenge
informational Community — 51 ✓
thephilosophyforum.com The Philosophy Forum
informational Community — 51 ✓
timetravelinstitute.com Time Travel Institute
documentation Travel Community — 51 ✓
onehack.st OneHack a.k.a 1Hack
informational Education Community — 48 ✓ ✓
musicpusic.com Music Pusic - LLM and AI Agent Guide
documentation AI and machine learning Media IR 50 ✓
ukr.media UkrMedia — Instructions for AI Agents and LLMs
documentation AI and machine learning Media DE 55 ✓
coldstonecreamery.com Cold Stone Creamery
transactional Food and dining US 60 ✓
beyondblue.org.au Beyond Blue (Australia)
informational Healthcare AU 58 ✓
wazuh.com Wazuh
documentation Cybersecurity — 62 ✓
zenrows.com Zenrows
interface AI and machine learning APIs and SDKs +1 ES 61 ✓ ✓
computechstore.in Computech Store
transactional Ecommerce Consumer electronics IN 59 ✓ ✓
interlochen.org Interlochen Center For the Arts
documentation Education Arts and culture US 59 ✓
sociallykeeda.com SociallyKeeda
informational Media — 58 ✓ ✓
smartdatacollective.com SmartDataCollective
informational AI and machine learning Data and analytics +1 US 57 ✓
bonobolabs.com Bonobo
documentation Productivity US 56 ✓ ✓
e-jaib.com JAIB - Digital Financial Solutions
documentation Finance and payments US 53 ✓
fuelcycle.com Fuel Cycle: Insights that Ignite Action
documentation AI and machine learning Data and analytics — 53 ✓
rhinorank.io Rhino Rank
interface Marketing AI agents and MCP — 52 ✓ ✓
clearlyacquired.com ClearlyAcquired
informational Finance and payments Business software — 51 ✓
mirovina.hr mirovina.hr
informational Media HR 51 ✓
pikabooshop.com Pikabooshop
transactional Ecommerce Home and garden DE 51 ✓
vouchercloud.com Vouchercloud
transactional Ecommerce — 51 ✓
finch.com Finch LLC: E-commerce Performance Marketing Agency
informational Ecommerce Marketing US 49 ✓
tramadolusa.com Tramadol USA
transactional AI and machine learning Healthcare +1 US 47 ✓
seba.health Seba
interface Documentation and reference Science and research — 46 ✓ ✓
getpublii.com Publii Static CMS
documentation Developer tools Open source projects PL 59 ✓
adkernel.com AdKernel
documentation Marketing Data and analytics GB 64 ✓
thegww.com TheGWW\.com: News, Reviews, and Updates for Cards, Comics, Collectibles, Games, Entertainment \& Technology
documentation Gaming Media — 58 ✓
doetsreizen.nl Doets Reizen
transactional Travel NL 50 ✓
armature.tech Armature
transactional AI agents and MCP Marketing US 49 ✓
cactusjacksauto.com Cactus Jack's Auto
transactional Automotive US 49 ✓
lpuonline.com LPU Online – llms.txt
US 47 ✓
greythr.com greytHR
documentation Finance and payments Recruiting and careers +1 IN 59 ✓
rocketpool.net Rocket Pool
interface Crypto and Web3 — 58 ✓
names.co.uk llms.txt file for https://www.names.co.uk/
transactional Infrastructure GB 62 ✓ ✓
adalo.com Adalo
transactional No-code and automation — 64 ✓ ✓
mxp.de Managed Service Provider & Digitalagentur Augsburg
documentation Infrastructure Business software DE 47 ✓
juce.com JUCE: The JUCE Framework
documentation Developer tools APIs and SDKs GB 58 ✓
opensanctions.org OpenSanctions
interface Data and analytics Finance and payments US 61 ✓
casio-mea.com LLMS.txt for https://casio-mea.com/
transactional Ecommerce — 50 ✓
alumio.com Alumio - a scalable integration backbone to connect digital landscapes
documentation Marketing Business software +1 US 55 ✓
searchenginejournal.com Search Engine Journal: The AI Search \& SEO News That Matters
informational Marketing AI and machine learning +1 GB 68 ✓
l2beat.com L2BEAT
documentation Crypto and Web3 Science and research — 59 ✓
duplicati.com Duplicati
documentation Cybersecurity Infrastructure CA 60 ✓
noloco.io Noloco
documentation Business software No-code and automation — 47 ✓
stork.ai Stork.ai
documentation AI and machine learning Media US 60 ✓ ✓
paradyz.com Ceramika Paradyz
informational Industry and manufacturing Construction and architecture PL 56 ✓
particle.io Particle
interface Infrastructure APIs and SDKs US 55 ✓
gatling.io Load testing designed for DevOps and CI/CD | Gatling
documentation Developer tools DevOps and observability US 48 ✓
built2winweb.com BuiltToWinWeb
interface Design Marketing +5 US 53 ✓ ✓
rankai.ai RankAI
documentation Marketing Business software US 54 ✓
smartbear.com SmartBear Software
documentation Developer tools DevOps and observability — 63 ✓
rdp.monster rdp.monster
transactional Infrastructure — 55 ✓ ✓
near.org NEAR Protocol
documentation AI and machine learning Crypto and Web3 US 61 ✓
goniyo.com Niyo
transactional Travel Finance and payments US 56 ✓
modx.com MODX
documentation Developer tools Media +1 — 55 ✓
legiit.com Legiit
transactional Business software Recruiting and careers — 59 ✓
thunkable.com Thunkable
documentation No-code and automation AI and machine learning US 58 ✓
obkio.com Obkio Network Observability & Network Monitoring SaaS Solution
documentation DevOps and observability Infrastructure CA 56 ✓
arduino.cc Arduino homepage
transactional Consumer electronics Education — 72 ✓ ✓
ispazio.net iSpazio
documentation Media DE 57 ✓
angelcam.com Angelcam
interface Media APIs and SDKs US 58 ✓
cambiumnetworks.com Cambium Networks
interface Telecom APIs and SDKs US 62 ✓
cakewallet.com Cake Wallet
documentation Crypto and Web3 — 57 ✓
cloudcannon.com CloudCannon
documentation Media Developer tools NZ 57 ✓
localseoguide.com Local SEO Guide
informational Marketing Ecommerce US 57 ✓
georgianholidays.com Georgian Holidays
transactional Travel — 52 ✓
eliterooter.com Elite Rooter
transactional Local business US 51 ✓
vodafone.al Vodafone Albania
transactional Telecom AL 51 ✓
instaluj.cz INSTALUJ.cz
documentation CZ 50 ✓
youapply.com YouApply
documentation Education — 47 ✓
epiusers.help Epicor User Help Forum
informational Community US 57 ✓ ✓
textkit.com Textkit Greek and Latin
informational Education Community US 56 ✓
xpeng-fahrer.de Xpeng-Fahrer Community
documentation Community Automotive DE 47 ✓
denverwebsitedesigns.com Denver Website Designs
documentation Ecommerce Marketing +3 CA 51 ✓
whisperweb.dev Whisper Web
transactional Media AI and machine learning — 51 ✓ ✓
sunbeltrentals.com Sunbelt Rentals LLMs.txt
documentation US 58 ✓
againstdata.com AgainstData
documentation AI and machine learning US 55 ✓ ✓
weaviate.io Weaviate
interface Databases AI and machine learning US 55 ✓
piwik.pro Piwik PRO
documentation SE 54 ✓
bitwarden.com Bitwarden
documentation Cybersecurity Authentication and identity — 65 ✓
yugabyte.com YugabyteDB
documentation Data and analytics Databases +1 US 63 ✓ ✓
centrifuge.io Centrifuge
documentation Finance and payments Crypto and Web3 — 47 ✓
alpaca.markets Alpaca Main Page
interface Developer tools Finance and payments +1 US 57 ✓ ✓
plotly.com Plotly
documentation AI and machine learning Data and analytics +1 US 64 ✓ ✓
cosmos.network Cosmos
documentation Crypto and Web3 Developer tools — 61 ✓
launchx.com LaunchX
documentation Education US 52 ✓
fansgain.com FansGain
informational Marketing FR 50 ✓
hardwareforums.com Hardware Forums - Computer Support Forums
documentation Community — 57 ✓
llmrefs.com LLMrefs
documentation AI and machine learning Marketing US 55 ✓
blogdrip.com BlogDrip
documentation US 49 ✓
kanoplay.com Kanoplay
informational Gaming US 60 ✓
xovi.de Xovi.de
informational Marketing DE 50 ✓
nianticspatial.com Niantic Spatial
informational AI and machine learning US 52 ✓
youcanbook.me YouCanBookMe — Online Scheduling Software
transactional Productivity Finance and payments +2 US 55 ✓
homefirstindia.com Home First Finance Company India Limited
documentation Finance and payments US 55 ✓
cs2a.run No AI Scraping Allowed
informational RU 47 ✓
koreatimes.co.kr The Korea Times
documentation Industry and manufacturing Media KR 61 ✓
interlinkbooks.com Interlink Publishing
transactional Media Arts and culture — 54 ✓
evertune.ai Evertune
documentation Marketing AI and machine learning — 53 ✓
outspot.be Outspot
transactional Ecommerce Travel BE 51 ✓
parsers.vc Parsers VC
interface AI and machine learning Data and analytics +2 — 50 ✓
outspot.de Outspot
transactional Ecommerce DE 49 ✓
clawhub.ai ClawHub
interface AI agents and MCP APIs and SDKs US 56 ✓
appbasis.com App Basis Inc
documentation Developer tools Ecommerce +2 SG 57 ✓ ✓