cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
leboat.com llm.txt for Le Boat LLM Crawling Guide
US 50 ✓
agleader.com llm.txt for agleader.com
documentation AI and machine learning US 46 ✓
jewelershowcase.com LLM Crawling Guidance
informational AI and machine learning US 47 ✓
cchcpelink.com llms.txt - Guidelines for Large Language Models and AI Crawlers
informational AI and machine learning US 48 ✓
vistancenetworks.com Vistance Networks
informational SE 51 ✓
trainocate.co.jp トレノケート株式会社 LLMs.txtポリシー / Trainocate Corporation AI Content Policy (LLMs.txt) | This file defines AI crawling and training permissions for LLMs. Please read carefully. | LLMs.txt Policy for AI Crawler
informational JP 47 ✓
scrapegraphai.com llms.txt
interface AI agents and MCP APIs and SDKs US 48 ✓ ✓
llmstxt.org llms.txt
documentation AI and machine learning Documentation and reference 60 ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
emorahealth.com Emora Health
interface Healthcare 51 ✓ ✓
tavily.com Tavily Docs
interface AI and machine learning AI agents and MCP +1 65 ✓ ✓
akuity.io Akuity
documentation Developer tools DevOps and observability +1 NL 47 ✓
otis.edu llms.txt — AI-friendly site summary for LLM crawlers
US 45 ✓
context.dev Context.dev
interface AI and machine learning APIs and SDKs +1 US 55 ✓ ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
hbzhan.com llms.txt - 专为AI模型提供的环保在线网站内容导航
CN 46 ✓
olostep.com Olostep
interface AI and machine learning APIs and SDKs 57 ✓
agarwalpackers.in Agarwal Packers and Movers (DRS Group)
documentation Logistics and shipping IN 45 ✓
qshop.ai Qshop
transactional Design Marketing +1 52 ✓
shaneco.com llm.txt — Guidance for Large Language Models (LLMs)
informational AI and machine learning FR 49 ✓
mylittlebigweb.com My Little Big Web - Agence Web à Montréal. Tous les services au même endroit. SEO, conception de site Web, Google Ads, Facebook Ads, réseaux sociaux.
documentation Marketing 48 ✓
pebblehost.com llms.txt — Guidance for Large Language Models
informational AI and machine learning 45 ✓
hiredigital.com Hire Digital
transactional Business software Recruiting and careers US 47 ✓
zoetisus.com LLMS.txt - Large Language Model Sitemap
documentation AI and machine learning US 49 ✓
startupn.kr 스타트업엔 (StartupN) 공식 LLM 가이드 문서
documentation AI and machine learning Media KR 47 ✓
legalzoom.com :blue_book: LegalZoom LLMs.txt — Guidance for AI Crawlers & Synthesizers
50 ✓
mostbetitalian.it llms.txt
informational AI and machine learning IT 46 ✓
gkzhan.com llms.txt - 专为AI模型提供的智能制造网网站内容导航
CN 46 ✓
creatorlink.net == Creatorlink LLM Instructions ==
US 48 ✓
ooredoo.ps Ooredoo Palestine
transactional Telecom PS 56 ✓ ✓
logo-teka.com LogoTeka (logo-teka.com)
documentation Design RU 47 ✓
swk.de llms.txt – SWK High Priority Pages for LLMs
DE 46 ✓ ✓
crawl4ai.com Crawl4AI Cloud API
interface AI and machine learning APIs and SDKs DE 45 ✓
arcteryx.co.kr llms.txt
documentation Fashion and beauty KR 45 ✓ ✓
lesraffineurs.com Les Raffineurs — contexte pour crawlers IA
informational 50 ✓ ✓
stytch.com Stytch
interface Authentication and identity APIs and SDKs 55 ✓ ✓
afzhan.com llms.txt - 专为AI模型提供的智慧城市网内容导航
CN 45 ✓
artisantg.com LLM Guide for www.artisantg.com
US 45 ✓
univ-danubius.ro llms.txt – Universitatea Internațională Danubius
documentation Education RO 52 ✓
sita.aero llm.txt
SE 47 ✓
mythfactoryshop.com Myth Factory Shop
transactional Industry and manufacturing Ecommerce 46 ✓ ✓
healthmug.com Healthmug AI Agent Guidelines
documentation AI and machine learning IN 56 ✓
wikiprompt.org Wikiprompt
interface AI and machine learning US 46 ✓
lyfta.app Lyfta
transactional Sports and fitness US 47 ✓
safe.com llm.txt for safe.com
US 47 ✓
belmont.edu LLM Crawling Directives for Belmont University
informational Education 56 ✓
grupobimbo.com Grupo Bimbo - LLM Context Gateway
documentation Food and dining US 54 ✓
avtoelon.uz Instructions for AI models crawling avtoelon.uz
informational AI and machine learning UZ 54 ✓ ✓
codetrendy.com CodeTrendy
interface Marketing Productivity 51 ✓ ✓
luciq.ai llms.txt for luciq.ai
45 ✓
semrush.com Semrush
interface Marketing US 69 ✓
sbicard.com =====================================================
IN 49 ✓
instabug.com llms.txt for luciq.ai
US 49 ✓
openeye.net llms.txt — OpenEye
CA 45 ✓
medexdtc.com Medex Diagnostic and Treatment Center in Queens, NY LLMS
informational AI and machine learning Healthcare US 46 ✓
visittrentino.info LLMs.txt - Sitemap for AI content discovery
DE 51 ✓
impact.com llms.txt for impact.com
53 ✓
bilinguistics.com Bilinguistics — llms.txt
transactional Education Healthcare 51 ✓ ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
gomechanic.in llms.txt for gomechanic.in
documentation IN 47 ✓
kernel-error.de Kernel-Error — llms.txt
documentation Cybersecurity Personal sites and blogs DE 51 ✓ ✓
kombo.co Kombo
transactional Travel 52 ✓
albemarle-london.com London Theatre Direct LLM Content Guide
informational AI and machine learning GB 47 ✓
dealls.com Dealls llms.txt (optimized v2)
45 ✓
lg.com LG Electronics (lg.com) – Official Content Guide for AI (Global Hub)
transactional Consumer electronics CA 68 ✓
treatwell.co.uk llm.txt — Treatwell
GB 45 ✓
plazapremiumlounge.com Plaza Premium Lounge
transactional AI and machine learning Travel 56 ✓ ✓
xecor.com Xecor (Xecor Electronics) - AI Procurement & Indexing Manifest
transactional AI and machine learning Industry and manufacturing 45 ✓
kom.pe KOM Agencia Digital – Official AI & LLM Context File
documentation Marketing PE 51 ✓ ✓
hemp.com Hemp.com
documentation Education Agriculture US 50 ✓ ✓
obeautycase.com Obeautycase
documentation Industry and manufacturing Fashion and beauty +1 51 ✓ ✓
brussels-school.be Brussels School of Governance
documentation Education BE 52 ✓ ✓
coinunited.io CoinUnited.io
transactional Finance and payments Crypto and Web3 47 ✓ ✓
eduyush.com Eduyush
transactional Education CA 53 ✓ ✓
assemblyai.com AssemblyAI
interface AI and machine learning APIs and SDKs +1 US 61 ✓
booksnbobs.com Books n Bobs
transactional Ecommerce SG 46 ✓
flame.edu.in FLAME University
informational Education IN 55 ✓ ✓
tweaktown.com TweakTown
documentation Media Consumer electronics 59 ✓
tut-news.ru ТУТ НОВОСТИ: Последние новости Москвы, России и мира. Новости сегодня онлайн
informational Media RU 46 ✓
s-fl.ru SFLowers
transactional Ecommerce RU 50 ✓
omegapediatrics.com Omega Pediatrics
transactional AI and machine learning Healthcare 51 ✓ ✓
digital-loop.com Digital Loop
documentation Marketing AI and machine learning +1 DE 46 ✓
polysimulator.com PolySimulator
documentation Finance and payments 51 ✓ ✓
mock-server.com MockServer
interface Developer tools AI agents and MCP 52 ✓ ✓
aeldresagen.dk Ældre Sagen
documentation Nonprofit and advocacy DK 48 ✓
lightweb-media.de Lightweb Media GmbH
documentation Marketing Media +2 DE 52 ✓ ✓
spotdraft.com SpotDraft
documentation Legal Business software +1 US 61 ✓ ✓
i.pn i.pn
interface APIs and SDKs PN 50 ✓ ✓
codeminer42.com Codeminer42
documentation Developer tools Business software 48 ✓
sahistory.org.za South African History Online (SAHO)
documentation Education Science and research ZA 52 ✓
wildbeimwild.com IG Wild beim Wild
documentation AI and machine learning Nonprofit and advocacy US 49 ✓
daskeyboard.com Das Keyboard
transactional Consumer electronics US 64 ✓ ✓
crawlbase.com Crawlbase
interface APIs and SDKs Data and analytics 60 ✓ ✓
hexahealth.com HexaHealth
transactional Healthcare IN 59 ✓
cyberpowerpc.com CyberPowerPC – llms.txt
informational 54 ✓
themallbd.com The Mall BD
transactional Ecommerce Fashion and beauty 51 ✓ ✓
lorealparisusa.com LLMs.txt - Master File
53 ✓
knmasters.com KNmasters
documentation Marketing 45 ✓
simeononsecurity.com llms.txt for SimeonOnSecurity
45 ✓ ✓
twi-global.com TWI — Technical Excellence in Materials Joining and Engineering
documentation Science and research GB 56 ✓
code-brew.com Code Brew Labs
informational AI and machine learning US 50 ✓
smartage.pl SmartAge.pl - Information for Large Language Models (LLMs) and AI Agents
informational AI and machine learning Media PL 47 ✓
concentrix.com Concentrix llms.txt
US 47 ✓
sidecare.com SideCare
documentation Healthcare Insurance 50 ✓
senico-electronics.com SENICO Electronics
informational 47 ✓
michael.com Michael J. Saylor
informational Education Data and analytics +1 FR 54 ✓ ✓
rhinoafrica.com Rhino Africa Safaris
documentation Travel 51 ✓
a-foreign-affair.com A Foreign Affair > Meet beautiful foreign women seeking men for marriage.
informational Community US 47 ✓ ✓
pokekipe.com Pokékipe, Competitive Pokémon Analytics
documentation Data and analytics Gaming 51 ✓ ✓
zr.ru Za Rulem (zr.ru)
informational Media Automotive RU 58 ✓
buylikesservices.com BuyLikesServices
transactional Marketing 49 ✓ ✓
freiburg.de www.freiburg.de
informational Government and public services DE 61 ✓ ✓
kast.xyz KAST
documentation Finance and payments Crypto and Web3 55 ✓ ✓
progressiverobot.com llms.txt - Progressive Robot
informational 48 ✓ ✓
clac.de CLAC Citylogistik
transactional Logistics and shipping DE 46 ✓
dixie.com Dixie® (dixie.com)
informational 46 ✓ ✓
eventticketscenter.com Event Tickets Center - LLM Context & Documentation
US 51 ✓
gurully.com Gurully
transactional Education IN 61 ✓ ✓
reepl.io Reepl
interface AI and machine learning Marketing 50 ✓ ✓
6clicks.com 6clicks
documentation Cybersecurity Business software 54 ✓ ✓
abyssale.com Abyssale
interface APIs and SDKs Marketing +1 52 ✓ ✓
roatpkz.com Roat Pkz
documentation Gaming 55 ✓
asuperiormechanical.com A Superior Mechanical
transactional Travel Local business US 47 ✓
ergodirect.com ErgoDirect
transactional Ecommerce Healthcare +2 46 ✓ ✓
spartan.com Spartan Race
transactional Sports and fitness Events and conferences 66 ✓ ✓
easyapplai.com EasyApplAI
transactional Productivity Recruiting and careers US 55 ✓ ✓
kyivindependent.com The Kyiv Independent
documentation Media 64 ✓
free3d.io free3D.io
transactional Design 46 ✓ ✓
discount-plomberie.com AI Crawling Directives for Displomb
documentation AI and machine learning Ecommerce FR 47 ✓
import.io Import.io
interface AI and machine learning Data and analytics +1 56 ✓ ✓
url.edu llms.txt — Universitat Ramon Llull (URL)
ES 45 ✓
e-inscricao.com E-inscricao
transactional Events and conferences 56 ✓
bebepolis.es AI Crawling Directives for Bebepolis
transactional AI and machine learning Ecommerce ES 47 ✓
espc.com llms.txt
documentation Local business Real estate GB 58 ✓
warminboxes.com WarmInboxes.com
transactional Marketing Communications 51 ✓ ✓
easeofbiz.com EaseOfBiz
documentation 49 ✓
pricelists.org Pricelists.org
interface AI and machine learning APIs and SDKs +1 58 ✓ ✓
craftyartapp.com Crafty Art — AI Overview (llms.txt)
52 ✓ ✓
beautycoiffure.com AI Crawling Directives for PrestaShop
transactional AI and machine learning Ecommerce +1 59 ✓
seret.co.il Seret (סרט)
informational Media Entertainment and streaming IL 45 ✓
craftholsters.com Craft Holsters
transactional Ecommerce US 55 ✓ ✓
websupergoo.com WebSupergoo
documentation Developer tools Documentation and reference US 50 ✓ ✓
dialzara.com Dialzara
transactional AI and machine learning Communications +1 55 ✓ ✓
magnaquest.com Magnaquest
documentation Business software IN 49 ✓
stb.com.mk Stopanska Banka AD — Skopje
documentation Finance and payments MK 54 ✓ ✓
hotelkompas.pl HotelKompas
informational Travel PL 50 ✓ ✓
puzl.com AI Access Policy for https://www.Puzl.com
US 46 ✓
buildologyinc.com Buildology Landscaping Supply Store
informational Construction and architecture 45 ✓
triptotemples.com Trip To Temples
transactional Travel IN 50 ✓ ✓
quotidianoarte.it Quotidianoarte.it
informational Arts and culture Media IT 45 ✓ ✓
optro.ai Optro LLM Directive & Information
documentation AI and machine learning Business software +1 CA 52 ✓
shoplive.cloud Shoplive
documentation Ecommerce Media US 64 ✓ ✓
infracloud.io llms.txt - Structured Data Guide for LLMs
documentation Developer tools Infrastructure DE 46 ✓
snapcraft.io Snapcraft
documentation Developer tools Open source projects GB 65 ✓ ✓
charmhub.io Charmhub
documentation Infrastructure DevOps and observability +1 GB 54 ✓ ✓
tciexpress.in TCI Express Limited
documentation Logistics and shipping IN 61 ✓ ✓
onitdigital.com Onit Digital
documentation Automotive Marketing +1 US 51 ✓ ✓
wdracing.eu AI Crawling Directives for WD Racing
transactional AI and machine learning Ecommerce +1 48 ✓
agentwebmap.com Agent Web Map
interface AI and machine learning AI agents and MCP +1 FR 60 ✓ ✓
acemapp.org LLM Instructions for AI Models
informational AI and machine learning US 47 ✓
optimerch.de Optimerch GmbH
documentation Marketing DE 47 ✓
binadit.com Binadit
documentation Ecommerce Infrastructure +1 NL 51 ✓ ✓
prerender.io Prerender.io
documentation AI and machine learning Infrastructure 53 ✓
partbase.com Partbase — Industrial MRO Supplies for Europe
transactional Industry and manufacturing Ecommerce 47 ✓ ✓
appsecsanta.com AppSec Santa
informational Cybersecurity 51 ✓ ✓
seosherpa.com SEO SHERPA™
documentation Marketing 52 ✓
thewaternetwork.com The Water Network | by AquaSPE
informational Community DE 49 ✓
51papaya.com 51PAPAYA
documentation Entertainment and streaming AI and machine learning US 52 ✓ ✓
teamretro.com About TeamRetro: Think together. Improve together
interface Productivity US 50 ✓ ✓
sebraeplay.com.br Sebrae Play
documentation Education BR 51 ✓ ✓
medlife.ro MedLife
documentation Healthcare RO 56 ✓ ✓
tax-fin-lex.si Tax-Fin-Lex: Slovenia's Premier Legal Information Portal
SI 51 ✓ ✓
mida.so Mida
interface Ecommerce Marketing +1 58 ✓ ✓
qafiyah.com قافية (Qafiyah)
interface Arts and culture APIs and SDKs 46 ✓
magnet.me Magnet.me
informational Recruiting and careers NL 47 ✓
osclass-classifieds.com ============================================================================
documentation Recruiting and careers Marketing +2 SK 48 ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 51 ✓ ✓
bioleaderpack.com Bioleader®
transactional Food and dining Industry and manufacturing 46 ✓
siteway.de Siteway
documentation Marketing DE 52 ✓ ✓
stuller.com Stuller Inc. — LLM Guidance File (llms.txt)
documentation US 51 ✓
apify.com Apify
interface Developer tools AI agents and MCP +1 57 ✓
marcapiumamaterassi.it AI Crawling Directives for Marcapiuma
transactional AI and machine learning Ecommerce +1 IT 48 ✓
stark.marketing stark.marketing
documentation Marketing Ecommerce 46 ✓
zoundhouse.de zoundhouse.de
transactional AI and machine learning Ecommerce DE 47 ✓
kuipersagrishop.nl AI Crawling Directives for Kuipers Agrishop
transactional AI and machine learning Ecommerce +1 NL 46 ✓
motorstore.pl AI Crawling Directives for motorstore.pl
transactional AI and machine learning Ecommerce +1 PL 48 ✓
ornibird.com AI Crawling Directives for Ornibird SRL
transactional AI and machine learning Ecommerce +1 50 ✓
camelactive.com camelactive.com
transactional AI and machine learning Ecommerce 51 ✓
mediatel.at media.tel
transactional AI and machine learning Ecommerce AT 49 ✓
bell-anesse-en-provence.fr AI Crawling Directives for BellAnesse
transactional AI and machine learning Ecommerce +1 FR 46 ✓
carclean.com Carclean
transactional AI and machine learning Ecommerce NL 49 ✓
decoplus-parquet.com AI Crawling Directives for Decoplus FRANCE
documentation AI and machine learning Ecommerce +1 FR 46 ✓
dmsports.fr AI Crawling Directives for DM Sports V2
transactional AI and machine learning Ecommerce +1 FR 45 ✓
groix-et-nature.com AI Crawling Directives for Groix et Nature
transactional AI and machine learning Ecommerce FR 46 ✓
bresser.de Bresser
transactional AI and machine learning Ecommerce +1 DE 52 ✓
fischersports.com Fischer Sports
transactional AI and machine learning Ecommerce +1 DE 59 ✓
groupmap.com About GroupMap: Better group decisions, together
transactional Business software Productivity US 47 ✓ ✓
raab-verlag.de Raab Verlag
transactional AI and machine learning Ecommerce DE 49 ✓
noch.com NOCH
transactional AI and machine learning Ecommerce DE 47 ✓
pfundskerl-xxl.de Pfundskerl Onlineshop
transactional AI and machine learning Ecommerce DE 48 ✓
seit1887.de Kräuterparadies seit 1887
transactional AI and machine learning Ecommerce DE 47 ✓
evomi.com Evomi
interface Infrastructure 59 ✓
joshua.hu Joshua Rogers' Scribbles — joshua.hu
informational Cybersecurity Personal sites and blogs HU 52 ✓ ✓
news1.kr News1 (뉴스1)
informational Media KR 58 ✓
mbrenndoerfer.com Michael Brenndoerfer
documentation AI and machine learning Personal sites and blogs +1 47 ✓
spice.ai Spice AI
interface AI and machine learning Data and analytics US 52 ✓ ✓
firecrawl.dev Firecrawl
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
activesearchresults.com Active Search Results
documentation Marketing US 51 ✓
built2winweb.com BuiltToWinWeb
interface Design Marketing +5 52 ✓ ✓
proxidize.com Proxidize
interface Infrastructure APIs and SDKs 57 ✓ ✓
rankai.ai RankAI
documentation Marketing Business software US 49 ✓
actowizsolutions.com Actowiz Solutions
documentation Data and analytics US 48 ✓
dejan.ai DEJAN
documentation AI and machine learning Ecommerce +1 US 58 ✓
seomator.com SEOmator
documentation Marketing Data and analytics US 57 ✓ ✓
celebree.com Celebree School of Henrico
transactional Marketing Education +1 US 48 ✓
aioseo.com AIOSEO
documentation Marketing 66 ✓ ✓
brightdata.com Bright Data
interface AI agents and MCP APIs and SDKs +1 65 ✓ ✓
cmarix.com CMARIX Infotech
informational Developer tools Business software 52 ✓
semtrix.de Semtrix – Full-Service Search-Experience-Agentur für SEO, SEA und GEO
documentation Marketing DE 51 ✓ ✓
promptcloud.com PromptCloud
documentation AI and machine learning Ecommerce +2 US 48 ✓
zyte.com Zyte – The complete toolbox for web scraping, data extraction, and proxy infrastructure at scale. Trusted by thousands of companies worldwide.
interface APIs and SDKs Data and analytics 55 ✓
joost.blog joost.blog
documentation Marketing Personal sites and blogs 52 ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
theirstack.com TheirStack
interface Data and analytics Business software US 55 ✓ ✓
getmacha.com Macha
interface AI and machine learning AI agents and MCP +1 US 53 ✓ ✓
erowave.com Erowave.com
informational Community 51 ✓ ✓
genhealth.ai GenHealth.ai
interface Healthcare AI and machine learning +1 US 51 ✓ ✓
airfleet.co Airfleet
documentation AI and machine learning Marketing 50 ✓ ✓
fdb4.io FDB4
interface AI agents and MCP APIs and SDKs +1 50 ✓ ✓
stablediffusion.fr Stable Diffusion AI online
documentation AI and machine learning FR 50 ✓ ✓
0din.ai 0DIN
documentation AI and machine learning Cybersecurity 48 ✓
cultofweird.com Cult of Weird
informational Marketing Media US 48 ✓
tripleminds.co Triple Minds
documentation Automotive Ecommerce +1 48 ✓
ascend.de Ascend GmbH
interface Ecommerce Telecom DE 47 ✓
kaivalinfotech.com Kaival Infotech
informational Ecommerce Marketing IN 47 ✓
your-erotic-stories.com Your Erotic Stories
informational Marketing CZ 47 ✓
majestic.com Majestic
interface Marketing Data and analytics GB 64 ✓
xstory-fr.com XStory
informational Media FR 63 ✓ ✓
reed.co.uk llms.txt for Reed.co.uk
documentation Recruiting and careers GB 57 ✓
lightpanda.io Lightpanda
documentation Developer tools AI agents and MCP US 55 ✓ ✓
michaelzingraf.com Michael Zingraf Real Estate
documentation Real estate Local business FR 53 ✓ ✓
thechinajourney.com The China Journey
documentation Marketing US 53 ✓ ✓
gaeta.it Gaeta.it
informational Media IT 52 ✓ ✓
proxyrotator.com Proxy Rotator
transactional Infrastructure Developer tools US 52 ✓ ✓
bloomberry.com Bloomberry
interface Marketing Data and analytics +2 51 ✓
simplescraper.io Simplescraper
interface AI agents and MCP APIs and SDKs +1 51 ✓ ✓
skinory.io Skinory
documentation Gaming 50 ✓ ✓
amlcompany.com AML Company
documentation Real estate Legal +2 47 ✓ ✓
depinprojectslist.com DeFin Projects
informational Marketing Crypto and Web3 FR 47 ✓