icade.fr Specific rules for AI crawlers
informational FR 47 ✓
cdnetworks.com 规则组 1: 针对所有爬虫的通用规则
documentation FR 48 ✓
gomechanic.in llms.txt for gomechanic.in
documentation IN 47 ✓
timly.com LLMs.txt - AI access policy
documentation — 45 ✓
4netplayers.com llms.txt for 4Netplayers
informational DE 47 ✓
iai.tv IAI.tv
informational GB 55 ✓
newshosting.com llms.txt for https://www.newshosting.com
informational US 55 ✓
piketa.com Piketa - AI Access Policy
documentation FI 49 ✓
canadianimmigrant.ca llms.txt – LLM guidance for canadianimmigrant.ca
informational CA 47 ✓
crowdstreet.com Allow AI engines that can enhance discoverability and citations.
informational NL 47 ✓
uncove.com Uncove
transactional Entertainment and streaming — 52 ✓
centinelanalytica.com Centinel Analytica
documentation AI and machine learning Cybersecurity DE 58 ✓ ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
tafelberg.co.za Tafelberg Furnishers
transactional AI and machine learning Ecommerce +1 ZA 46 ✓
krita-artists.org Krita Artists
informational Arts and culture Community DE 56 ✓
cai.com CAI.com
interface AI and machine learning AI agents and MCP +1 — 59 ✓
menafn.com Directories
documentation US 50 ✓
pornmate.com Porn Mate
informational Entertainment and streaming — 63 ✓ ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
impactguru.com ImpactGuru
documentation Nonprofit and advocacy Finance and payments — 58 ✓
mytuner-radio.com myTuner Radio
documentation Media FR 61 ✓
klimatron.ua Klimatron.ua — КлімаТрон / КлимаТрон
documentation UA 47 ✓
kerovit.com Kerovit by Kajaria
informational Industry and manufacturing Construction and architecture — 55 ✓ ✓
cchcpelink.com llms.txt - Guidelines for Large Language Models and AI Crawlers
informational AI and machine learning US 48 ✓
groundy.com Groundy
documentation AI and machine learning Developer tools — 51 ✓
purple.ai Purple
interface Marketing Healthcare +1 US 61 ✓ ✓
kickoffscore.com KickoffScore
documentation Sports and fitness — 55 ✓
vodafone.al Vodafone Albania
transactional Telecom AL 51 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
skidrowgamereloaded.co Skidrow Game Reloaded
documentation Gaming DE 47 ✓
nivona.com NIVONA
transactional Consumer electronics DE 53 ✓
gauntletai.com Gauntlet AI — AI Crawler Policy & Site Description
US 49 ✓ ✓
chinapp.com 抓取间隔
informational CN 49 ✓
thefrenchsimracer.com The French Simracer
transactional Ecommerce Consumer electronics CA 55 ✓ ✓
findip.net FindIP
interface Infrastructure Cybersecurity +1 — 59 ✓ ✓
gyde.ai Gyde
documentation Finance and payments Healthcare +3 US 51 ✓
piperic.com Piperic
documentation Ecommerce Data and analytics FI 53 ✓
nsin.ir nsin (انسین)
interface Infrastructure Cybersecurity IR 48 ✓
alleasyseo.com AllEasySEO
interface Marketing Developer tools — 51 ✓ ✓
tamburins.com TAMBURINS Official Online Store
transactional Ecommerce Fashion and beauty — 55 ✓
howthelightgetsin.org howthelightgetsin.org
documentation GB 51 ✓
schuhe-lueke.com Schuhe Lüke – online shop
transactional Ecommerce Fashion and beauty DE 51 ✓
moneypilot.com MoneyPilot
documentation Finance and payments US 52 ✓
seba.health Seba
interface Documentation and reference Science and research — 46 ✓ ✓
llmpulse.ai LLM Pulse
interface AI and machine learning Marketing — 55 ✓ ✓
startupnetworks.co.uk Startup Networks
documentation Marketing Community GB 55 ✓ ✓
claimavenue.com Claim Avenue
transactional — 52 ✓ ✓
ai-customs.tw 台灣入境AI放大鏡
informational AI and machine learning TW 45 ✓
jobiba.com Jobiba
interface Recruiting and careers — 56 ✓ ✓
coinex.com CoinEx
transactional Crypto and Web3 — 65 ✓ ✓
sleekplan.com Sleekplan
interface Business software Productivity DE 61 ✓ ✓
bellicon.com bellicon
transactional Industry and manufacturing Sports and fitness +1 — 60 ✓ ✓
maaseuduntulevaisuus.fi Maaseudun Tulevaisuus
documentation Media Agriculture FI 59 ✓
coveredca.com Covered California
documentation Healthcare Government and public services ZA 60 ✓
markethealth.com MarketHealth — Affiliate Network
documentation Healthcare Marketing CA 59 ✓ ✓
hibu.com Hibu Inc.
documentation Marketing US 60 ✓
thedrugclub.com TheDrugClub
transactional Healthcare US 60 ✓ ✓
sdlccorp.com SDLC Corp
informational Developer tools Gaming +2 — 58 ✓
uic.org UIC — International Union of Railways
documentation Industry and manufacturing US 60 ✓
alfabit.org AlfaBit
interface Crypto and Web3 APIs and SDKs +1 CN 59 ✓ ✓
bydfi.com BYDFi Exchange
transactional Crypto and Web3 JP 59 ✓
vaperanger.com Allow all known major AI and LLM bots to crawl the entire site
informational US 45 ✓
pangolin.net Pangolin
transactional Cybersecurity Infrastructure US 61 ✓ ✓
jvid.com JVID - 亞洲領先的在線視頻寫真平台, 海量高清寫真在線觀看
transactional Entertainment and streaming US 59 ✓ ✓
bigdug.co.uk LLMS permissions for bigdug.co.uk
informational GB 46 ✓
coinex.network CoinEx
transactional Crypto and Web3 — 58 ✓ ✓
mesotheliomalawyercenter.org Mesothelioma Lawyer Center
informational Legal US 58 ✓ ✓
koneviesti.fi Koneviesti
informational Automotive Media FI 55 ✓
gramodesky.cz GRAMODESKY.CZ
transactional Ecommerce CZ 57 ✓ ✓
inettools.net Inettools.net
informational DE 57 ✓
paraglidingsecondhand.com ParaglidingSecondHand.com
transactional Ecommerce — 57 ✓ ✓
nethris.com Access granted for GPTBot
documentation GB 48 ✓
arbaccessories.in ARB Accessories
transactional Ecommerce Industry and manufacturing IN 56 ✓ ✓
causeartist.com Causeartist
informational Media Nonprofit and advocacy — 56 ✓
coinex.zone CoinEx
transactional Crypto and Web3 Finance and payments — 56 ✓ ✓
gteek.com Gteek.com - Roadmap and Context for AI Agents and Large Language Models (LLMs)
documentation AI and machine learning BG 47 ✓
massivemonster.com Massive Monster
documentation Gaming — 56 ✓ ✓
tryhoverify.com Hoverify
transactional Marketing Developer tools US 56 ✓ ✓
gorbatech.com GorbaTech Solutions
documentation Infrastructure Marketing GR 55 ✓ ✓
natu.care Natu.Care
informational Ecommerce US 55 ✓
aksysgames.com Allow AI search assistants
documentation US 47 ✓
cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
perpfuse.com perpfuse
documentation Finance and payments Data and analytics RU 52 ✓ ✓
groovespin.com GROOVESPIN.COM
transactional Ecommerce — 54 ✓ ✓
palmate.ai Palmate AI
documentation AI and machine learning Business software DE 54 ✓
buffget.com Buffget
transactional Gaming Ecommerce HK 53 ✓
leboat.com llm.txt for Le Boat LLM Crawling Guide
US 50 ✓
calvado.com Calvado
transactional Ecommerce Home and garden FR 53 ✓
codeminer42.com Codeminer42
documentation Developer tools Business software — 53 ✓
groovespin.nl GROOVESPIN.NL
interface Ecommerce NL 53 ✓ ✓
groovespin.pl GROOVESPIN.PL
interface Ecommerce PL 53 ✓ ✓
avtoelon.uz Instructions for AI models crawling avtoelon.uz
informational AI and machine learning UZ 54 ✓ ✓
tosheroon.com Tosheroon
interface Data and analytics FI 53 ✓
flooringking.co.uk Flooring King
transactional Home and garden Ecommerce GB 52 ✓
groovespin.fr GROOVESPIN.FR
interface Ecommerce FR 52 ✓ ✓
idcband.com ID&C (UK) – Custom Wristbands, Lanyards, Event Passes & RFID
transactional Events and conferences Ecommerce — 52 ✓
movingto.com Movingto
documentation Legal US 52 ✓
1pc.co.il 1PC — מחשבים, סלולר ואלקטרוניקה בישראל
transactional Ecommerce Consumer electronics IL 51 ✓ ✓
edgeone.ai Tencent EdgeOne
interface Infrastructure Cybersecurity SG 57 ✓
cgcosmetic.com CG Cosmetic
documentation Healthcare Local business — 51 ✓
fabrento.com Fabrento
transactional Home and garden Ecommerce US 51 ✓
imamu.edu.sa جامعة الإمام محمد بن سعود الإسلامية
informational Education Science and research SA 51 ✓
agleader.com llm.txt for agleader.com
documentation AI and machine learning US 51 ✓
kunstnet.org Kunstnet
documentation DE 51 ✓
motabaqahtrading.net Motabaqah Trading
documentation Industry and manufacturing GB 51 ✓
nemeckyeshop.cz NěmeckýEshop.cz
transactional Ecommerce CZ 51 ✓
okfit.in Okfit
documentation Business software Sports and fitness IN 51 ✓
outspot.be Outspot
transactional Ecommerce Travel BE 51 ✓
jogos.com Jogos
informational Gaming — 50 ✓
motoaction.se MotoAction
transactional Automotive SE 50 ✓
yamakei-online.com AIクローラーによる要約と引用の許可設定
informational JP 49 ✓
casino-mundo.com llms.txt for https://casino-mundo.com/
informational Gaming — 45 ✓
llms-txt-generator.de llms.txt Generator
interface AI and machine learning Developer tools DE 45 ✓ ✓
granades.com Live Granades
informational Marketing Personal sites and blogs — 52 ✓
nohacks.co No Hacks - AI Agent Guidelines
informational AI and machine learning Media +1 — 51 ✓ ✓
stattimes.com llms.txt - AI Usage Policy for https://www.stattimes.com
documentation — 47 ✓
jewelershowcase.com LLM Crawling Guidance
informational AI and machine learning US 47 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
actiroute.com ActiROUTE
informational FR 46 ✓
bretton.com Bretton
documentation US 50 ✓
d3corp.com D3
informational US 48 ✓
intrinsec.com llms.txt — INTRINSEC (for AI/LLM crawlers)
— 46 ✓
cs2a.run No AI Scraping Allowed
informational RU 47 ✓
gaiaonline.com Gaia Online - AI Crawler Guidance
US 50 ✓
systematic-paris-region.org Builder KN
informational FR 47 ✓
healthmug.com Healthmug AI Agent Guidelines
documentation AI and machine learning IN 56 ✓
freehost.com.ua Разрешаем OpenAI GPTBot (ChatGPT)
documentation UA 50 ✓
biorender.com -----------------------------------------------------------------------------
documentation Science and research Design US 60 ✓
indiashelter.in https://www.indiashelter.in/llms.txt
documentation Finance and payments IN 52 ✓
be-terna.com BE-TERNA
informational Business software Data and analytics SI 51 ✓
licious.in llms.txt - LLM Usage Policy for this Repository
documentation IN 49 ✓
outspot.de Outspot
transactional Ecommerce DE 49 ✓
anete.lv Anete.lv
transactional Ecommerce LV 48 ✓
baitukas.lt Baitukas.lt
transactional Gaming Ecommerce LT 48 ✓
edinos.ru Единый Оператор Связи
documentation RU 48 ✓ ✓
nisonco.com NisonCo: SEO and AI marketing for any industry
documentation Marketing US 48 ✓
tntel.kz Твой Новый Телеком
documentation KZ 48 ✓ ✓
betpokies.com BetPokies
documentation Gaming — 47 ✓
dajanarodriguez.sk Dajana Rodriguez
transactional Fashion and beauty SK 47 ✓
gaugedb.com GaugeDB
documentation Gaming Data and analytics — 47 ✓
groovespin.de GROOVESPIN.DE
interface Consumer electronics Ecommerce DE 47 ✓ ✓
groovespin.hu GROOVESPIN.HU
interface Ecommerce HU 47 ✓ ✓
groovespin.it GROOVESPIN.IT
interface Ecommerce IT 47 ✓ ✓
groovespin.sk GROOVESPIN.SK
interface Ecommerce SK 47 ✓ ✓
mymoneyrain.com MyMoneyRain
documentation Gaming — 47 ✓
taktika.lv Taktika.lv - specializēts skrituļslidu un skrejriteņu veikals | est. 2002
transactional Ecommerce Sports and fitness LV 47 ✓
vakantio.de Vakantio
interface Personal sites and blogs Travel DE 47 ✓
groovespin.es GROOVESPIN.ES
interface Ecommerce ES 46 ✓ ✓
upvote.club Upvote.club
documentation Community — 46 ✓
3nv.io 3NV
documentation Infrastructure — 45 ✓ ✓
wow-drama.com llms.txt for wow-drama.com
— 45 ✓
liferaftlabs.com Liferaft llms.txt
documentation US 46 ✓
launchpadly.co Launchpadly
informational Marketing Community — 51 ✓ ✓
fionnapps.com FionnApps
transactional Productivity Consumer electronics US 52 ✓
equipt.com Equipt
transactional Industry and manufacturing — 49 ✓
sheshaayurveda.com Shesha Ayurveda
transactional Fashion and beauty CA 60 ✓ ✓
lyrsense.com Lyrsense
informational Entertainment and streaming Media RU 59 ✓
port.ac.uk University of Portsmouth
informational Education GB 59 ✓
360iresearch.com 360iResearch
informational Data and analytics Science and research — 58 ✓
garmtech.com GARMTECH
transactional Infrastructure GB 58 ✓
beinsure.com Beinsure
informational AI and machine learning Insurance +1 — 57 ✓
chowtaifook.com Chow Tai Fook Jewellery (周大福珠寶)
transactional Ecommerce Fashion and beauty HK 57 ✓
evkx.net EVKX.net
documentation Media Automotive NO 57 ✓
hanos.nl HANOS
transactional Food and dining Ecommerce NL 57 ✓
lsretail.com LS Retail
documentation Ecommerce Business software IS 57 ✓
onlinecasinorank.org OnlineCasinoRank
documentation Gaming Entertainment and streaming SE 57 ✓
chosenstore.in CHOSEN Store
transactional Fashion and beauty IN 56 ✓ ✓
ironvest.com IronVest
documentation Cybersecurity Finance and payments +1 — 56 ✓
meingpt.com MeinGPT
documentation AI and machine learning — 56 ✓ ✓
tendata.com Tendata
documentation Marketing Data and analytics +1 — 56 ✓
vusecurity.com VU Security
documentation Cybersecurity Authentication and identity US 56 ✓ ✓
mesaartscenter.com Mesa Arts Center
transactional AI and machine learning Arts and culture US 55 ✓
motorwatt.com MotorWatt
transactional Automotive Media US 55 ✓
potensic.com Potensic Official Website
transactional Consumer electronics US 55 ✓
rishtaywalay.com Rishtay Walay
informational Local business IN 55 ✓ ✓
studiestimer.com StudiesTimer
transactional Productivity Education US 55 ✓ ✓
vpsbenchmarks.com VPSBenchmarks
interface Infrastructure US 55 ✓
bbcincorp.com BBCIncorp
documentation Business software — 54 ✓
belindoc.com Belin Doc
interface AI and machine learning Productivity — 54 ✓ ✓
hilelectronic.com Highleap Electronic: PCB Fabrication, PCBA, and EMS Provider
transactional Industry and manufacturing — 54 ✓
malekpourmie.net Malekpour MIE | موسسه ملکپور
transactional Education — 54 ✓
tinyhouselistings.com Tiny House Listings
transactional Real estate — 54 ✓
typhur.com Typhur Robotics (https://www.typhur.com/)
transactional Home and garden Consumer electronics CA 54 ✓
yudiz.com Yudiz Solutions Ltd.
informational AI and machine learning IN 54 ✓
botghost.com BotGhost
documentation Developer tools No-code and automation — 53 ✓
dermstore.com Dermstore LLMs.txt
— 53 ✓
goproblems.com Go Problems
documentation Education Community US 53 ✓
melag.com MELAG Medizintechnik
transactional Industry and manufacturing Healthcare US 53 ✓
nationfiles.com NationFiles
documentation Cybersecurity DE 53 ✓ ✓
obyava.org Obyava
informational Real estate Ecommerce RU 53 ✓
slotsrank.com SlotsRank
documentation Entertainment and streaming SE 53 ✓
uweed.ch uWeed CBD Shop
transactional Ecommerce CH 53 ✓
bettingranker.com BettingRanker
documentation Media SE 52 ✓
cigarjournal.com Cigar Journal
informational Media — 52 ✓
cultbeauty.co.uk Cult Beauty LLMs.txt
GB 52 ✓
genesabz.com GeneSabz
transactional Ecommerce IR 52 ✓
georgianholidays.com Georgian Holidays
transactional Travel — 52 ✓
globalconference.ca Global Conference Alliance Inc.
transactional Events and conferences CA 52 ✓
maggiesorganics.com Maggie's Organics
transactional Fashion and beauty Ecommerce — 52 ✓
mbrenndoerfer.com Michael Brenndoerfer
documentation AI and machine learning Personal sites and blogs +1 — 52 ✓
mojogem.com MojoGem
transactional Gaming Ecommerce IR 52 ✓
pharmbills.com Pharmbills
informational Healthcare Business software US 52 ✓
solverify.net Solverify
interface APIs and SDKs — 52 ✓
supersonic.co.za Supersonic
transactional Infrastructure Telecom ZA 52 ✓ ✓
sysvine.com Sysvine Technologies
documentation IN 52 ✓
zero1gaming.com Zero1Gaming
informational Gaming Media NL 52 ✓
astrologyhub.com Astrology Hub
transactional Media Personal sites and blogs +1 — 51 ✓
ccanh.com Capitol Center for the Arts
transactional AI and machine learning Arts and culture US 51 ✓
cultbeauty.com Cult Beauty LLMs.txt
— 51 ✓
ennenergy.com ENN Energy
documentation Energy and utilities HK 51 ✓
football-logos.cc football-logos.cc
documentation Sports and fitness — 51 ✓
foreck.info FORECK.INFO
informational Media DE 51 ✓
idukki.io Idukki
documentation Marketing Ecommerce US 51 ✓ ✓
kayakapi.com Kayakapi Premium Caves - Cappadocia
transactional Travel — 51 ✓
magdek.com.ua Magdek
transactional Ecommerce Home and garden UA 51 ✓ ✓
musicquizly.com Music Quizly
documentation Media Gaming +1 NO 51 ✓
mystage.ro mySTAGE
transactional Arts and culture Events and conferences RO 51 ✓ ✓
nextechna.com Nextech Commercial HVAC and Refrigeration
informational Local business ZA 51 ✓
360growthlab.com 360 Growth Lab
transactional Marketing Business software US 50 ✓ ✓
bapacthousandoaks.com Bank of America Performing Arts Center Thousand Oaks
documentation Arts and culture AI and machine learning US 50 ✓
cupresents.org CU Presents
transactional AI and machine learning Arts and culture +1 US 50 ✓
ensemblestudiotheatre.org Ensemble Studio Theatre
transactional AI and machine learning Arts and culture US 50 ✓
flushingtownhall.org Flushing Town Hall
transactional AI and machine learning Arts and culture US 50 ✓
marintheatre.org Marin Theatre
transactional AI and machine learning Arts and culture US 50 ✓
nashvilleballet.com Nashville Ballet
transactional AI and machine learning Arts and culture US 50 ✓
razegrowth.com Raze
interface Design Marketing US 50 ✓ ✓
bidauto.online Bidauto
transactional Automotive — 47 ✓
eclipse.edu.au Eclipse Education
transactional Education AU 47 ✓
freevideogenerator.io Van Gogh Studio (freevideogenerator.io)
transactional AI and machine learning US 47 ✓
retouchmarket.com Retouch Market
transactional Design — 47 ✓
telaeris.com llms.txt for https://telaeris.com/
documentation — 47 ✓
xoxostickers.com XOXO Stickers
documentation — 47 ✓
advita.com Advita Ortho
informational Industry and manufacturing Healthcare US 46 ✓
hoodscore.com HoodScore
documentation Real estate — 46 ✓
interpreters.travel Interpreters Travel
transactional Travel — 46 ✓
kampatour.com Kampá Tour - Asia Travel Agency
transactional Travel — 46 ✓
mammoth-hire.co.uk Mammoth Hire
transactional Construction and architecture GB 46 ✓
pennutrition.com PEN Nutrition
documentation Healthcare Education US 46 ✓
szerek.hu SZEREK.HU
transactional Ecommerce HU 46 ✓
pushleads.com PushLeads | Asheville SEO Services: We’re Asheville SEO Experts at growing your business. Our SEO services boost your bottom line with powerful onsite and offsite SEO. Be visible and grow!
documentation Marketing — 49 ✓
host-tracker.com HostTracker
transactional DevOps and observability — 60 ✓
coursehunter.net CourseHunter
documentation Education NL 53 ✓
coursetrain.net CourseTrain
documentation Education NL 53 ✓
a-premium.com A-Premium
transactional Automotive Ecommerce — 59 ✓
superbet.ro Superbet Romania
documentation Gaming Entertainment and streaming RO 57 ✓
rjlsoftware.com RJL Software
documentation Productivity — 54 ✓
sme.sk SME.sk
informational Media SK 54 ✓
scoutbasketball.com Scoutbasketball
documentation Sports and fitness Data and analytics DE 52 ✓
hurra.com encoding: UTF-8
DE 49 ✓
betftw.com Bet For The Win (BetFTW)
documentation Sports and fitness — 46 ✓