| cdnetworks.com |
规则组 1: 针对所有爬虫的通用规则 |
documentation |
FR |
48 |
✓ |
|
|
| icade.fr |
Specific rules for AI crawlers |
informational |
FR |
47 |
✓ |
|
|
| nudify.online |
Nudify.Online |
transactional AI and machine learning |
— |
55 |
✓ |
|
|
| columbia.com.tr |
Columbia Türkiye |
transactional Ecommerce Fashion and beauty |
TR |
50 |
✓ |
|
|
| linens.com.tr |
Linens |
transactional Ecommerce Home and garden |
TR |
46 |
✓ |
|
|
| coursehunter.net |
CourseHunter |
documentation Education |
NL |
53 |
✓ |
|
|
| coursetrain.net |
CourseTrain |
documentation Education |
NL |
53 |
✓ |
|
|
| championturkiye.com |
Champion Türkiye |
interface Ecommerce Fashion and beauty |
— |
47 |
✓ |
|
|
| seemehired.com |
SeeMeHired (seemehired.com) |
documentation Recruiting and careers Business software |
IE |
50 |
✓ |
✓ |
|
| recordstoreday.co.uk |
Record Store Day UK |
informational Entertainment and streaming |
GB |
58 |
✓ |
|
|
| ltbjeans.com |
LTB Jeans |
transactional Ecommerce Fashion and beauty |
— |
57 |
✓ |
|
|
| theplancollection.com |
The Plan Collection |
transactional Construction and architecture |
US |
57 |
✓ |
|
|
| davidsbridal.com |
David's Bridal |
transactional Fashion and beauty |
CA |
56 |
✓ |
|
|
| verisure.co.uk |
Verisure UK |
transactional Cybersecurity |
GB |
56 |
✓ |
|
|
| yargici.com |
Yargıcı |
transactional Ecommerce Fashion and beauty |
TR |
56 |
✓ |
|
|
| foundrynet.com |
FoundryNet |
documentation AI and machine learning Infrastructure +1 |
— |
52 |
✓ |
✓ |
|
| themoosebay.com |
The Moose Bay |
transactional Ecommerce |
TR |
52 |
✓ |
|
|
| cinnagen.com |
CinnaGen - Leading Iranian Biotechnology Company |
documentation Healthcare |
IR |
46 |
✓ |
|
|
| mgretailer.com |
Sensitive paths |
|
— |
48 |
✓ |
|
|
| complex.com |
Complex |
informational Media |
US |
66 |
✓ |
|
|
| tac.com.tr |
Taç |
transactional Ecommerce |
TR |
46 |
✓ |
|
|
| menafn.com |
Directories |
documentation |
US |
50 |
✓ |
|
|
| jacksonandperkins.com |
Jackson & Perkins |
documentation |
— |
57 |
✓ |
|
|
| karaca.com.tr |
Çift Geyik Karaca |
transactional Ecommerce Fashion and beauty |
TR |
46 |
✓ |
|
|
| wearerealty.com |
WE ARE REALTY |
documentation Marketing Real estate |
— |
47 |
✓ |
|
|
| gofin.pl |
Zakaz trenowania, indeksowania i analizy |
informational |
PL |
46 |
✓ |
|
|
| brownbook.net |
Brownbook – Global Business Listing Database |
interface Data and analytics |
— |
57 |
✓ |
|
|
| klick.com |
Klick Health |
informational Healthcare |
— |
54 |
✓ |
|
|
| hoopr.ai |
Hoopr |
transactional Media |
IN |
46 |
✓ |
|
|
| bitraser.com |
BitRaser® llms.txt |
documentation Cybersecurity Business software |
— |
47 |
✓ |
|
|
| melag.com |
MELAG Medizintechnik |
transactional Industry and manufacturing Healthcare |
US |
48 |
✓ |
|
|
| knownagents.com |
Known Agents |
interface AI and machine learning AI agents and MCP +1 |
US |
57 |
✓ |
|
|
| icolistingonline.com |
ICO Listing Online |
documentation Crypto and Web3 |
US |
46 |
✓ |
|
|
| dibor.co.uk |
Dibor AI Usage Policy |
|
GB |
46 |
✓ |
✓ |
|
| msgameonline.com |
MSGameOnline |
informational Gaming |
IN |
46 |
✓ |
|
|
| pafi.hu |
pafi.hu |
interface Government and public services Data and analytics |
HU |
51 |
✓ |
✓ |
|
| alfabit.org |
AlfaBit |
interface Crypto and Web3 APIs and SDKs +1 |
CN |
59 |
✓ |
✓ |
|
| codeminer42.com |
Codeminer42 |
documentation Developer tools Business software |
— |
48 |
✓ |
|
|
| salesforce.com |
Salesforce Trailhead |
documentation Education Recruiting and careers |
BR |
69 |
✓ |
|
|
| karaokecrowd.com |
KaraokeCrowd |
documentation Community |
DE |
52 |
✓ |
✓ |
|
| eroguide.cz |
Eroguide |
informational |
CZ |
47 |
✓ |
|
|
| alelo.com.br |
========================= |
|
BR |
46 |
✓ |
|
|
| causeartist.com |
Causeartist |
informational Media Nonprofit and advocacy |
— |
56 |
✓ |
|
|
| gomechanic.in |
llms.txt for gomechanic.in |
documentation |
IN |
47 |
✓ |
|
|
| buffget.com |
Buffget |
transactional Gaming Ecommerce |
— |
53 |
✓ |
|
|
| cruisinggays.com |
Cruising Gays |
transactional Travel |
US |
52 |
✓ |
|
|
| holla.world |
========================================= |
|
— |
47 |
✓ |
|
|
| autowin.com |
AutoWin |
transactional Ecommerce Automotive |
CA |
58 |
✓ |
✓ |
|
| sloterop.nl |
http://sloterop.nl llms-full.txt |
transactional Ecommerce |
NL |
46 |
✓ |
|
|
| fabrento.com |
Fabrento |
transactional Home and garden Ecommerce |
US |
46 |
✓ |
|
|
| monkey.app |
========================================= |
|
— |
48 |
✓ |
|
|
| lashbrookdesigns.com |
Lashbrook Designs |
transactional Fashion and beauty |
— |
53 |
✓ |
|
|
| kassellabs.io |
Kassel Labs |
documentation Media Entertainment and streaming |
US |
47 |
✓ |
|
|
| kosokubus.com |
========================================================= |
|
JP |
45 |
✓ |
|
|
| theentrustgroup.com |
The Entrust Group |
documentation Finance and payments |
US |
52 |
✓ |
✓ |
|
| jobot.com |
Jobot |
informational Recruiting and careers |
— |
55 |
✓ |
|
|
| shl.com |
llms.txt for shl.com |
|
DE |
45 |
✓ |
|
|
| clientam.com |
========================================================= |
|
FR |
45 |
✓ |
|
|
| stb.com.mk |
Stopanska Banka AD — Skopje |
documentation Finance and payments |
MK |
54 |
✓ |
✓ |
|
| adbutler.com |
========================================================= |
|
US |
51 |
✓ |
✓ |
|
| abrites.com |
DiagNodes |
transactional Automotive |
BG |
54 |
✓ |
|
|
| viseer.co |
Viseer.co Developer Utility Platform |
documentation Developer tools |
TR |
47 |
✓ |
|
|
| 3ecpa.com.my |
3E Accounting Malaysia |
documentation Finance and payments Business software |
MY |
47 |
✓ |
|
|
| swissactivities.com |
Swiss Activities |
informational Travel |
CH |
56 |
✓ |
|
|
| html-color-codes.info |
======================================== |
|
US |
45 |
✓ |
|
|
| candelamedical.com |
============================================================ |
|
US |
49 |
✓ |
|
|
| tosheroon.com |
Tosheroon |
interface Data and analytics |
FI |
53 |
✓ |
|
|
| egypttoursclub.com |
Egypt Tours Club |
transactional Travel |
GB |
46 |
✓ |
|
|
| aao.org |
============================================================ |
|
— |
52 |
✓ |
|
|
| periskope.app |
Periskope |
interface AI and machine learning Communications +1 |
NL |
52 |
✓ |
|
|
| happyfares.in |
HappyFares — India's Flight Booking Platform |
interface Travel |
IN |
52 |
✓ |
✓ |
|
| bigideasdb.com |
BigIdeasDB — llms.txt |
documentation AI and machine learning Data and analytics |
US |
45 |
✓ |
✓ |
|
| chatwoot.com |
Chatwoot: AI-Powered, Open-Source Customer Support Platform - llms.txt |
interface Marketing AI and machine learning +3 |
US |
60 |
✓ |
|
|
| minlove.biz |
AI Usage Policy |
informational AI and machine learning |
— |
45 |
✓ |
|
|
| built2winweb.com |
BuiltToWinWeb |
interface Design Marketing +5 |
— |
53 |
✓ |
✓ |
|
| nipponhomes.com |
Nipponhomes |
documentation Real estate Data and analytics |
— |
50 |
✓ |
|
|
| better-messages.com |
Better Messages - WordPress Private Messaging Plugin |
documentation Developer tools Communications |
FR |
56 |
✓ |
✓ |
|
| taxgpt.com |
TaxGPT |
documentation AI and machine learning Finance and payments +1 |
— |
55 |
✓ |
|
|
| restaurantguru.com |
Restaurant Guru |
transactional Food and dining |
US |
65 |
✓ |
|
|
| exxpress.at |
eXXpress |
documentation Media |
AT |
58 |
✓ |
✓ |
|
| crowdstreet.com |
Allow AI engines that can enhance discoverability and citations. |
informational |
NL |
47 |
✓ |
|
|
| sbicard.com |
===================================================== |
|
IN |
49 |
✓ |
|
|
| timly.com |
LLMs.txt - AI access policy |
documentation |
— |
45 |
✓ |
|
|
| tommiecopper.com |
Tommie Copper |
informational |
US |
51 |
✓ |
|
|
| 4netplayers.com |
llms.txt for 4Netplayers |
informational |
DE |
47 |
✓ |
|
|
| eltiempo.es |
===================================================================== |
|
ES |
48 |
✓ |
|
|
| efirbet.com |
Efirbet |
informational Gaming Entertainment and streaming |
— |
58 |
✓ |
|
|
| theweathernetwork.com |
===================================================================== |
|
CA |
51 |
✓ |
|
|
| 7alalcasino.com |
7AlalCasino |
informational |
— |
47 |
✓ |
|
|
| nexhealth.com |
Last updated: 2026-10-17 |
informational |
US |
49 |
✓ |
|
|
| novahomeloans.com |
NOVA Home Loans |
informational Finance and payments Real estate |
— |
49 |
✓ |
|
|
| manjaro.news |
nerdvpn.de and all subdomains |
informational AI and machine learning |
CA |
46 |
✓ |
|
|
| vertbaudet.fr |
FICHIER LLMS.TXT - CONTROLE D'ACCES AUX IA GENERATIVES (LLMs) |
|
FR |
48 |
✓ |
|
|
| aksysgames.com |
Allow AI search assistants |
documentation |
US |
47 |
✓ |
|
|
| opomarket.pl |
llms.txt guidelines for opomarket.pl |
transactional AI and machine learning Automotive +1 |
PL |
46 |
✓ |
|
|
| wardogs.com |
Wardogs |
interface |
US |
53 |
✓ |
|
|
| norax.de |
llms.txt – AI Usage & Content Guidance for norax.de (Norax Online-Baumarkt) |
transactional Construction and architecture Home and garden +1 |
DE |
46 |
✓ |
|
|
| casino.com |
LLMS.txt - Large Language Model Access Policy for Casino.com |
informational AI and machine learning Gaming |
— |
50 |
✓ |
|
|
| foodandfuel.co.uk |
Non Gamstop Sites |
|
GB |
55 |
✓ |
|
|
| newshosting.com |
llms.txt for https://www.newshosting.com |
informational |
US |
55 |
✓ |
|
|
| instaproofs.com |
instaproofs |
|
US |
52 |
✓ |
|
|
| tuna.am |
Tuna Документация |
documentation Developer tools Infrastructure |
RU |
56 |
✓ |
✓ |
|
| nerdvpn.de |
nerdvpn.de and all subdomains |
informational AI and machine learning |
DE |
45 |
✓ |
|
|
| agleader.com |
llm.txt for agleader.com |
documentation AI and machine learning |
US |
51 |
✓ |
|
|
| biorender.com |
----------------------------------------------------------------------------- |
documentation Science and research Design |
US |
60 |
✓ |
|
|
| nostrabet.com |
llms.txt - AI Usage & Content Policy for Nostrabet.com |
informational Media |
BG |
56 |
✓ |
|
|
| fragra.de |
Fragra.de |
transactional Ecommerce Marketing +1 |
DE |
46 |
✓ |
✓ |
|