fandangoseo.com FandangoSEO: Website crawler \& SEO Log Analyzer
documentation Marketing IE 51 ✓
crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools — 55 ✓ ✓
spotibo.com Spotibo is not a web spider. It is a web dog! drumbigoo
transactional Marketing Developer tools — 51 ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
pr-cy.io pr-cy.io — SEO Tools & Website Analysis
informational Marketing AI and machine learning +1 RU 51 ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 — 54 ✓
webcrawlerapi.com Web crawling and data extraction | Webcrawlerapi
interface Developer tools APIs and SDKs — 46 ✓
konabayev.com Konabayev — B2B Marketing, Browser Extensions, Apify Actors
transactional Marketing AI and machine learning — 54 ✓ ✓
aerosend.io Aerosend
transactional Marketing Communications — 50 ✓ ✓
gauntletai.com Gauntlet AI — AI Crawler Policy & Site Description
US 49 ✓ ✓
forumone.com Forum One
documentation Business software US 53 ✓
infomatic.cz INFOMATIC s.r.o. – Kontextový soubor pro AI agenty a crawlery
documentation AI and machine learning Business software CZ 46 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
hasdata.com HasData
interface APIs and SDKs Data and analytics — 56 ✓
coozmoo.com LLMs.txt for https://coozmoo.com/
documentation Marketing NL 47 ✓
tech-now.io TechNow (tech-now.io)
documentation DE 47 ✓
honour.app Honour.app
informational Recruiting and careers Education IN 49 ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 61 ✓ ✓
potensic.com Potensic Official Website
transactional Consumer electronics US 55 ✓
piperic.com Piperic
documentation Ecommerce Data and analytics FI 53 ✓
angeo.dev angeo.dev
documentation Ecommerce AI and machine learning FR 46 ✓ ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
crawlbase.com Crawlbase
interface Infrastructure AI agents and MCP +1 — 60 ✓ ✓
activesearchresults.com Active Search Results
documentation Marketing US 58 ✓
searchstax.com SearchStax
documentation Healthcare Education +3 US 57 ✓
specification.website The Website Specification
interface Documentation and reference — 53 ✓ ✓
igears.net iGears Technology Limited
documentation AI and machine learning Cybersecurity — 52 ✓
stileex.xyz Stileex
interface Data and analytics APIs and SDKs — 46 ✓
tstprep.com TST Prep
transactional Education — 56 ✓ ✓
octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
documentation Data and analytics No-code and automation US 59 ✓
thriveweb.com.au Thrive Digital Web Design Gold Coast: Websites that grow with your business ⋆ Gold Coast Web Design ⋆ Wordpress Support \& Hosting ⋆ Coding \& Development ⋆ Based in Burleigh Heads, Gold Coast, Austra
informational Design Local business AU 48 ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 55 ✓
lovedby.ai LovedByAI
transactional Marketing AI and machine learning — 54 ✓ ✓
smytten.com Smytten
transactional Ecommerce Fashion and beauty IN 57 ✓
pushleads.com PushLeads | Asheville SEO Services: We’re Asheville SEO Experts at growing your business. Our SEO services boost your bottom line with powerful onsite and offsite SEO. Be visible and grow!
documentation Marketing — 49 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
onwebchat.com onWebChat
documentation AI and machine learning Communications DE 54 ✓
okhten.com Artem Okhten Official Website
documentation AI and machine learning US 57 ✓ ✓
chinapp.com 抓取间隔
informational CN 49 ✓
sanyglobal.com SANY Group | Global Website
documentation Industry and manufacturing Construction and architecture FR 58 ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
igears.com.hk iGears Technology Limited
documentation Marketing Business software +1 HK 46 ✓
powerthesaurus.org Power Thesaurus
documentation Education Documentation and reference — 54 ✓
supermemory.ai supermemory
interface AI and machine learning AI agents and MCP +1 — 60 ✓ ✓
verifieddr.com VerifiedDR
interface AI and machine learning Marketing +1 — 51 ✓ ✓
askeygeek.com ASKEYGEEK.com
documentation AI and machine learning — 48 ✓
tisdigitech.com TIS
documentation Marketing IN 48 ✓
zenmedia.com Zen Media: B2B Marketing and PR Agency for B2B Brands
documentation Marketing Media — 53 ✓
svenkersten.nl Sven Kersten
documentation Marketing NL 46 ✓
gaiaonline.com Gaia Online - AI Crawler Guidance
US 50 ✓
haha750w.space Mailchimp - Complete Web Content Analysis
— 48 ✓
jump.bg llms.txt for Jump.BG
transactional Infrastructure BG 50 ✓ ✓
evisio.co Evisio
documentation Marketing Productivity +1 US 47 ✓
ottuhr.com OTTUHR
informational FR 45 ✓
seba.health Seba
interface Documentation and reference Science and research — 46 ✓ ✓
scraperapi.com ScraperAPI
interface AI and machine learning APIs and SDKs +2 US 58 ✓
laserliner.com Laserliner - AI Crawler Index
documentation AI and machine learning Consumer electronics DE 53 ✓ ✓
armature.tech Armature
transactional AI agents and MCP Marketing US 49 ✓
maxburst.com Web Design Company & SEO Marketing Agency in NYC & Long Island
informational Marketing Design US 60 ✓ ✓
levamo.com Levamo
documentation Education Infrastructure — 56 ✓ ✓
aegro.com.br Website Information for LLMs
BR 45 ✓
icade.fr Specific rules for AI crawlers
informational FR 47 ✓
searchenginejournal.com Search Engine Journal: The AI Search \& SEO News That Matters
informational Marketing AI and machine learning +1 GB 68 ✓
stork.ai Stork.ai
documentation AI and machine learning Media US 60 ✓ ✓
paradyz.com Ceramika Paradyz
informational Industry and manufacturing Construction and architecture PL 56 ✓
institutedfa.com LLM.txt - Website Content Structure
US 45 ✓
janbaskdigitaldesign.com JanBask Digital Design — Web Design, Web Development & Digital Marketing Agency
documentation Developer tools Marketing +1 — 57 ✓ ✓
doetsreizen.nl Doets Reizen
transactional Travel NL 50 ✓
cwnp.com LLM.txt - Website Content Structure
US 47 ✓
localseoguide.com Local SEO Guide
informational Marketing Ecommerce US 57 ✓
georgianholidays.com Georgian Holidays
transactional Travel — 52 ✓
eliterooter.com Elite Rooter
transactional Local business US 51 ✓
maxcontact.com LLM.txt - Website Content Structure
US 46 ✓
youapply.com YouApply
documentation Education — 47 ✓
coopersurgical.com LLM.txt - Website Content Structure
US 48 ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
built2winweb.com BuiltToWinWeb
interface Design Marketing +5 US 53 ✓ ✓
appbasis.com App Basis Inc
documentation Developer tools Ecommerce +2 SG 57 ✓ ✓
homefirstindia.com Home First Finance Company India Limited
documentation Finance and payments US 55 ✓
clarkewillmott.com Clarke Willmott LLP
informational GB 46 ✓
koreatimes.co.kr The Korea Times
documentation Industry and manufacturing Media KR 61 ✓
evertune.ai Evertune
documentation Marketing AI and machine learning — 53 ✓
roommaster.com LLM.txt - Website Content Structure
US 46 ✓
cuny.com C.U.N.Y. Digital
documentation Marketing Nonprofit and advocacy — 57 ✓
xovi.de Xovi.de
informational Marketing DE 50 ✓
siteor.com Siteor — website builder and CMS
transactional Design Ecommerce +1 IE 54 ✓
builtwith.com BuiltWith
informational Marketing Data and analytics CA 63 ✓ ✓
dash.app LLM.txt - Website Content Structure
US 48 ✓
yourgpt.ai YourGPT - The Complete AI Agent Platform
interface AI and machine learning — 57 ✓ ✓
itagroup.com ITA Group
informational Data and analytics Marketing US 56 ✓
shifter.io Shifter
interface Infrastructure APIs and SDKs FR 56 ✓ ✓
scanmalware.com ScanMalware.com
interface Cybersecurity — 53 ✓
thetechnovate.com Technovate
documentation Developer tools Business software — 51 ✓
easymusic.ai EasyMusic.AI
transactional AI and machine learning Media GB 54 ✓
denverwebsitedesigns.com Denver Website Designs
documentation Ecommerce Marketing +3 CA 51 ✓
365i.co.uk 365i, llms.txt
transactional Infrastructure GB 53 ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs LV 49 ✓
a-premium.com A-Premium
transactional Automotive Ecommerce — 59 ✓
sunbeltrentals.com Sunbelt Rentals LLMs.txt
documentation US 58 ✓
wpcu.coop llms.txt for www.wpcu.coop
— 46 ✓
writesonic.com Writesonic
documentation Marketing AI and machine learning US 64 ✓
simplescraper.io Simplescraper
interface AI agents and MCP APIs and SDKs +1 — 51 ✓ ✓
91cg1.com LLMs.txt for www.91cg1.com
— 48 ✓
vistancenetworks.com Vistance Networks
informational SE 51 ✓
elmaouid.dz جريدة الموعد اليومي (Elmaouid Daily Newspaper)
informational Media DZ 46 ✓
newshosting.com llms.txt for https://www.newshosting.com
informational US 55 ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 56 ✓ ✓
bbcincorp.com BBCIncorp
documentation Business software — 54 ✓
getpublii.com Publii Static CMS
documentation Developer tools Open source projects PL 59 ✓
truenorthsocial.com True North Social - Full-Service Digital Marketing Agency
informational Marketing Media US 59 ✓
teenager365.to llms.txt
— 47 ✓
vodafone.al Vodafone Albania
transactional Telecom AL 51 ✓
casrai.org CASRAI — AI Citation Guide
documentation Documentation and reference Science and research — 58 ✓
launchx.com LaunchX
documentation Education US 52 ✓
names.co.uk llms.txt file for https://www.names.co.uk/
transactional Infrastructure GB 62 ✓ ✓
parsers.vc Parsers VC
interface AI and machine learning Data and analytics +2 — 50 ✓
poppinspayroll.com Title: LLMs Access Policy for PoppinsPayroll.com
informational AI and machine learning US 47 ✓
knownagents.com Known Agents
interface AI and machine learning Marketing +2 US 57 ✓
parkingo.com ParkinGO
transactional Travel — 57 ✓
mailbutler.io Mailbutler
transactional Productivity Marketing DE 53 ✓
merraine.com Merraine Group
informational Healthcare Arts and culture +2 — 51 ✓
as-pl.com AS-PL
documentation Automotive PL 58 ✓
primewealth.co.in PrimeWealth
documentation Finance and payments IN 46 ✓
wbtc.network WBTC
interface Crypto and Web3 APIs and SDKs — 62 ✓ ✓
lsretail.com LS Retail
documentation Ecommerce Business software IS 57 ✓
agila.de AGILA Haustierversicherung
documentation Insurance DE 51 ✓
scott-equipment.com Scott Equipment
transactional Local business Industry and manufacturing — 46 ✓
tryghed.dk TryghedsGruppen - AI Agent Information
informational AI and machine learning DK 46 ✓
ehu.eus llms.txt for uik.eus (Cursos de Verano de la UPV/EHU)
ES 50 ✓
cactusjacksauto.com Cactus Jack's Auto
transactional Automotive US 49 ✓
lpuonline.com LPU Online – llms.txt
US 47 ✓
casio-mea.com LLMS.txt for https://casio-mea.com/
transactional Ecommerce — 50 ✓
cs2a.run No AI Scraping Allowed
informational RU 47 ✓
html-color-codes.info ========================================
US 45 ✓
nohacks.co No Hacks - AI Agent Guidelines
informational AI and machine learning Media +1 — 51 ✓ ✓
pr-cy.ru pr-cy.ru — SEO-инструменты и анализ сайтов
documentation Marketing Data and analytics RU 57 ✓
qiddiya.com Qiddiya Investment Company
documentation Finance and payments Entertainment and streaming +1 — 52 ✓
pdq.com PDQ
documentation Cybersecurity DevOps and observability +1 US 61 ✓
divaniapollonhotel.com Divani Apollon Palace & Thalasso
transactional Travel GB 51 ✓
clippingpathoutsource.com Clipping Path Outsource
transactional Design Ecommerce — 46 ✓
advologysolution.com Advology Solution
informational Marketing IN 52 ✓
bikiniberlin.de bikiniberlin.de
informational Arts and culture DE 51 ✓
chonburi33.com Cash Loans Online
informational Finance and payments — 50 ✓
designmemarketing.com Designme Marketing
documentation Marketing — 51 ✓ ✓
thinkgeo.com ThinkGeo
interface Developer tools AI agents and MCP +1 US 54 ✓
ariseserver.com Arise Server
documentation Infrastructure DE 51 ✓
savvydealer.com Savvy Dealer
documentation Marketing Automotive US 56 ✓
razegrowth.com Raze
interface Design Marketing US 50 ✓ ✓
uzmovi.net UZMOVi
informational Entertainment and streaming Media — 50 ✓
cvgorilla.com CVGorilla
transactional Marketing — 49 ✓
maximo-moto.com SECURTEX MOTOR S.L (t/a MaximoMoto)
transactional Ecommerce Automotive CA 52 ✓ ✓
jetoctopus.com Technical SEO Platform for Google & AI Search
documentation Marketing AI and machine learning EE 53 ✓
okhtengroup.com Okhten Group LLC
documentation US 46 ✓ ✓
llms-text.com llms.txt Generator — Free AI SEO & llms.txt Platform
interface Marketing AI and machine learning +1 — 48 ✓ ✓
speediance.com Speediance
transactional Consumer electronics CA 61 ✓ ✓
kruizinga.nl Kruizinga.nl Nederland
transactional Industry and manufacturing Logistics and shipping NL 48 ✓
rocketpool.net Rocket Pool
interface Crypto and Web3 — 58 ✓
carpocratian.org The Carpocratian Church & School
documentation Nonprofit and advocacy US 52 ✓
digitalapplied.com Digital Applied
documentation AI and machine learning Ecommerce +1 US 61 ✓
seoworks.co.uk The SEO Works
documentation Marketing GB 57 ✓
joven-electric.com JOVEN Malaysia
informational MY 47 ✓
leocwlau.com leocwlau
informational AI and machine learning Personal sites and blogs FR 46 ✓
szerek.hu SZEREK.HU
transactional Ecommerce HU 46 ✓
biotique.com llm.txt — Biotique AI Brand Reference File
informational Fashion and beauty CA 54 ✓ ✓
voog.com Voog CMS
documentation Design Ecommerce +3 NL 59 ✓
ezstats.io ezStats
documentation AI and machine learning Data and analytics +1 — 45 ✓ ✓
rozz.site ROZZ - B2B SaaS AI Search Visibility Platform
documentation AI and machine learning Marketing — 50 ✓ ✓
trainocate.co.jp トレノケート株式会社 LLMs.txtポリシー / Trainocate Corporation AI Content Policy (LLMs.txt) | This file defines AI crawling and training permissions for LLMs. Please read carefully. | LLMs.txt Policy for AI Crawler
informational JP 47 ✓
biltmorehotel.com The Biltmore Hotel
transactional Travel US 62 ✓ ✓
fibr.ai fibr.ai
documentation AI and machine learning Marketing — 56 ✓ ✓
c3controls.com c3controls
transactional Industry and manufacturing US 54 ✓
firsteigen.com FirstEigen
documentation AI and machine learning Data and analytics +1 US 52 ✓
konery.com Konery
documentation ES 48 ✓ ✓
sdlccorp.com SDLC Corp
informational Developer tools Gaming +2 — 58 ✓
ironvest.com IronVest
documentation Cybersecurity Finance and payments +1 — 56 ✓
etailment.de etailment — Das Fachmagazin für E-Commerce und Handel
informational Ecommerce Media DE 52 ✓
aramgg.com aramgg.com
documentation Data and analytics Gaming — 48 ✓
softway.net Softway - Web Professionals
documentation Marketing Ecommerce — 46 ✓
axcessnews.com AxcessNews
informational Media — 52 ✓
klimatron.ua Klimatron.ua — КлімаТрон / КлимаТрон
documentation UA 47 ✓
usclimatedata.com US Climate Data
documentation Science and research Data and analytics US 59 ✓
motphimc.com MotChill - MotPhim
informational Entertainment and streaming — 59 ✓ ✓
mediaslide.com Mediaslide
documentation Business software Recruiting and careers US 54 ✓
atlantadumpsters.com ADM Waste (atlantadumpsters.com)
documentation Local business US 52 ✓
iavaz.com Imagine Audio Video — LLMS Content Index
documentation AI and machine learning Media +2 — 51 ✓
motphimtvb.com MotChill - MotPhim
informational Entertainment and streaming Media — 51 ✓ ✓
mypremierpain.com Premier Pain Centers
informational Healthcare US 46 ✓
firecrawl.dev Firecrawl
interface AI and machine learning APIs and SDKs +2 US 65 ✓ ✓
gathern.co Gathern
interface Travel AI agents and MCP — 50 ✓
optimerch.de Optimerch GmbH
documentation Marketing DE 47 ✓
nera.com NERA Economic Consulting
documentation Business software US 57 ✓
astrologyhub.com Astrology Hub
transactional Media Personal sites and blogs +1 — 51 ✓
webbroi.com WebbROI
documentation Developer tools Marketing — 46 ✓
serveravatar.com ServerAvatar
documentation Infrastructure IN 63 ✓ ✓
aumcore.com Aumcore
documentation Developer tools Marketing US 53 ✓
autopro.com.vn AutoPro
documentation Automotive Media VN 54 ✓ ✓
bodhost.com bodHOST
transactional Infrastructure — 50 ✓
tendata.com Tendata
documentation Marketing Data and analytics +1 — 56 ✓
context.dev Context.dev
interface AI and machine learning AI agents and MCP +2 US 60 ✓ ✓
machalliance.org MACH Alliance
informational US 57 ✓
harbor365.com Harbor365
transactional Industry and manufacturing FR 52 ✓
getairefs.com Airefs
interface AI and machine learning Marketing — 49 ✓
imperiascm.com Imperia | AI-native Supply Chain Planning Software
documentation Business software DE 55 ✓ ✓
fordham.edu LLMS.txt
documentation Education US 55 ✓
pharmbills.com Pharmbills
informational Healthcare Business software US 52 ✓
vectoral.com Vectoral
documentation AI and machine learning Cybersecurity US 51 ✓
kaiserkraft.de kaiserkraft (KAISER+KRAFT GmbH) — LLM-Guidance (llms.txt)
transactional Ecommerce DE 51 ✓
digital.marketing Digital.Marketing
documentation Marketing — 46 ✓
socure.com Socure
documentation Cybersecurity Authentication and identity US 59 ✓
garmtech.com GARMTECH
transactional Infrastructure GB 58 ✓
mail.tm Mail.tm
interface Communications — 53 ✓ ✓
knup.com Knup
informational Real estate US 53 ✓
coloradowebimpressions.com Colorado Web Impressions
documentation Marketing Local business PL 52 ✓ ✓
eweka.nl llms.txt for https://www.eweka.nl/en
NL 51 ✓
growthx.ai Docs
documentation AI and machine learning Marketing +1 US 47 ✓ ✓
cinetree.nl Cinetree
informational NL 49 ✓ ✓
stonemania.ro STONEMANIA
transactional Ecommerce RO 50 ✓
timmermansmedia.nl Timmermans Media
documentation Marketing NL 45 ✓ ✓
teknorapex.com llms.txt for Teknor Apex
documentation Healthcare Industry and manufacturing US 54 ✓
incytecares.com llms.txt
documentation Healthcare — 46 ✓
speedkit.com Speed Kit
documentation Ecommerce Marketing — 56 ✓ ✓
harbingergroup.com Harbinger Group | Advisory-Led Performance Enablement for Modern Workforce
documentation Media Business software US 52 ✓
aiiimpact.com AIIImpact
informational Marketing FR 48 ✓
stripchat.red stripchat.red
documentation — 56 ✓ ✓
elehear.com ELEHEAR
transactional Healthcare Consumer electronics CA 59 ✓ ✓
gecdesigns.com GEC Designs
transactional Design US 52 ✓
finnoytravel.com Finnoy Travel
documentation Travel — 51 ✓
saigon.digital Saigon Digital
documentation Marketing US 46 ✓
multilipi.com MultiLipi
documentation Marketing HK 55 ✓ ✓
lightsite.ai LightSite AI
documentation AI and machine learning Ecommerce +1 US 50 ✓
actowizsolutions.com Actowiz Solutions
documentation Data and analytics US 53 ✓
coldstonecreamery.com Cold Stone Creamery
transactional Food and dining US 60 ✓
dreadxp.com Dread XP
informational Gaming US 50 ✓
malekpourmie.net Malekpour MIE | موسسه ملکپور
transactional Education AT 54 ✓
beyondblue.org.au Beyond Blue (Australia)
informational Healthcare AU 58 ✓
thaqalayn.com Thaqalayn
interface Documentation and reference — 51 ✓
mikgroup.ch MIK Group GmbH — SEO & GEO Agentur Schweiz
documentation Marketing CH 50 ✓ ✓
xoxostickers.com XOXO Stickers
documentation — 47 ✓
drfranklipman.com Dr. Frank Lipman
documentation Education Healthcare US 53 ✓
chowtaifook.com Chow Tai Fook Jewellery (周大福珠寶)
transactional Ecommerce Fashion and beauty HK 57 ✓
tryhoverify.com Hoverify
transactional Marketing Developer tools US 56 ✓ ✓
alphaomegaconsulting.com Alpha Omega Consulting — LLMS Content Index
documentation AI and machine learning US 51 ✓
aps-bom.com Authorized Procurement Solutions (APS)
documentation Industry and manufacturing — 52 ✓
journity.com Journity
documentation Marketing — 49 ✓
atninfo.com ATNinfo - UAE B2B Business Directory
informational Industry and manufacturing Local business — 56 ✓
blogseo.io BlogSEO
interface AI and machine learning Marketing +1 US 51 ✓ ✓
mxp.de Managed Service Provider & Digitalagentur Augsburg
documentation Infrastructure Business software DE 47 ✓
natu.care Natu.Care
documentation Ecommerce US 59 ✓ ✓
xiaomiflashtool.com Xiaomi Flash Tool
documentation Developer tools — 52 ✓
supernova.io Supernova.io
documentation AI and machine learning Design +1 US 51 ✓