fandangoseo.com FandangoSEO: Website crawler \& SEO Log Analyzer
documentation Marketing IE 46 ✓
crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools 50 ✓ ✓
spotibo.com Spotibo is not a web spider. It is a web dog! drumbigoo
transactional Marketing Developer tools 46 ✓
pr-cy.io pr-cy.io — SEO Tools & Website Analysis
informational Marketing AI and machine learning +1 RU 51 ✓
konabayev.com Konabayev — B2B Marketing, Browser Extensions, Apify Actors
transactional Marketing AI and machine learning 51 ✓ ✓
aerosend.io Aerosend
transactional Marketing Communications 50 ✓ ✓
octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
interface Data and analytics US 59 ✓
jump.bg llms.txt for Jump.BG
transactional Infrastructure BG 50 ✓ ✓
hasdata.com HasData
interface APIs and SDKs Data and analytics 51 ✓
tech-now.io TechNow (tech-now.io)
documentation 47 ✓
honour.app Honour.app
informational Recruiting and careers Education IN 46 ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
potensic.com Potensic Official Website
transactional Consumer electronics US 50 ✓
piperic.com Piperic
documentation Ecommerce Data and analytics FI 48 ✓
angeo.dev angeo.dev
documentation Ecommerce AI agents and MCP FR 45 ✓ ✓
specification.website The Website Specification
interface Documentation and reference 52 ✓ ✓
activesearchresults.com Active Search Results
documentation Marketing US 51 ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 50 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
igears.net iGears Technology Limited
documentation AI and machine learning Cybersecurity 47 ✓
tstprep.com TST Prep
transactional Education 51 ✓ ✓
thriveweb.com.au Thrive Digital Web Design Gold Coast: Websites that grow with your business ⋆ Gold Coast Web Design ⋆ Wordpress Support \& Hosting ⋆ Coding \& Development ⋆ Based in Burleigh Heads, Gold Coast, Austra
informational Design Local business AU 48 ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 56 ✓ ✓
smytten.com Smytten
transactional Ecommerce Fashion and beauty IN 57 ✓
onwebchat.com onWebChat
documentation AI and machine learning Communications DE 49 ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
okhten.com Artem Okhten Official Website
documentation AI and machine learning US 52 ✓ ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 54 ✓
sanyglobal.com SANY Group | Global Website
documentation Industry and manufacturing Construction and architecture FR 58 ✓
chinapp.com 抓取间隔
informational CN 49 ✓
powerthesaurus.org Power Thesaurus
documentation Education Documentation and reference 54 ✓
supermemory.ai supermemory
interface AI and machine learning AI agents and MCP +1 55 ✓ ✓
verifieddr.com VerifiedDR
interface AI and machine learning Marketing +1 50 ✓ ✓
zenmedia.com Zen Media: B2B Marketing and PR Agency for B2B Brands
documentation Marketing Media 48 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
svenkersten.nl Sven Kersten
documentation Marketing NL 46 ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
evisio.co Evisio
documentation Marketing Productivity +1 US 47 ✓
laserliner.com Laserliner - AI Crawler Index
documentation AI and machine learning Consumer electronics DE 48 ✓ ✓
seba.health Seba
interface Documentation and reference Science and research 45 ✓ ✓
crawlbase.com Crawlbase
interface APIs and SDKs Data and analytics 60 ✓ ✓
levamo.com Levamo
documentation Education Infrastructure 51 ✓ ✓
maxburst.com Web Design Company & SEO Marketing Agency in NYC & Long Island
informational Marketing Design US 55 ✓ ✓
armature.tech Armature
transactional AI agents and MCP Marketing US 48 ✓
searchenginejournal.com Search Engine Journal: The AI Search \& SEO News That Matters
informational Marketing AI and machine learning +1 GB 68 ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs 49 ✓
rankai.ai RankAI
documentation Marketing Business software US 49 ✓
stork.ai Stork.ai
documentation AI and machine learning Media US 55 ✓ ✓
paradyz.com Ceramika Paradyz
informational Industry and manufacturing Construction and architecture PL 51 ✓
doetsreizen.nl Doets Reizen
transactional Travel NL 50 ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
janbaskdigitaldesign.com JanBask Digital Design — Web Design, Web Development & Digital Marketing Agency
documentation Developer tools Marketing +1 52 ✓ ✓
localseoguide.com Local SEO Guide
informational Marketing Ecommerce US 52 ✓
georgianholidays.com Georgian Holidays
transactional Travel 47 ✓
nextbuzz.nl Next Buzz
documentation Infrastructure NL 52 ✓
eliterooter.com Elite Rooter
transactional Local business US 46 ✓
built2winweb.com BuiltToWinWeb
interface Design Marketing +5 52 ✓ ✓
simplescraper.io Simplescraper
interface AI agents and MCP APIs and SDKs +1 51 ✓ ✓
easymusic.ai EasyMusic.AI
transactional AI and machine learning Entertainment and streaming 49 ✓
homefirstindia.com Home First Finance Company India Limited
documentation Finance and payments US 50 ✓
koreatimes.co.kr The Korea Times
documentation Industry and manufacturing Media KR 61 ✓
appbasis.com App Basis Inc
documentation Developer tools Ecommerce +2 SG 52 ✓ ✓
xovi.de Xovi.de
informational Marketing DE 50 ✓
scraperapi.com ScraperAPI
interface AI and machine learning APIs and SDKs +2 US 53 ✓
evertune.ai Evertune
documentation Marketing AI and machine learning 48 ✓
yourgpt.ai YourGPT - The Complete AI Agent Platform
interface AI and machine learning 57 ✓ ✓
siteor.com Siteor — website builder and CMS
transactional Design Ecommerce +1 IE 49 ✓
builtwith.com BuiltWith
informational Marketing Data and analytics CA 63 ✓ ✓
itagroup.com ITA Group
informational Data and analytics Marketing US 51 ✓
newshosting.com llms.txt for https://www.newshosting.com
informational US 55 ✓
denverwebsitedesigns.com Denver Website Designs
documentation Ecommerce Marketing +3 CA 46 ✓
seomator.com SEOmator
documentation Marketing Data and analytics US 57 ✓ ✓
igears.com.hk iGears Technology Limited
documentation Marketing Business software +1 HK 46 ✓
sunbeltrentals.com Sunbelt Rentals LLMs.txt
documentation 58 ✓
thetechnovate.com Technovate
documentation Developer tools Business software 46 ✓
forge12.com Forge12 Interactive GmbH
documentation Business software AI and machine learning 51 ✓
365i.co.uk 365i, llms.txt
transactional Infrastructure GB 48 ✓
a-premium.com A-Premium
transactional Automotive Ecommerce 59 ✓
ifpm.institute ifpm.institute WordPress
informational DE 45 ✓
writesonic.com Writesonic
documentation Marketing AI and machine learning 64 ✓
knmasters.com KNmasters
documentation Marketing 45 ✓
designmemarketing.com Designme Marketing
documentation Marketing 46 ✓ ✓
ariseserver.com Arise Server
documentation Infrastructure DE 46 ✓
savvydealer.com Savvy Dealer
documentation Marketing Automotive US 51 ✓
elmaouid.dz جريدة الموعد اليومي (Elmaouid Daily Newspaper)
informational Media DZ 46 ✓
bbcincorp.com BBCIncorp
documentation Business software 49 ✓
advologysolution.com Advology Solution
informational Marketing 47 ✓
cuny.com C.U.N.Y. Digital
documentation Marketing Nonprofit and advocacy 52 ✓
truenorthsocial.com True North Social - Full-Service Digital Marketing Agency
informational Marketing Media 52 ✓
gaiaonline.com Gaia Online - AI Crawler Guidance
US 50 ✓
getpublii.com Publii Static CMS
documentation Developer tools Open source projects PL 52 ✓
firecrawl.dev Firecrawl
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
teenager365.to llms.txt
47 ✓
siteinspire.com Siteinspire
interface Design APIs and SDKs US 57 ✓ ✓
nohacks.co No Hacks - AI Agent Guidelines
informational AI and machine learning Media +1 51 ✓ ✓
zyte.com Zyte – The complete toolbox for web scraping, data extraction, and proxy infrastructure at scale. Trusted by thousands of companies worldwide.
interface APIs and SDKs Data and analytics 55 ✓
shkspr.mobi Shkspr.mobi Website
documentation Personal sites and blogs GB 56 ✓
dataprovider.com Dataprovider.com
interface Data and analytics AI agents and MCP US 51 ✓ ✓
webharvy.com WebHarvy
documentation Developer tools Data and analytics US 47 ✓
webbouwenaandekeukentafel.nl Webbouwen aan de keukentafel: Voor een stralende website
documentation Marketing NL 46 ✓
casrai.org CASRAI — AI Citation Guide
documentation Documentation and reference Science and research 53 ✓
sen7ry.com Sentry
46 ✓ ✓
errortrace.dev Sentry
US 57 ✓ ✓
vodafone.al Vodafone Albania
transactional Telecom AL 51 ✓
mailbutler.io Mailbutler
documentation Productivity DE 53 ✓
launchx.com LaunchX
documentation Education US 47 ✓
parsers.vc Parsers VC
interface AI and machine learning Data and analytics +2 50 ✓
names.co.uk llms.txt file for https://www.names.co.uk/
transactional Infrastructure GB 62 ✓ ✓
tryghed.dk TryghedsGruppen - AI Agent Information
informational AI and machine learning DK 46 ✓
parkingo.com ParkinGO
transactional Travel 57 ✓
knownagents.com Known Agents
interface AI and machine learning AI agents and MCP +1 US 52 ✓
merraine.com Merraine Group
informational Healthcare Arts and culture +2 46 ✓
as-pl.com AS-PL
documentation Automotive 58 ✓
wbtc.network WBTC
interface Crypto and Web3 APIs and SDKs 57 ✓ ✓
lsretail.com LS Retail
documentation Ecommerce Business software 52 ✓
agila.de AGILA Haustierversicherung
documentation Insurance DE 51 ✓
qiddiya.com Qiddiya Investment Company
documentation Finance and payments Entertainment and streaming +1 47 ✓
pdq.com PDQ
documentation Cybersecurity DevOps and observability +1 61 ✓
divaniapollonhotel.com Divani Apollon Palace & Thalasso
transactional Travel GB 46 ✓
okhtengroup.com Okhten Group LLC
documentation US 45 ✓ ✓
cvgorilla.com CVGorilla
transactional Marketing 48 ✓
pr-cy.ru pr-cy.ru — SEO-инструменты и анализ сайтов
documentation Marketing Data and analytics RU 57 ✓
cloudflare.com Cloudflare
interface AI and machine learning Cybersecurity +1 85 ✓ ✓
lpuonline.com LPU Online – llms.txt
US 47 ✓
casio-mea.com LLMS.txt for https://casio-mea.com/
transactional Ecommerce 45 ✓
chonburi33.com Cash Loans Online
informational Finance and payments 49 ✓
uzmovi.net UZMOVi
informational Entertainment and streaming Media 50 ✓
maximo-moto.com SECURTEX MOTOR S.L (t/a MaximoMoto)
transactional Ecommerce Automotive CA 47 ✓ ✓
spd.tech SPD Technology
documentation AI and machine learning Developer tools +1 51 ✓
llms-text.com llms.txt Generator — Free AI SEO & llms.txt Platform
interface Marketing AI and machine learning +1 45 ✓ ✓
kruizinga.nl Kruizinga.nl Nederland
transactional Industry and manufacturing Logistics and shipping NL 48 ✓
rocketpool.net Rocket Pool
interface Crypto and Web3 53 ✓
carpocratian.org The Carpocratian Church & School
documentation Nonprofit and advocacy US 49 ✓
seoworks.co.uk The SEO Works
documentation Marketing GB 52 ✓
szerek.hu SZEREK.HU
transactional Ecommerce HU 46 ✓
biotique.com llm.txt — Biotique AI Brand Reference File
informational Fashion and beauty CA 49 ✓ ✓
voog.com Voog CMS
documentation Design Ecommerce +3 NL 59 ✓
trainocate.co.jp トレノケート株式会社 LLMs.txtポリシー / Trainocate Corporation AI Content Policy (LLMs.txt) | This file defines AI crawling and training permissions for LLMs. Please read carefully. | LLMs.txt Policy for AI Crawler
informational JP 47 ✓
biltmorehotel.com The Biltmore Hotel
transactional Travel US 57 ✓ ✓
fibr.ai fibr.ai
documentation AI and machine learning Marketing 51 ✓ ✓
c3controls.com c3controls
transactional Industry and manufacturing 49 ✓
firsteigen.com FirstEigen
documentation AI and machine learning Data and analytics +1 US 47 ✓
multilipi.com MultiLipi
documentation Marketing HK 50 ✓ ✓
noc.org NOC.org
interface Infrastructure Cybersecurity GB 48 ✓
etailment.de etailment — Das Fachmagazin für E-Commerce und Handel
informational Ecommerce Media DE 52 ✓
ironvest.com IronVest
documentation Cybersecurity Finance and payments +1 51 ✓
sdlccorp.com SDLC Corp
informational Developer tools Gaming +2 51 ✓
aramgg.com aramgg.com
documentation Data and analytics Gaming 47 ✓
usclimatedata.com US Climate Data
documentation Science and research Data and analytics US 59 ✓
motphimc.com MotChill - MotPhim
informational Entertainment and streaming 56 ✓ ✓
motphimtvb.com MotChill - MotPhim
informational Entertainment and streaming Media 50 ✓ ✓
atlantadumpsters.com ADM Waste (atlantadumpsters.com)
documentation Local business US 47 ✓
iavaz.com Imagine Audio Video — LLMS Content Index
documentation AI and machine learning Media +2 46 ✓
gathern.co Gathern
interface Travel AI agents and MCP 50 ✓
optimerch.de Optimerch GmbH
documentation Marketing DE 47 ✓
nera.com NERA Economic Consulting
documentation Business software US 52 ✓
astrologyhub.com Astrology Hub
transactional Media Personal sites and blogs +1 46 ✓
autopro.com.vn AutoPro
documentation Automotive Media VN 54 ✓ ✓
bodhost.com bodHOST
transactional Infrastructure 45 ✓
tendata.com Tendata
documentation Marketing Data and analytics +1 51 ✓
scalperscompany.com Scalpers España
transactional Fashion and beauty Ecommerce CA 63 ✓ ✓
machalliance.org MACH Alliance
informational US 52 ✓
harbor365.com Harbor365
transactional Industry and manufacturing FR 47 ✓
getairefs.com Airefs
interface AI and machine learning Marketing 46 ✓
imperiascm.com Imperia | AI-native Supply Chain Planning Software
documentation Business software DE 50 ✓ ✓
fordham.edu LLMS.txt
documentation Education US 55 ✓
natu.care Natu.Care
informational Ecommerce US 50 ✓
pharmbills.com Pharmbills
informational Healthcare Business software US 47 ✓
vectoral.com Vectoral
documentation AI and machine learning Cybersecurity US 46 ✓
kaiserkraft.de kaiserkraft (KAISER+KRAFT GmbH) — LLM-Guidance (llms.txt)
transactional Ecommerce DE 51 ✓
socure.com Socure
documentation Cybersecurity Authentication and identity US 59 ✓
garmtech.com GARMTECH
transactional Infrastructure GB 58 ✓
mail.tm Mail.tm
interface Communications 53 ✓ ✓
knup.com Knup
informational Real estate US 48 ✓
eweka.nl llms.txt for https://www.eweka.nl/en
NL 51 ✓
coloradowebimpressions.com Colorado Web Impressions
documentation Marketing Local business PL 47 ✓ ✓
cinetree.nl Cinetree
informational NL 49 ✓ ✓
stonemania.ro STONEMANIA
transactional Ecommerce RO 50 ✓
teknorapex.com llms.txt for Teknor Apex
documentation Healthcare Industry and manufacturing US 49 ✓
speedkit.com Speed Kit
documentation Ecommerce Marketing 51 ✓ ✓
harbingergroup.com Harbinger Group | Advisory-Led Performance Enablement for Modern Workforce
documentation Media Business software US 47 ✓
stripchat.red stripchat.red
documentation 51 ✓ ✓
elehear.com ELEHEAR
transactional Healthcare Consumer electronics CA 54 ✓ ✓
gecdesigns.com GEC Designs
transactional Design US 47 ✓
finnoytravel.com Finnoy Travel
documentation Travel 46 ✓
lightsite.ai LightSite AI
documentation AI and machine learning Ecommerce +1 US 47 ✓
actowizsolutions.com Actowiz Solutions
documentation Data and analytics US 48 ✓
coldstonecreamery.com Cold Stone Creamery
transactional Food and dining US 60 ✓
dreadxp.com Dread XP
informational Gaming US 45 ✓
malekpourmie.net Malekpour MIE | موسسه ملکپور
transactional Education 49 ✓
beyondblue.org.au Beyond Blue (Australia)
informational Healthcare AU 58 ✓
mikgroup.ch MIK Group GmbH — SEO & GEO Agentur Schweiz
documentation Marketing CH 50 ✓ ✓
thaqalayn.com Thaqalayn
interface Documentation and reference 46 ✓
xoxostickers.com XOXO Stickers
documentation 47 ✓
drfranklipman.com Dr. Frank Lipman
documentation Education Healthcare US 48 ✓
chowtaifook.com Chow Tai Fook Jewellery (周大福珠寶)
transactional Ecommerce Fashion and beauty HK 57 ✓
tryhoverify.com Hoverify
transactional Marketing Developer tools US 51 ✓ ✓
alphaomegaconsulting.com Alpha Omega Consulting — LLMS Content Index
documentation AI and machine learning US 46 ✓
aps-bom.com Authorized Procurement Solutions (APS)
documentation Industry and manufacturing 47 ✓
atninfo.com ATNinfo - UAE B2B Business Directory
informational Industry and manufacturing Local business 51 ✓
blogseo.io BlogSEO
interface AI and machine learning Marketing +1 US 51 ✓ ✓
mxp.de Managed Service Provider & Digitalagentur Augsburg
documentation Infrastructure Business software DE 47 ✓
supernova.io Supernova.io
documentation AI and machine learning Design +2 US 51 ✓
xiaomiflashtool.com Xiaomi Flash Tool
documentation Developer tools 47 ✓
betenemy.bg Betenemy.bg
informational Gaming Entertainment and streaming BG 47 ✓
football-logos.cc football-logos.cc
documentation Sports and fitness 48 ✓
globallearnandearn.com Global Learn and Earn
transactional Ecommerce AT 48 ✓
usenetserver.com llms.txt for https://www.usenetserver.com/
US 48 ✓
bozzuto.com Bozzuto
informational Real estate Insurance 61 ✓ ✓
steamunlocked.org SteamUnlocked
documentation Gaming 61 ✓
prisync.com Prisync
documentation Ecommerce Data and analytics 53 ✓
columbiaasia.com Columbia Asia Hospitals
transactional Healthcare 46 ✓
seosherpa.com SEO SHERPA™
documentation Marketing 52 ✓
wow-drama.com llms.txt for wow-drama.com
45 ✓
nudgesecurity.com https://nudgesecurity.com llms.txt (Updated November 2025)
documentation Cybersecurity AI and machine learning +1 US 48 ✓
mediaslide.com Mediaslide
documentation Business software Marketing US 49 ✓
sencogoldanddiamonds.com LLMS.txt for Senco Gold & Diamonds
transactional Fashion and beauty Ecommerce IN 57 ✓
kerovit.com Kerovit by Kajaria
informational Industry and manufacturing Construction and architecture 50 ✓ ✓
hibu.com Hibu Inc.
documentation Marketing 60 ✓
series-fulls.com llms.txt for series-fulls.com
documentation 46 ✓
actualgroup.com Actual group
interface Recruiting and careers AI agents and MCP US 52 ✓ ✓
doctormgt.com DoctorMGT
documentation Healthcare US 48 ✓
webless.ai Webless
documentation AI and machine learning Business software US 46 ✓
siteway.de Siteway
documentation Marketing DE 52 ✓ ✓
boulderseomarketing.com Boulder SEO Marketing
documentation Marketing US 52 ✓ ✓
accops.com Accops
documentation Cybersecurity Productivity +1 IN 49 ✓
nutritioninsight.com Nutrition Insight
informational Media Food and dining US 59 ✓
morganphilips.com Morgan Philips Group - llms.txt
documentation Recruiting and careers NL 58 ✓
apify.com Apify
interface Developer tools AI agents and MCP +1 57 ✓
personalcareinsights.com Personal Care Insights
informational Media US 52 ✓
agent-ready.dev Agent Ready
interface AI and machine learning AI agents and MCP US 45 ✓ ✓
krowdbase.com Krowdbase
documentation Ecommerce Marketing +2 45 ✓
a-sales.co A-Sales
documentation Marketing 50 ✓ ✓
mytuner-radio.com myTuner Radio
documentation Media FR 61 ✓
deltavdigital.com DeltaV Digital
documentation Marketing 46 ✓
packaginginsights.com Packaging Insights
informational Media Industry and manufacturing US 59 ✓
brale.xyz Brale
interface Finance and payments Crypto and Web3 +1 51 ✓ ✓
freejobalert.com FreeJobAlert.com
informational Recruiting and careers Government and public services 63 ✓
cgcosmetic.com CG Cosmetic
documentation Healthcare Local business 46 ✓
surmado.com Surmado: Context for AI Models
documentation AI and machine learning Marketing US 46 ✓
farbentour.de Farbentour
documentation Marketing DE 46 ✓
zhaket.com Zhaket (ژاکت)
transactional Ecommerce IR 51 ✓
9mothers.com Starter
documentation US 45 ✓
loansjagat.com LoansJagat - India's 1st Loan Consolidation Marketplace
transactional Finance and payments 49 ✓
idcband.com ID&C (UK) – Custom Wristbands, Lanyards, Event Passes & RFID
transactional Events and conferences Ecommerce 47 ✓
bellicon.com bellicon
transactional Industry and manufacturing Sports and fitness +1 55 ✓ ✓
arzdigital.com Arzdigital
documentation Finance and payments Media IR 60 ✓
fionnapps.com FionnApps
transactional Productivity Consumer electronics US 49 ✓
uptimerobot.com UptimeRobot
interface DevOps and observability 65 ✓
rackspeed.de rack::SPEED
transactional Infrastructure DE 50 ✓