webcrawlerapi.com Web crawling and data extraction | Webcrawlerapi
interface Developer tools APIs and SDKs — 46 ✓
crawlbase.com Crawlbase
interface Infrastructure AI agents and MCP +1 — 60 ✓ ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
spotibo.com Spotibo is not a web spider. It is a web dog! drumbigoo
transactional Marketing Developer tools — 51 ✓
llms-txt-generator.de llms.txt Generator
interface AI and machine learning Developer tools DE 45 ✓ ✓
feedoptimise.com Feedoptimise
documentation Ecommerce Marketing +1 — 53 ✓
wonderchat.io Wonderchat
documentation AI and machine learning AI agents and MCP — 50 ✓
crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools — 55 ✓ ✓
pixsy.com Pixsy
transactional Legal Business software US 57 ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
oncrawl.com Oncrawl
interface Marketing Business software +1 FR 59 ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 — 54 ✓
catalystmarketingagency.com Catalyst Marketing Agency
informational Marketing US 47 ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 61 ✓ ✓
webstacks.com Webstacks
documentation Marketing US 57 ✓ ✓
promptcloud.com PromptCloud
documentation AI and machine learning Ecommerce +2 US 53 ✓
gardena.com GARDENA
documentation Home and garden NL 62 ✓
tisdigitech.com TIS
documentation Marketing IN 48 ✓
chatlab.com ChatLab
documentation AI and machine learning No-code and automation PL 56 ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
jetoctopus.com Technical SEO Platform for Google & AI Search
documentation Marketing AI and machine learning EE 53 ✓
steamunlocked.org SteamUnlocked
documentation Gaming — 61 ✓
chinapp.com 抓取间隔
informational CN 49 ✓
gauntletai.com Gauntlet AI — AI Crawler Policy & Site Description
US 49 ✓ ✓
grupobimbo.com Grupo Bimbo - LLM Context Gateway
documentation Food and dining US 54 ✓
octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
documentation Data and analytics No-code and automation US 59 ✓
tavily.com Tavily Docs
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
rankai.ai RankAI
documentation Marketing AI and machine learning US 54 ✓
ottuhr.com OTTUHR
informational FR 45 ✓
onwebchat.com onWebChat
documentation AI and machine learning Communications DE 54 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
semrush.com Semrush
interface Marketing US 69 ✓
nuxt.com Nuxt Docs
documentation Developer tools Open source projects +1 US 67 ✓ ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 55 ✓
getmacha.com Macha
interface AI and machine learning AI agents and MCP +1 US 58 ✓ ✓
fandangoseo.com FandangoSEO: Website crawler \& SEO Log Analyzer
documentation Marketing IE 51 ✓
bloomberry.com Bloomberry
interface Marketing Data and analytics +2 US 56 ✓
price2spy.com Price2Spy
documentation Ecommerce Data and analytics FI 56 ✓
pricerest.com Competitor Price Monitoring & Price Tracking Software
documentation Ecommerce Industry and manufacturing +1 FR 50 ✓ ✓
digitalapplied.com Digital Applied
documentation AI and machine learning Ecommerce +1 US 61 ✓
coleggwent.ac.uk Coleg Gwent
informational Education GB 49 ✓
lovedby.ai LovedByAI
transactional Marketing AI and machine learning — 54 ✓ ✓
tripleminds.co Triple Minds
documentation Automotive Ecommerce +1 — 48 ✓
jewelershowcase.com LLM Crawling Guidance
informational AI and machine learning US 47 ✓
mobilebettingparsi.com mobilebettingparsi.com
informational — 48 ✓
smartcustomer.com SmartCustomer
informational US 60 ✓
clarkewillmott.com Clarke Willmott LLP
informational GB 46 ✓
bazipasoor.com bazipasoor.com
informational — 47 ✓
axopar.com Axopar Boats
documentation Consumer electronics — 55 ✓
scraperapi.com ScraperAPI
interface AI and machine learning APIs and SDKs +2 US 58 ✓
advologysolution.com Advology Solution
informational Marketing IN 52 ✓
cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
bet303review.com bet303review.com
informational — 47 ✓
bloggerspassion.com BloggersPassion
informational Ecommerce Marketing +1 — 54 ✓
betbroreview.com betbroreview.com
informational — 47 ✓
sweetpproductions.com SweetP Productions
transactional Consumer electronics Productivity US 51 ✓
gteek.com Gteek.com - Roadmap and Context for AI Agents and Large Language Models (LLMs)
documentation AI and machine learning BG 47 ✓
herb.md HERB.MD
transactional Ecommerce Fashion and beauty — 47 ✓
evisio.co Evisio
documentation Marketing Productivity +1 US 47 ✓
matka-tips.org Matka-Tips.org
informational Community — 54 ✓
downloadhotbet.com downloadhotbet.com
informational — 47 ✓
icade.fr Specific rules for AI crawlers
informational FR 47 ✓
clickmasters.pk Clickmasters Digital Marketing Agency
informational Marketing Ecommerce PK 47 ✓
digitalacademy360.com https://www.digitalacademy360.com/ llms.txt
documentation AI and machine learning Marketing +1 US 47 ✓
joinhotbet.com joinhotbet.com
informational — 47 ✓
haha750w.space Mailchimp - Complete Web Content Analysis
— 48 ✓
searchenginejournal.com Search Engine Journal: The AI Search \& SEO News That Matters
informational Marketing AI and machine learning +1 GB 68 ✓
seomator.com SEOmator
documentation Marketing US 62 ✓ ✓
proginter.com Proginter
transactional Infrastructure IL 52 ✓ ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 56 ✓ ✓
michaelkors.co.nl michaelkors.co.nl
transactional Fashion and beauty Ecommerce NL 45 ✓
googiehost.com GoogieHost
transactional Infrastructure Design — 58 ✓
vistancenetworks.com Vistance Networks
informational SE 51 ✓
hosting.de hosting.de - llm.txt
transactional Infrastructure DE 52 ✓
simplescraper.io Simplescraper
interface AI agents and MCP APIs and SDKs +1 — 51 ✓ ✓
hyperbrowser.ai Hyperbrowser
interface AI and machine learning AI agents and MCP +2 US 51 ✓
kinlan.me Modern Web Development with Chrome
documentation Developer tools Personal sites and blogs US 48 ✓
techbooky.com TechBooky
informational Finance and payments Media IE 53 ✓
pogodnik.com Pogodnik — Ukrainian Weather Forecast Service
informational Media — 51 ✓
siteway.de Siteway
documentation Marketing DE 52 ✓ ✓
vaperanger.com Allow all known major AI and LLM bots to crawl the entire site
informational US 45 ✓
lsretail.com LS Retail
documentation Ecommerce Business software IS 57 ✓
wemonde.com wemonde
documentation Developer tools Marketing FR 47 ✓
brandclickx.com Brand ClickX
documentation Marketing FR 48 ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs LV 49 ✓
coozmoo.com LLMs.txt for https://coozmoo.com/
documentation Marketing NL 47 ✓
norsumediagroup.com Norsu Media Group
informational Marketing GB 48 ✓
agip.com Abu-Ghazaleh Intellectual Property (AGIP)
interface Legal JO 56 ✓ ✓
aegro.com.br Website Information for LLMs
BR 45 ✓
seospidy.in Seospidy Web Solution (SEOSpidy)
informational Marketing IN 51 ✓
4eck-media.de 4eck-media
interface Media AI and machine learning +3 DE 45 ✓ ✓
news9live.com News9Live - English News Website
informational Media IN 60 ✓
institutedfa.com LLM.txt - Website Content Structure
US 45 ✓
cwnp.com LLM.txt - Website Content Structure
US 47 ✓
maxcontact.com LLM.txt - Website Content Structure
US 46 ✓
socialwalls.com Social Walls by Taggbox
transactional Media Marketing +1 — 52 ✓
coopersurgical.com LLM.txt - Website Content Structure
US 48 ✓
appsecsanta.com AppSec Santa
informational Cybersecurity — 56 ✓ ✓
seozoom.com SEOZoom: The best software for Search Marketing: analyse, monitor and boost your website with over 40 tools
documentation Marketing IT 47 ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
eduyush.com Eduyush
transactional Education CA 58 ✓ ✓
cmarix.com CMARIX Infotech
informational Developer tools Business software US 57 ✓
doctutorials.com DocTutorials
transactional Education IN 52 ✓
toxsl.com Software and mobile app development company - ToXSL Technologies
documentation Developer tools Business software IN 52 ✓
mediana.kz Mediana
informational Infrastructure Business software KZ 48 ✓
tagembed.com tagembed.com llms.txt
transactional Marketing Business software — 59 ✓
lithionbattery.com Lithion Battery
documentation Industry and manufacturing US 51 ✓
sahistory.org.za South African History Online (SAHO)
documentation Education Science and research +1 ZA 52 ✓
constructor.university Constructor University
informational Education DE 60 ✓
orakel.com Orakel
transactional Industry and manufacturing Events and conferences +1 BE 50 ✓ ✓
thediscountcodes.co.uk theDiscountCodes
informational Ecommerce GB 48 ✓
entrancezone.com EntranceZone
documentation Education — 49 ✓
knmasters.com KNmasters
documentation Marketing TH 50 ✓
magnaquest.com Magnaquest
documentation Business software IN 54 ✓
purwell.com Purwell
transactional Ecommerce FR 51 ✓
corpedgroup.com www.corpedgroup.com
documentation Productivity Education US 51 ✓
findacrew.com Find a Crew™
documentation Recruiting and careers — 51 ✓
ooredoo.ps Ooredoo Palestine
transactional Telecom PS 56 ✓ ✓
southmelody.com SouthMelody
informational Media Entertainment and streaming — 45 ✓
anatolyfit.com Anatoly Fit
transactional Sports and fitness SE 52 ✓
prelearning.de PreLearning
documentation Education DE 45 ✓ ✓
muzline.ua Muzline — спеціалізується на на продажі та обслуговуванні музичних інструментів, звукового та світлового обладнання
transactional Local business Consumer electronics UA 52 ✓
drishtiias.com Drishti IAS
transactional Education — 62 ✓ ✓
nongsandungha.com Nông Sản Dũng Hà
documentation Food and dining VN 50 ✓
streameast.cool StreamEast
informational Media Sports and fitness NL 46 ✓
code-brew.com Code Brew Labs
informational AI and machine learning US 55 ✓
poppinspayroll.com Title: LLMs Access Policy for PoppinsPayroll.com
informational AI and machine learning US 47 ✓
finedinemenu.com FineDine – AI-Powered Restaurant Growth & Customer Experience Platform
transactional Food and dining US 48 ✓
rocketgames.io RocketGames.io
informational Gaming FR 52 ✓ ✓
manjaro.news nerdvpn.de and all subdomains
informational AI and machine learning CA 46 ✓
nerdvpn.de nerdvpn.de and all subdomains
informational AI and machine learning DE 45 ✓
medlife.ro MedLife
documentation Healthcare RO 56 ✓ ✓
towardspackaging.com Towards Packaging Analytics & Consulting
documentation — 52 ✓ ✓
imtec.ba Imtec.ba
transactional Ecommerce Consumer electronics BA 47 ✓
aiiimpact.com AIIImpact
informational Marketing FR 48 ✓
outscraper.com Outscraper
interface Data and analytics APIs and SDKs FR 60 ✓ ✓
bambinos.live Bambinos.live
transactional Education US 52 ✓
airfleet.co Airfleet
documentation AI and machine learning Marketing — 50 ✓ ✓
bemyapp.com BeMyApp
documentation Community Events and conferences FR 55 ✓
ecoutez.com Ecoutez
transactional Healthcare DE 52 ✓
edmontoncounsellingservices.ca Edmonton Counselling Services
transactional Healthcare CA 50 ✓
cyberhelpindia.com Cyber Help India
documentation Developer tools Healthcare +3 US 53 ✓
diwu.pl llms.txt — diwu.pl (AI Marketing Optimized)
PL 45 ✓
hamleys.com Hamleys
transactional Ecommerce — 54 ✓
czechvr.com Czech VR
transactional Media Entertainment and streaming — 51 ✓
buildologyinc.com Buildology Landscaping Supply Store
informational Construction and architecture US 50 ✓
scrapingpros.com Scraping Pros - Fully Managed Web Scraping Services
documentation Data and analytics — 46 ✓
mythfactoryshop.com Myth Factory Shop
transactional Industry and manufacturing Ecommerce — 51 ✓ ✓
go-creative.com.au GO Creative
documentation Marketing AU 45 ✓ ✓
firecrawl.dev Firecrawl
interface AI and machine learning APIs and SDKs +2 US 65 ✓ ✓
incrementors.com Incrementors Web Solutions
documentation Marketing PL 57 ✓
fabritekengineers.com Fabri-Tek Engineers
documentation Industry and manufacturing FR 46 ✓
puzl.com AI Access Policy for https://www.Puzl.com
US 46 ✓
dravenstales.ch Dravens Tales from the Crypt
informational Media Personal sites and blogs CH 47 ✓
olostep.com Olostep
interface AI and machine learning APIs and SDKs — 57 ✓
gecdesigns.com GEC Designs
transactional Design US 52 ✓
built2winweb.com BuiltToWinWeb
interface Design Marketing +5 US 53 ✓ ✓
equalweb.com EqualWeb
transactional Business software — 60 ✓ ✓
fcsapi.com FCS API
interface APIs and SDKs Data and analytics US 51 ✓
horizon.sa Horizon
informational Cybersecurity Business software +1 SA 48 ✓
seosly.com SEOSLY - Olga Zarr
documentation Marketing US 52 ✓
iias.ca IIAS - Internet Advertising
documentation Marketing CA 46 ✓
woensdag.nl Woensdag
documentation Infrastructure Marketing +1 NL 52 ✓
pharmbills.com Pharmbills
informational Healthcare Business software US 52 ✓
u4n.com U4N
transactional Gaming Ecommerce HK 58 ✓
vinayakinfosoft.com #1 Vinayak InfoSoft - SEO Company Ahmedabad
informational Marketing IN 54 ✓
coldstonecreamery.com Cold Stone Creamery
transactional Food and dining US 60 ✓
pageai.pro PageAI
documentation Marketing Design +3 — 49 ✓
agentwebmap.com Agent Web Map
interface AI and machine learning AI agents and MCP +1 FR 60 ✓ ✓
kaiserkraft.de kaiserkraft (KAISER+KRAFT GmbH) — LLM-Guidance (llms.txt)
transactional Ecommerce DE 51 ✓
beautycoiffure.com AI Crawling Directives for PrestaShop
transactional AI and machine learning Ecommerce +1 — 59 ✓
motorstore.pl AI Crawling Directives for motorstore.pl
transactional AI and machine learning Ecommerce +1 PL 53 ✓
discount-plomberie.com AI Crawling Directives for Displomb
transactional AI and machine learning Ecommerce FR 52 ✓
bell-anesse-en-provence.fr AI Crawling Directives for BellAnesse
transactional AI and machine learning Ecommerce +1 FR 51 ✓
mcpservers.org Awesome MCP Servers
documentation AI and machine learning Developer tools +1 — 50 ✓
marcapiumamaterassi.it AI Crawling Directives for Marcapiuma
transactional AI and machine learning Ecommerce +1 IT 48 ✓
bebepolis.es AI Crawling Directives for Bebepolis
transactional AI and machine learning Ecommerce ES 47 ✓
yy-ic.com YY-IC Electronic Components Sourcing Platform
transactional Ecommerce Industry and manufacturing — 47 ✓
reepl.io Reepl
transactional AI and machine learning Marketing IN 50 ✓ ✓
ornibird.com AI Crawling Directives for Ornibird SRL
transactional AI and machine learning Ecommerce +1 BE 55 ✓
aplaceinthesun.com A Place in the Sun
informational Real estate — 53 ✓
decoplus-parquet.com AI Crawling Directives for Decoplus FRANCE
documentation AI and machine learning Ecommerce +1 FR 51 ✓
wdracing.eu AI Crawling Directives for WD Racing
transactional AI and machine learning Ecommerce +1 — 48 ✓
kuipersagrishop.nl AI Crawling Directives for Kuipers Agrishop
transactional AI and machine learning Ecommerce +1 NL 46 ✓
lalqueriaonline.com AI Crawling Directives for Lalqueria Online
transactional AI and machine learning Ecommerce — 46 ✓
plazapremiumlounge.com Plaza Premium Lounge
transactional AI and machine learning Travel — 56 ✓ ✓
tabstack.ai Tabstack
interface APIs and SDKs AI agents and MCP +1 US 48 ✓ ✓
groix-et-nature.com AI Crawling Directives for Groix et Nature
transactional AI and machine learning Ecommerce FR 51 ✓
dmsports.fr AI Crawling Directives for DM Sports V2
transactional AI and machine learning Ecommerce +1 FR 50 ✓
optimerch.de Optimerch GmbH
documentation Marketing DE 47 ✓
tricorbraunwinepak.com TricorBraun WinePak
transactional Ecommerce — 46 ✓
trex.com Trex: Composite Decking & Outdoor Products
documentation Industry and manufacturing Construction and architecture +1 US 61 ✓
newamericanjackets.com New American Jackets - LLM Instructions
US 48 ✓
northamericanjackets.com North American Jackets - LLM Instructions
SG 46 ✓
noibu.com Noibu
documentation Ecommerce Data and analytics +1 US 61 ✓
impossible.sg Digital & Online Marketing Agency in Singapore
documentation Marketing SG 52 ✓
parken-und-meer.de PARKEN & MEER
transactional Travel DE 46 ✓
jessewaites.com Jesse Waites
documentation Personal sites and blogs US 51 ✓
qshop.ai Qshop
transactional Design Marketing +1 KR 57 ✓
hibu.com Hibu Inc.
documentation Marketing US 60 ✓
dpbosssnet.in Dpboss | Satta Matka Result | Live Chart Update Sattamatka
documentation IN 52 ✓
nriol.com NRIOL Insurance Portal - LLM Reference File
documentation Insurance US 52 ✓
digiemperor.com Digi Emperor
documentation Ecommerce Marketing SG 48 ✓
scrapehero.com ScrapeHero
interface Data and analytics AI and machine learning +1 US 57 ✓
coastalcapitaldispensary.com Coastal Capital Dispensary
documentation Local business US 50 ✓
lyfta.app Lyfta
transactional Sports and fitness US 52 ✓
tricorbraun.com TricorBraun
transactional Industry and manufacturing Ecommerce — 56 ✓
its.co.uk ITS
transactional Ecommerce GB 58 ✓
leadpipe.com Leadpipe
interface AI and machine learning Developer tools +2 US 54 ✓
lalpathlabs.com llms.txt for Dr Lal PathLabs
IN 45 ✓
acemapp.org LLM Instructions for AI Models
informational AI and machine learning US 47 ✓
context.dev Context.dev
interface AI and machine learning AI agents and MCP +2 US 60 ✓ ✓
scrapegraphai.com llms.txt
interface AI agents and MCP APIs and SDKs US 53 ✓ ✓
beonemedicines.com Guidance for Large Language Model (LLM) crawlers
informational AI and machine learning Healthcare +1 US 58 ✓
import.io Import.io
interface AI and machine learning Data and analytics +2 US 58 ✓ ✓
i.pn i.pn
interface APIs and SDKs PN 55 ✓ ✓
djjohal.com DJJohal
informational Entertainment and streaming Media — 53 ✓
activesearchresults.com Active Search Results
documentation Marketing US 58 ✓
scrape.do Scrape.do
interface APIs and SDKs DO 51 ✓
entiretravel.com.au Entire Travel Group
informational Travel AU 48 ✓
celebree.com Celebree School of Cary
informational Marketing Education +1 US 53 ✓
net-craft.com Net-Craft.com
documentation Ecommerce US 54 ✓
leboat.com llm.txt for Le Boat LLM Crawling Guide
US 50 ✓
fionnapps.com FionnApps
transactional Productivity Consumer electronics US 52 ✓
pagespeedmatters.com PageSpeed Matters
transactional Marketing — 50 ✓ ✓
hoburne.com Hoburne
transactional Travel GB 54 ✓
sdc.com SDC.com - Swinger and Open Lifestyle Media
informational Community Media — 60 ✓
cloudkilat.com CloudKilat
transactional Infrastructure ID 51 ✓
cleveland.edu Cleveland University-Kansas City: Chiropractic and Health Sciences
transactional Education Healthcare US 51 ✓
meta.vn META.vn
transactional Ecommerce Industry and manufacturing VN 53 ✓
trimble.com Corporate
informational CA 64 ✓
limecube.co Limecube
documentation Marketing Business software AU 47 ✓
stark.marketing stark.marketing
documentation Marketing Ecommerce — 51 ✓
onlinecasinos247.com Online Casinos 247
documentation Entertainment and streaming — 55 ✓ ✓
conductor.com Conductor PDF Resources
documentation Marketing US 63 ✓
loansjagat.com LoansJagat - India's 1st Loan Consolidation Marketplace
transactional Finance and payments — 54 ✓
seo.com SEO.com
documentation Marketing US 62 ✓
licenseplates.tv LicensePlates.tv
transactional Automotive Ecommerce AF 49 ✓
pageforge.pro PageForge
documentation Marketing FR 46 ✓ ✓
proxy-seller.com Proxy-Seller
transactional Infrastructure CY 59 ✓
purplepublish.com purplepublish.com
documentation Business software Media US 53 ✓
bloggingjoy.com Santanu Debnath
informational Marketing Personal sites and blogs FR 48 ✓
triptotemples.com Trip To Temples
transactional Travel IN 55 ✓ ✓
majestic.com Majestic
interface Marketing Data and analytics GB 64 ✓
trainocate.co.jp トレノケート株式会社 LLMs.txtポリシー / Trainocate Corporation AI Content Policy (LLMs.txt) | This file defines AI crawling and training permissions for LLMs. Please read carefully. | LLMs.txt Policy for AI Crawler
informational JP 47 ✓
exboyfriendrecovery.com Ex Boyfriend Recovery
informational Personal sites and blogs US 56 ✓
snapcraft.io Snapcraft
documentation Developer tools Open source projects GB 65 ✓ ✓
xthreemarketing.com X3 Marketing Group
documentation Marketing — 52 ✓
visualsitemaps.com VisualSitemaps
documentation Marketing Design — 51 ✓