octoparse.com Web Scraping Tool & Free Web Crawlers | Octoparse
interface Data and analytics US 59 ✓
spider.cloud Spider
interface APIs and SDKs AI agents and MCP US 51 ✓ ✓
potentpages.com Potent Pages
documentation AI and machine learning Data and analytics US 50 ✓
crawlee.dev Crawlee for Python · Fast, reliable Python web crawlers.
documentation Open source projects Developer tools 50 ✓ ✓
crawlbase.com Crawlbase
interface APIs and SDKs Data and analytics 60 ✓ ✓
scrapegraphai.com llms.txt
interface AI agents and MCP APIs and SDKs US 48 ✓ ✓
anakin.io Anakin.io
interface AI and machine learning AI agents and MCP +1 IN 51 ✓ ✓
crawl4ai.com Crawl4AI Cloud API
interface AI and machine learning APIs and SDKs DE 45 ✓
diffbot.com Diffbot
interface APIs and SDKs AI and machine learning +1 US 56 ✓
crawlora.net Crawlora
interface Real estate Ecommerce +3 50 ✓ ✓
fandangoseo.com FandangoSEO: Website crawler \& SEO Log Analyzer
documentation Marketing IE 46 ✓
spotibo.com Spotibo is not a web spider. It is a web dog! drumbigoo
transactional Marketing Developer tools 46 ✓
legionisci.com Legionisci.com — AI crawler information
PL 48 ✓
publicwww.com PublicWWW
documentation Developer tools Data and analytics DE 56 ✓ ✓
hankookilbo.com Hankook Ilbo — AI Agent & Web Crawler Guidelines
documentation AI and machine learning Media KR 66 ✓ ✓
geniecrawl.com Genie Crawl
documentation Marketing US 46 ✓
firecrawl.dev Firecrawl
interface AI and machine learning AI agents and MCP +2 US 65 ✓ ✓
fintreeindia.com Fintree - AI Crawler Guidance
IN 46 ✓
scrapehero.com ScrapeHero
interface Data and analytics AI and machine learning +1 US 57 ✓
suvudu.com llms.txt — AI crawler instructions for this site (November 2025)
US 45 ✓
nimbleway.com Nimble
interface AI and machine learning AI agents and MCP +2 US 56 ✓
bitkom.org llms.txt — Machine-readable site overview for AI and research crawlers
DE 55 ✓
tavily.com Tavily Docs
interface AI and machine learning AI agents and MCP +1 65 ✓ ✓
sen7ry.com Sentry
46 ✓ ✓
pr-cy.io pr-cy.io — SEO Tools & Website Analysis
informational Marketing AI and machine learning +1 RU 51 ✓
olostep.com Olostep
interface AI and machine learning APIs and SDKs 57 ✓
errortrace.dev Sentry
US 57 ✓ ✓
piperic.com Piperic
documentation Ecommerce Data and analytics FI 48 ✓
supermemory.ai supermemory
interface AI and machine learning AI agents and MCP +1 55 ✓ ✓
prescriptionaestheticandwellnessspa.com Prescription Aesthetic and Wellness
transactional Marketing Healthcare FR 46 ✓
zenrows.com Zenrows
interface AI and machine learning APIs and SDKs +1 61 ✓ ✓
iproxy.online iProxy Online
interface Infrastructure 49 ✓
cancer.ca llms.txt — AI / LLM crawling preferences
CA 49 ✓
webrecorder.net Webrecorder
documentation Developer tools Infrastructure US 48 ✓
webscraper.io Web Scraper
interface Data and analytics APIs and SDKs 49 ✓
promptcloud.com PromptCloud
documentation AI and machine learning Ecommerce +2 US 48 ✓
webscrapingapi.com WebScrapingAPI
interface Data and analytics APIs and SDKs 50 ✓ ✓
chatbotgen.com ChatbotGen
documentation AI and machine learning No-code and automation 51 ✓ ✓
seocom.agency SEOCOM
documentation Marketing 48 ✓
clinicorp.com llms.txt for www.clinicorp.com
45 ✓
scrapfly.io Scrapfly
interface AI agents and MCP APIs and SDKs +1 US 47 ✓
joinmassive.com Massive
interface APIs and SDKs Data and analytics US 64 ✓
actowizsolutions.com Actowiz Solutions
documentation Data and analytics US 48 ✓
functionize.com LLM.txt - Website Content Structure
US 49 ✓
theplusaddons.com The Plus Addons for Elementor
documentation 56 ✓ ✓
browse.ai Browse AI - No-code web scraping and website monitoring platform
interface AI and machine learning Data and analytics +2 54 ✓
geonode.com Geonode
interface Infrastructure APIs and SDKs US 62 ✓ ✓
scraperapi.com ScraperAPI
interface AI and machine learning APIs and SDKs +2 US 53 ✓
chatforest.com ChatForest
documentation Data and analytics Developer tools +2 US 47 ✓
oncrawl.com Oncrawl
interface Marketing Business software +1 FR 52 ✓
aerosend.io Aerosend
transactional Marketing Communications 50 ✓ ✓
avtoelon.uz Instructions for AI models crawling avtoelon.uz
informational AI and machine learning UZ 54 ✓ ✓
hasdata.com HasData
interface APIs and SDKs Data and analytics 51 ✓
simplescraper.io Simplescraper
interface AI agents and MCP APIs and SDKs +1 51 ✓ ✓
rankai.ai RankAI
documentation Marketing Business software US 49 ✓
scrape.do Scrape.do
interface APIs and SDKs DO 51 ✓
spd.tech SPD Technology
documentation AI and machine learning Developer tools +1 51 ✓
otis.edu llms.txt — AI-friendly site summary for LLM crawlers
US 45 ✓
nohacks.co No Hacks - AI Agent Guidelines
informational AI and machine learning Media +1 51 ✓ ✓
leboat.com llm.txt for Le Boat LLM Crawling Guide
US 50 ✓
openclaw.ai OpenClaw
documentation AI and machine learning AI agents and MCP US 47 ✓
mountwebtech.com Professional Digital Marketing Services
documentation Ecommerce Marketing FR 47 ✓
webscraping.ai WebScraping.AI
interface AI and machine learning APIs and SDKs +1 51 ✓ ✓
litslink.com LITSLINK
documentation Developer tools Cybersecurity +2 48 ✓
scrapingdog.com Scrapingdog
interface Recruiting and careers APIs and SDKs 52 ✓
mylittlebigweb.com My Little Big Web - Agence Web à Montréal. Tous les services au même endroit. SEO, conception de site Web, Google Ads, Facebook Ads, réseaux sociaux.
documentation Marketing 48 ✓
eweka.nl llms.txt for https://www.eweka.nl/en
NL 51 ✓
juiceddigital.com Digital PR Expert | Digital Marketing & Search Leaders
documentation Marketing US 46 ✓
conceptfeedback.com wingify.com
46 ✓
wonderchat.io Wonderchat
documentation AI and machine learning AI agents and MCP 50 ✓
visualwebsiteoptimizer.com wingify.com
US 66 ✓
scrappey.com Scrappey.com
interface APIs and SDKs Developer tools 46 ✓
skyfire.xyz Skyfire Developer Portal Documentation
interface Developer tools AI and machine learning +3 47 ✓
seotoolsforexcel.com SeoTools for Excel
documentation Marketing DE 51 ✓ ✓
hashtag.org hashtag.org
documentation AI and machine learning 53 ✓ ✓
localdigital.com.au Local Digital
documentation Marketing AU 47 ✓
janbaskdigitaldesign.com JanBask Digital Design — Web Design, Web Development & Digital Marketing Agency
documentation Developer tools Marketing +1 52 ✓ ✓
runbear.io Runbear
interface Marketing AI agents and MCP US 51 ✓ ✓
prisync.com Prisync
documentation Ecommerce Data and analytics 53 ✓
jump.bg llms.txt for Jump.BG
transactional Infrastructure BG 50 ✓ ✓
aerones.com Robotic Wind Turbine Care Systems | Aerones
informational Industry and manufacturing Energy and utilities 48 ✓
modpagespeed.com mod_pagespeed 2.1
documentation Infrastructure Developer tools US 53 ✓ ✓
airfleet.co Airfleet
documentation AI and machine learning Marketing 50 ✓ ✓
webless.ai Webless
documentation AI and machine learning Business software US 46 ✓
encryptionconsulting.com Encryption Consulting LLC LLMS.TXT
documentation Cybersecurity 56 ✓ ✓
context7.com Context7
interface AI agents and MCP Developer tools +1 US 59 ✓
website.com Website.com
transactional Ecommerce Infrastructure +2 62 ✓ ✓
angeo.dev angeo.dev
documentation Ecommerce AI agents and MCP FR 45 ✓ ✓
eminenture.com Eminenture
documentation Data and analytics Marketing +1 US 46 ✓
specification.website The Website Specification
interface Documentation and reference 52 ✓ ✓
open.cx Open - The unified AI customer support platform
documentation Marketing 53 ✓ ✓
e-dimensionz.com e-dimensionz Inc.
documentation Developer tools Business software US 51 ✓ ✓
igears.net iGears Technology Limited
documentation AI and machine learning Cybersecurity 47 ✓
dataprovider.com Dataprovider.com
interface Data and analytics AI agents and MCP US 51 ✓ ✓
apify.com Apify
interface Developer tools AI agents and MCP +1 57 ✓
clickwisedesign.com Click Wise Design
informational Design Marketing 46 ✓
thriveweb.com.au Thrive Digital Web Design Gold Coast: Websites that grow with your business ⋆ Gold Coast Web Design ⋆ Wordpress Support \& Hosting ⋆ Coding \& Development ⋆ Based in Burleigh Heads, Gold Coast, Austra
informational Design Local business AU 48 ✓
visualsitemaps.com VisualSitemaps
documentation Marketing Design 46 ✓
yourgpt.ai YourGPT - The Complete AI Agent Platform
interface AI and machine learning 57 ✓ ✓
recyclingportal.eu RecyclingPortal
informational Industry and manufacturing Media DE 48 ✓
activesearchresults.com Active Search Results
documentation Marketing US 51 ✓
seiten-werk.com Seiten-Werk (https://seiten-werk.com/)
documentation Marketing DE 48 ✓
marketingblendz.com Marketing Blendz
documentation Marketing 47 ✓
calgaryseocompany.ca MRC SEO Consulting
transactional Marketing CA 47 ✓
okhten.com Artem Okhten Official Website
documentation AI and machine learning US 52 ✓ ✓
irisagent.com IrisAgent | AI Customer Support Automation Platform
interface Marketing Business software +1 US 51 ✓ ✓
webhostingtalk.nl Webhostingtalk.nl
informational Infrastructure Community NL 57 ✓
smytten.com Smytten
transactional Ecommerce Fashion and beauty IN 57 ✓
gardena.com GARDENA
documentation Home and garden NL 62 ✓
freehost.com.ua Разрешаем OpenAI GPTBot (ChatGPT)
documentation UA 50 ✓
cdnetworks.com 规则组 1: 针对所有爬虫的通用规则
documentation FR 48 ✓
asterthemes.com asterthemes.com
transactional Developer tools CA 56 ✓ ✓
pixsy.com Pixsy
documentation Legal Cybersecurity US 52 ✓
truenorthsocial.com True North Social - Full-Service Digital Marketing Agency
informational Marketing Media 52 ✓
semrush.com Semrush
interface Marketing US 69 ✓
honour.app Honour.app
informational Recruiting and careers Education IN 46 ✓
tech-now.io TechNow (tech-now.io)
documentation 47 ✓
potensic.com Potensic Official Website
transactional Consumer electronics US 50 ✓
grupobimbo.com Grupo Bimbo - LLM Context Gateway
documentation Food and dining US 54 ✓
tstprep.com TST Prep
transactional Education 51 ✓ ✓
ud.hk UD (UDomain) — ud.hk
documentation Cybersecurity Infrastructure +1 HK 46 ✓
eu-recycling.com EU-Recycling
informational Media DE 47 ✓
farbentour.de Farbentour
documentation Marketing DE 46 ✓
teenager365.to llms.txt
47 ✓
kontron-ais.com kontron-ais.com - llms.txt (for AI crawlers and generative engines)
documentation Industry and manufacturing Business software DE 49 ✓
casrai.org CASRAI — AI Citation Guide
documentation Documentation and reference Science and research 53 ✓
sanyglobal.com SANY Group | Global Website
documentation Industry and manufacturing Construction and architecture FR 58 ✓
niural.com Niural
interface Finance and payments Productivity +1 US 47 ✓
chatlab.com ChatLab
documentation AI and machine learning No-code and automation 56 ✓
evisio.co Evisio
documentation Marketing Productivity +1 US 47 ✓
zzpstudio.nl llms.txt van ZZPstudio maatwerk Webdesign en Full-stack / Laravel Webapplicatie development
informational Marketing NL 47 ✓
searchenginejournal.com Search Engine Journal: The AI Search \& SEO News That Matters
informational Marketing AI and machine learning +1 GB 68 ✓
siteway.de Siteway
documentation Marketing DE 52 ✓ ✓
itagroup.com ITA Group
informational Data and analytics Marketing US 51 ✓
steamunlocked.org SteamUnlocked
documentation Gaming 61 ✓
googiehost.com GoogieHost
transactional Infrastructure Design 58 ✓
maxburst.com Web Design Company & SEO Marketing Agency in NYC & Long Island
informational Marketing Design US 55 ✓ ✓
kom.pe KOM Agencia Digital – Official AI & LLM Context File
documentation Marketing PE 51 ✓ ✓
mycvcreator.com MyCVCreator
documentation Recruiting and careers 56 ✓ ✓
michaelkors.co.nl michaelkors.co.nl
transactional Fashion and beauty Ecommerce NL 45 ✓
mobilebettingparsi.com mobilebettingparsi.com
informational 45 ✓
powerthesaurus.org Power Thesaurus
documentation Education Documentation and reference 54 ✓
wk.se Webbkonsulterna AB
documentation Infrastructure SE 47 ✓
hostedscan.com HostedScan
interface Cybersecurity Infrastructure +1 DE 47 ✓
socure.com Socure
documentation Cybersecurity Authentication and identity US 59 ✓
core.app Core
transactional Crypto and Web3 53 ✓
flatlineagency.com Flatline Agency
documentation AI and machine learning Ecommerce +1 NL 47 ✓
cointraffic.com Cointraffic
documentation Marketing Crypto and Web3 51 ✓ ✓
laverne.com لافيرن - Official website
interface Fashion and beauty Ecommerce 61 ✓ ✓
epicwebstudios.com Epic Web Studios
informational Developer tools Marketing US 47 ✓
natu.care Natu.Care
informational Ecommerce US 50 ✓
mailbutler.io Mailbutler
documentation Productivity DE 53 ✓
hosting.de hosting.de - llm.txt
transactional Infrastructure DE 52 ✓
eesel.ai eesel AI
transactional AI and machine learning AI agents and MCP US 64 ✓ ✓
laserliner.com Laserliner - AI Crawler Index
documentation AI and machine learning Consumer electronics DE 48 ✓ ✓
ocaml.org OCaml
documentation Developer tools Open source projects FR 61 ✓
context.dev Context.dev
interface AI and machine learning APIs and SDKs +1 US 55 ✓ ✓
purcorpest.com PURCOR
documentation Local business US 53 ✓ ✓
webotit.ai Webotit.ai
documentation AI and machine learning US 53 ✓ ✓
easymusic.ai EasyMusic.AI
transactional AI and machine learning Entertainment and streaming 49 ✓
tryghed.dk TryghedsGruppen - AI Agent Information
informational AI and machine learning DK 46 ✓
seosly.com SEOSLY - Olga Zarr
documentation Marketing 47 ✓
sellingtoolz.com SellingToolz
transactional Real estate Marketing CA 52 ✓ ✓
zyte.com Zyte – The complete toolbox for web scraping, data extraction, and proxy infrastructure at scale. Trusted by thousands of companies worldwide.
interface APIs and SDKs Data and analytics 55 ✓
ortecadscience.com ORTEC Adscience - Online Marketing Tooling : Full Actionable Insights
documentation Data and analytics Design +1 47 ✓
marketingshift.nl Marketing Shift | Websites, Strategie, Marketing
informational Marketing NL 48 ✓
seono1.co.th SEONo1
documentation Marketing TH 48 ✓
versio.nl Versio
transactional Infrastructure NL 54 ✓
mupdf.com MuPDF
documentation Developer tools APIs and SDKs +1 US 55 ✓ ✓
zite.com Zite - Help Center
documentation No-code and automation US 52 ✓
markifact.com Markifact
interface AI and machine learning Marketing +1 US 48 ✓
sodah.de Sodah Webdesign
interface Marketing DE 52 ✓ ✓
verifieddr.com VerifiedDR
interface AI and machine learning Marketing +1 50 ✓ ✓
webwiki.fr webwiki
informational Cybersecurity FR 57 ✓
svenkersten.nl Sven Kersten
documentation Marketing NL 46 ✓
tomba.io Tomba Email Finder
interface Marketing 51 ✓
nbrgroup.in NBR Group
documentation Real estate IN 46 ✓
darwinapps.com DarwinApps
documentation Data and analytics Design +2 46 ✓
zenmedia.com Zen Media: B2B Marketing and PR Agency for B2B Brands
documentation Marketing Media 48 ✓
accesso.com accesso
documentation Business software US 48 ✓
overgrow.com Overgrow.com
informational Community CZ 57 ✓
sorumatik.co Sorumatik
documentation Education AI and machine learning 51 ✓
unicyclist.com Unicyclist.com
informational Community FI 50 ✓
ziggit.dev Ziggit
documentation Community Developer tools US 50 ✓
ifin.network IFIN
documentation US 49 ✓
isharkfly.com iSharkFly
documentation Community 49 ✓
sat4all.com Sat4all
documentation Communications Community NL 49 ✓
forospyware.com ForoSpyware
documentation Cybersecurity Community 48 ✓
clojureverse.org ClojureVerse
documentation Community Developer tools FR 47 ✓
cometforums.com CometForums
documentation Community 47 ✓
londonbikers.com londonbikers.com
informational Community 47 ✓
s4talk.in S4Talk
documentation Community IN 47 ✓
tiexuedanxin.net 铁血丹心
informational Community HK 47 ✓
sendegate.de Sendegate
documentation Community DE 46 ✓
routerich.ru Routerich AX3000
informational Consumer electronics RU 45 ✓
sagepilot.ai Sagepilot
interface AI and machine learning Marketing US 55 ✓
y2kwebs.com Y2K Webs
documentation Marketing Ecommerce 48 ✓
1solutions.biz 1Solutions
informational Design Developer tools +3 US 47 ✓
uptimewebhosting.com.au UpTime Web Hosting
documentation Infrastructure AU 46 ✓
krita-artists.org Krita Artists
informational Arts and culture Community DE 51 ✓
penpot.app Penpot
documentation Design 64 ✓ ✓
k8slens.dev Lens Forums
documentation GB 57 ✓
elitefourum.com Elite Fourum
informational Community US 51 ✓
dastereo.ru DA Stereo
informational Community RU 49 ✓
finsweet.com Finsweet
informational Marketing 49 ✓
troypointinsider.com TROYPOINT Insider
informational Community US 49 ✓
bcharts.com.br BCharts Fórum
informational Community Communications BR 48 ✓
hitmanforum.com Hitman Forum
informational Community 47 ✓
erlangforums.com Erlang Forums
documentation Community DE 46 ✓
idmforums.com IDM Forums
documentation Community US 46 ✓
image.sc Image.sc Forum
informational Community US 45 ✓
kenyatalk.com Kenya Talk
documentation Community 45 ✓
agentwebmap.com Agent Web Map
interface AI and machine learning AI agents and MCP +1 FR 60 ✓ ✓
louhi.fi Louhi Net Oy
documentation Infrastructure FI 57 ✓
richpacking020.com RICH PACKING: Pharmaceutical Processing and Packaging Machinery
documentation Industry and manufacturing US 50 ✓ ✓
mexseo.info MexSEO
documentation AI and machine learning Marketing 46 ✓
festivalregiony.cz REGIONY
informational Events and conferences Arts and culture CZ 48 ✓
incrementors.com Incrementors Web Solutions
documentation Marketing PL 52 ✓
levamo.com Levamo
documentation Education Infrastructure 51 ✓ ✓
bbcincorp.com BBCIncorp
documentation Business software 49 ✓
bthrust.com BThrust
documentation Developer tools Marketing SG 46 ✓
ameresco.com Ameresco
documentation Energy and utilities US 54 ✓
seosherpa.com SEO SHERPA™
documentation Marketing 52 ✓
cuny.com C.U.N.Y. Digital
documentation Marketing Nonprofit and advocacy 52 ✓
onehack.st OneHack a.k.a 1Hack
informational Education Community 48 ✓ ✓
wowlazymacros.com WoW Lazy Macros
informational Gaming Community 48 ✓
colorlibsupport.com Colorlib Support Forum
informational Community 47 ✓
failheap-challenge.com Failheap-Challenge
informational Community 46 ✓
thephilosophyforum.com The Philosophy Forum
informational Community 46 ✓
timetravelinstitute.com Time Travel Institute
documentation Travel Community 46 ✓
gcc-marketing.com GCC Marketing
documentation Ecommerce Marketing +1 47 ✓
musicpusic.com Music Pusic - LLM and AI Agent Guide
documentation AI and machine learning Media IR 45 ✓
bgw.live Bitget Wallet
documentation Crypto and Web3 52 ✓
ukr.media UkrMedia — Instructions for AI Agents and LLMs
documentation AI and machine learning Media DE 50 ✓
getmacha.com Macha
interface AI and machine learning AI agents and MCP +1 US 53 ✓ ✓
beyondblue.org.au Beyond Blue (Australia)
informational Healthcare AU 58 ✓
onwebchat.com onWebChat
documentation AI and machine learning Communications DE 49 ✓
wazuh.com Wazuh
documentation Cybersecurity 62 ✓
computechstore.in Computech Store
transactional Ecommerce Consumer electronics IN 59 ✓ ✓
interlochen.org Interlochen Center For the Arts
documentation Education Arts and culture US 59 ✓
kapa.ai kapa.ai
documentation Developer tools Community +2 CA 59 ✓
smartdatacollective.com SmartDataCollective
informational AI and machine learning Data and analytics +1 US 57 ✓
sociallykeeda.com SociallyKeeda
informational Media 53 ✓ ✓
rhinorank.io Rhino Rank
interface Marketing AI agents and MCP 52 ✓ ✓
wpbacklinks.com WPBacklinks.com - SEO Backlinks And Campaigns For WordPress Sites
documentation Marketing 52 ✓ ✓
bloomberry.com Bloomberry
interface Marketing Data and analytics +2 51 ✓
bonobolabs.com Bonobo
documentation Productivity US 51 ✓ ✓
mirovina.hr mirovina.hr
informational Media HR 51 ✓
price2spy.com Price2Spy
documentation Ecommerce Data and analytics FI 51 ✓
lief.re LIEF
documentation Developer tools Open source projects +1 US 50 ✓ ✓