Credit to the original project
The llms.txt proposal was created by Jeremy Howard. The original project's website and public source repository are:
- Website: https://llmstxt.org/
- GitHub: https://github.com/AnswerDotAI/llms-txt
We credit the llms.txt project for the convention Agent Web Map builds on.
The original llms.txt proposal
llmstxt.org is the reference website for the llms.txt proposal, first published by Jeremy Howard in September 2024. It explains how a concise Markdown file can help an agent understand a website and find useful source material.
The proposal and its supporting tools are developed in the public AnswerDotAI/llms-txt repository. We credit that project for the convention that makes this directory possible.
Read the source
- The llms.txt proposal: the format, examples and guidance from the original project.
- Changes to the proposal: the August 2026 revision and its discovery conventions.
- Project source and discussion: documentation, tools, issues and contributions.
What the proposal helps publishers do
An llms.txt file starts with a project name and can add a short summary, context and organized links. Its purpose is to guide a reader toward relevant material. The detail can live in linked Markdown pages.
The August 2026 revision describes how a file can cover a website or a path within it, with the most specific file applying. It also recommends standard link relations to identify a page’s Markdown alternative and the llms.txt file describing it. The upstream proposal is the place to check current format guidance.
How Agent Web Map builds on that work
Agent Web Map is an independent discovery service for websites using the published llms.txt convention. Our work is to find publisher files, check them over time and make their source links searchable.
- Browse publishers: find sites by query, topic, country or the kind of material an agent can use.
- Explore shared sections: follow connections between headings found in publisher files.
- Use the API and MCP server: query the same directory from an agent and inspect source evidence and freshness.
The llms.txt project defines the proposal. Agent Web Map adds discovery, validation and navigation across publishers using it.
Publish your own file
Start with the original proposal, then use our format introduction, checker or draft generator. Review the contents before publishing. Once your file is available, you can submit the website to the directory. Discovery does not require an account, and ownership verification is optional.