---
title: "/llms.txt"
source: Panaversity Learn Agentic AI
source_url: https://github.com/panaversity/learn-agentic-ai
licence: Apache-2.0
domain: agentic-ai
subdomain: panaversity-agentic
date_added: 2026-04-25
---

# /llms.txt

The `/llms.txt` file is a proposed web standard designed to make website content more accessible and useful to large language models (LLMs) by providing a concise, structured, and machine-readable summary in Markdown format. Introduced by Jeremy Howard, co-founder of Answer.AI, on September 2, 2024, it aims to address the limitations LLMs face when processing complex web content, such as HTML-heavy pages or large documentation sites that exceed their context windows. Below are the complete details about `/llms.txt`, including its purpose, structure, implementation, adoption, and significance.

---

### Purpose of /llms.txt
The primary goal of `/llms.txt` is to optimize web content for LLMs, enabling them to quickly understand a website’s key information without needing to parse entire pages filled with extraneous elements like JavaScript, CSS, or navigation menus. Traditional web standards like `robots.txt` (for crawler instructions) and `sitemap.xml` (for search engine indexing) are not tailored for AI reasoning engines, which require concise and curated data to operate efficiently within their token limits. By providing a standardized, LLM-friendly file, websites can:
- Enhance AI comprehension of their content.
- Improve the accuracy of AI-generated responses about the site.
- Make critical information readily available for applications like coding assistants, answer engines, or research tools.

It’s particularly valuable for domains such as software documentation, e-commerce, education, and personal portfolios, where structured overviews can streamline AI interactions.

---

### Structure of /llms.txt
The `/llms.txt` file follows a specific Markdown-based format to ensure it is both human-readable and programmatically parsable. It must be placed in the root directory of a website (e.g., `https://example.com/llms.txt`). The structure includes the following sections, in this order:

1. **H1 Header (Required)**  
   - Starts with a single `#` followed by the project or website name.  
   - Example: `# FastHTML`

2. **Blockquote Summary (Optional but Recommended)**  
   - Begins with `>` and provides a brief overview of the site or project, including key information needed to understand the rest of the file.  
   - Example: `> FastHTML is a Python library for rapid web development...`

3. **Additional Information (Optional)**  
   - Freeform Markdown content (paragraphs, lists, etc., excluding headings) offering more context about the site or guidance on interpreting linked files.  
   - Example: `When using FastHTML, note that components are modular...`

4. **File Lists (Optional)**  
   - Sections delimited by `##` (H2) headers, each containing a list of URLs to additional resources.  
   - Each list item is a Markdown hyperlink `[name](url)` followed optionally by a colon `:` and a description.  
   - Common section names include `Docs`, `Examples`, and `Optional`.  
   - Example:  
     ```
     ## Docs
     - [Quick Start](https://example.com/quickstart.md): A brief overview of features
     - [API Reference](https://example.com/api.md): Detailed API docs
     ## Optional
     - [Community Forum](https://forum.example.com): User discussions
     ```

A minimal `/llms.txt` file could be as simple as:
```
# My Website
```

A more detailed example:
```
# FastHTML
> FastHTML is a Python library for building web apps quickly with minimal code.

Key features include modular components and HTMX integration.

## Docs
- [Quick Start](https://fasthtml.org/quickstart.md): Intro to core features
- [API Reference](https://fasthtml.org/api.md): Full API documentation

## Examples
- [Todo App](https://fasthtml.org/todo.py): A complete CRUD example

## Optional
- [Starlette Docs](https://starlette.io/docs.md): Related framework docs
```

---

### Companion Concepts
In addition to `/llms.txt`, related proposals enhance its utility:
- **Markdown Pages**: Websites are encouraged to provide clean Markdown versions of HTML pages by appending `.md` to URLs (e.g., `example.com/page.md` or `example.com/index.html.md`). This ensures LLMs can access content without parsing HTML.
- **/llms-full.txt**: An optional companion file that compiles all documentation into a single Markdown file, ideal for tools that can ingest larger contexts (e.g., 100k+ tokens). It’s not part of the core spec but is a common extension.

---

### Implementation Guidelines
To create an effective `/llms.txt` file:
1. **Location**: Save it as `llms.txt` in the website’s root directory (e.g., `https://yoursite.com/llms.txt`).
2. **Content**: Use clear, concise language, prioritize key information, and link to Markdown resources where possible.
3. **Testing**: Validate the file’s effectiveness by testing it with LLMs (e.g., ChatGPT, Claude) to ensure they can answer questions based on the content.
4. **Tools**: Several tools can automate generation:
   - **Mintlify**: Generates `/llms.txt` and `/llms-full.txt` for hosted documentation.
   - **Firecrawl’s llms.txt Generator**: Crawls a site and creates the file using GPT-4o-mini.
   - **dotenv’s llmstxt**: Builds it from a sitemap.xml.
   - **Answer.AI’s llms_txt2ctx**: Parses `/llms.txt` into XML context files for LLM use.

---

### Adoption and Momentum
Since its proposal in September 2024, `/llms.txt` has gained traction:
- **Early Adopters**: Companies like Anthropic, Mintlify, FastHTML, Cloudflare, and Stripe have implemented it for their documentation.
- **Community Tools**: Directories like `directory.llmstxt.cloud` and `llmstxt.directory` index sites using `/llms.txt`. Open-source tools (e.g., Firecrawl’s generator) support its creation.
- **Visibility**: Posts on X and articles on platforms like Towards Data Science highlight its growing popularity, with Jeremy Howard noting significant momentum by March 2025.

---

### Benefits
- **Efficiency**: Fits within LLM context windows by distilling content into a concise format.
- **Accuracy**: Reduces hallucinations by providing curated, authoritative data.
- **Visibility**: Enhances discoverability in AI-driven tools, akin to “SEO for LLMs.”
- **Versatility**: Applicable to diverse use cases, from tech docs to personal CVs.

---

### Challenges and Future Questions
- **Adoption**: Its success depends on widespread use by web developers.
- **Legal/Ethical**: Questions remain about AI rewriting content, copyright, and monetization when chatbots access `/llms.txt`.
- **Evolution**: It may integrate with or complement other standards (e.g., `robots.txt`) or emerging protocols like Model Context Protocols (MCP).

---

### How to Use /llms.txt
- **For Website Owners**: Add `/llms.txt` to make your site AI-friendly. Start with a basic version and expand as needed.
- **For LLM Users**: Load a site’s `/llms.txt` (or `/llms-full.txt`) into an AI tool (e.g., via copy-paste or upload) to query site-specific information accurately.
- **For Developers**: Parse `/llms.txt` programmatically (e.g., using Answer.AI’s Python library) to integrate into AI workflows.

---

In summary, `/llms.txt` is a forward-thinking standard that bridges the gap between web content and AI, reflecting a shift toward an AI-first internet. As of now, it’s still evolving, but its adoption by major players and community support suggest it could redefine how LLMs interact with the web. For more details, visit `llmstxt.org` or explore implementations like `docs.anthropic.com/llms.txt`.
