Metadata-Version: 2.4
Name: wagtail-machine-readable
Version: 0.3.0
Summary: Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.
Project-URL: Homepage, https://github.com/brett-allard-amp/wagtail-machine-readable
Project-URL: Documentation, https://github.com/brett-allard-amp/wagtail-machine-readable#readme
Project-URL: Changelog, https://github.com/brett-allard-amp/wagtail-machine-readable/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/brett-allard-amp/wagtail-machine-readable/issues
Author-email: Brett Allard <brett.allard@getamplified.co.uk>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,django,generative-engine-optimisation,llms,llms.txt,seo,wagtail
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: Django
Classifier: Framework :: Django :: 4.2
Classifier: Framework :: Django :: 5.2
Classifier: Framework :: Wagtail
Classifier: Framework :: Wagtail :: 6
Classifier: Framework :: Wagtail :: 7
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: wagtail>=6.3
Description-Content-Type: text/markdown

# wagtail-machine-readable

**Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.**

[![PyPI](https://img.shields.io/pypi/v/wagtail-machine-readable.svg)](https://pypi.org/project/wagtail-machine-readable/)
[![CI](https://github.com/brett-allard-amp/wagtail-machine-readable/actions/workflows/test.yml/badge.svg)](https://github.com/brett-allard-amp/wagtail-machine-readable/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/wagtail-machine-readable.svg)](https://pypi.org/project/wagtail-machine-readable/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

LLMs, answer engines and AI crawlers are becoming the front door to your
content — and they read [llms.txt](https://llmstxt.org/), not your carefully
tuned templates. `wagtail-machine-readable` turns your existing Wagtail page
tree into spec-compliant machine-readable outputs automatically: live pages
only, privacy restrictions respected, every Site in a multi-site install
served on its own hostname. Install it, add two lines, and `/llms.txt` works.

## What you get

- **`/llms.txt`** — a spec-compliant index of your site: H1 site name,
  blockquote description, H2 sections derived from your top-level pages,
  with `- [name](url): description` link lists.
- **`/llms-full.txt`** — the same page set with page content rendered to
  Markdown.
- **`.md` page variants** — every visible page served as Markdown on its
  own URL (`/about/team.md`, `/index.md` for the site root).
- **StreamField→Markdown extraction** — headings, links, images (alt text
  + URL), embeds and tables survive extraction instead of being stripped.
- **JSON-LD structured data** — schema.org `WebPage`/`Article`,
  `Organization` and `BreadcrumbList` per page via a template tag, with a
  per-page-type builder mapping.
- **AI crawler robots.txt** — opt-in allow/deny controls for known AI
  user agents (GPTBot, ClaudeBot, PerplexityBot, …) managed from settings.
- **Static export** — a management command that writes every output to
  disk for static hosting and CDN workflows.
- **Wagtail-native visibility rules** — only `live` pages, view-restricted
  (private) subtrees excluded, drafts excluded, multi-site aware, plus
  per-model and per-page opt-outs.

## Quickstart (60 seconds)

```bash
pip install wagtail-machine-readable
```

```python
# settings.py
INSTALLED_APPS = [
    # …
    "wagtail_machine_readable",
]
```

```python
# urls.py — before Wagtail's catch-all page serving include
urlpatterns = [
    # …
    path("", include("wagtail_machine_readable.urls")),
    path("", include(wagtail_urls)),
]
```

That's it. Zero configuration produces a valid llms.txt:

```
# Acme

> Acme makes modular widgets.

## Pages

- [Contact](https://example.com/contact/): Get in touch.

## About

- [About](https://example.com/about/): Who we are.
- [Team](https://example.com/about/team/): The people.

## Blog

- [Blog](https://example.com/blog/): News and articles.
- [First post](https://example.com/blog/first-post/): Our first post.
- [Second post](https://example.com/blog/second-post/)
```

Sections come from the children of each Site's root page; descriptions come
from `search_description` by default. Every Wagtail `Site` gets its own
document on its own hostname.

## Markdown page variants

Every visible page is also served as Markdown by appending `.md` to its
slug path — `/about/team/` becomes `/about/team.md`, and the site root is
`/index.md`. Responses use `text/markdown` and the same visibility rules as
llms.txt, so drafts and private pages 404. Set `MARKDOWN_ENABLED` to
`False` to turn the variants off.

Bodies are produced by `MarkdownContentExtractor`, which maps rich text
and StreamField blocks to Markdown — headings (demoted below the page
title), `[text](url)` links with rich-text page references expanded,
`![alt](url)` images, embeds and URL blocks as autolinks, and
`TableBlock` values as Markdown tables.

## Structured data (JSON-LD)

Add the template tag to your page template:

```html+django
{% load machine_readable %}
{% structured_data page %}
```

This renders a `<script type="application/ld+json">` element containing a
schema.org `@graph`: the page node (`WebPage` by default), a
`BreadcrumbList` from the page's ancestors and an `Organization` derived
from the Wagtail Site. Map page types to other builders — `Article` ships
ready to use — or your own `StructuredDataBuilder` subclasses:

```python
WAGTAIL_MACHINE_READABLE = {
    "STRUCTURED_DATA_BUILDERS": {
        "blog.BlogPage": "wagtail_machine_readable.structured_data.ArticleBuilder",
    },
}
```

Mappings match base classes too, so `"wagtailcore.Page"` changes the
default for every page type.

## AI crawler robots.txt

Opt in to serving `/robots.txt` with explicit rules for known AI user
agents:

```python
WAGTAIL_MACHINE_READABLE = {
    "ROBOTS_TXT_ENABLED": True,
    "ROBOTS_AI_DEFAULT": "allow",      # baseline for known AI agents
    "ROBOTS_AI_DENY": ["Bytespider"],  # per-agent overrides
    "ROBOTS_EXTRA": "Sitemap: https://example.com/sitemap.xml",
}
```

The built-in list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot,
Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot,
Meta-ExternalAgent and friends; the allow/deny lists also accept agents
not on the list. Other crawlers get `User-agent: *` / `Allow: /`. If your
project already serves robots.txt, keep its URL pattern above the package
include.

## Static generation

```bash
python manage.py generate_machine_readable --output ./static-export
python manage.py generate_machine_readable --site example.com
```

Single-site projects write `llms.txt`, `llms-full.txt`, the `.md` page
tree (`index.md`, `about/team.md`, …) and — when enabled — `robots.txt`
into the output directory; multi-site projects get one subdirectory per
hostname.

## Caching

Responses carry `Cache-Control: public, max-age=3600` by default
(`CACHE_MAX_AGE`). For sites where generation itself is expensive, opt in
to caching the generated documents server-side:

```python
WAGTAIL_MACHINE_READABLE = {
    "GENERATION_CACHE_TIMEOUT": 86400,
}
```

Rendered documents are stored in Django's default cache and invalidated
when a page is published or unpublished, so a long timeout is safe.

## Settings

All configuration lives in a single dict. Every key has a working default —
configure only what you want to change:

```python
WAGTAIL_MACHINE_READABLE = {
    "SITE_DESCRIPTION": "Acme makes modular widgets.",
    "MAX_PAGES_PER_SECTION": 25,
    "EXCLUDE_PAGE_MODELS": ["blog.BlogTagIndexPage"],
}
```

| Key | Default | Purpose |
| --- | --- | --- |
| `SITE_DESCRIPTION` | `None` | Blockquote description. A string, or a `{hostname: description}` dict for multi-site. Falls back to the root page's description chain. |
| `SECTION_STRATEGY` | `TopLevelSectionStrategy` path | Dotted path to the class deriving H2 sections from the page tree. |
| `MAX_PAGES_PER_SECTION` | `50` | Cap entries per section (`None` = unlimited). |
| `RESPECT_SHOW_IN_MENUS` | `False` | Only include pages with `show_in_menus=True`. |
| `EXCLUDE_PAGE_MODELS` | `[]` | Page models to exclude, as `"app_label.ModelName"` strings (exact type). |
| `EXCLUDE_PAGE_IDS` | `[]` | Specific page ids to exclude. |
| `DESCRIPTION_FIELDS` | `["machine_readable_description", "search_description"]` | Per-page description fallback chain; first non-empty field wins. |
| `FULL_TEXT_ENABLED` | `True` | Serve/write `llms-full.txt`. |
| `FULL_TEXT_MAX_PAGES` | `None` | Cap the number of pages in `llms-full.txt` (`None` = unlimited). |
| `MARKDOWN_ENABLED` | `True` | Serve/write `.md` page variants. |
| `EXTRACTOR` | `MarkdownContentExtractor` path | Dotted path to the `ContentExtractor` used for page bodies. |
| `CACHE_MAX_AGE` | `3600` | `Cache-Control: public, max-age=N` on responses (`0` = no header). |
| `GENERATION_CACHE_TIMEOUT` | `None` | Opt-in server-side caching of generated documents, in seconds. |
| `STRUCTURED_DATA_BUILDERS` | `{}` | Map `"app_label.ModelName"` to `StructuredDataBuilder` dotted paths. |
| `ROBOTS_TXT_ENABLED` | `False` | Serve/write `robots.txt` from this package. |
| `ROBOTS_AI_DEFAULT` | `"allow"` | Baseline policy (`"allow"`/`"deny"`) for known AI user agents. |
| `ROBOTS_AI_ALLOW` | `[]` | Agents to explicitly allow, overriding the baseline. |
| `ROBOTS_AI_DENY` | `[]` | Agents to explicitly deny, overriding the baseline. |
| `ROBOTS_EXTRA` | `""` | Raw text appended to robots.txt (e.g. a `Sitemap:` line). |

Misconfigured keys are caught by Django system checks at startup.

## Customisation

**Exclude a page type** — no mixin or migration needed:

```python
class InternalToolPage(Page):
    exclude_from_machine_readable = True
```

**Per-page editor controls** — adopt the optional mixin for a dedicated
AI-consumer description and a per-page exclusion flag, surfaced in the
page editor by `MachineReadablePanel`:

```python
from wagtail_machine_readable.models import MachineReadableMixin
from wagtail_machine_readable.panels import MachineReadablePanel

class ArticlePage(MachineReadableMixin, Page):
    promote_panels = Page.promote_panels + [MachineReadablePanel()]
```

The mixin adds fields to your page model, so run
`python manage.py makemigrations && python manage.py migrate` after
adopting it (and after upgrading from a version with fewer mixin
fields). Until the migration is applied, any query against the page
model — including the pages themselves — fails with a missing-column
error.

**Custom sections** — subclass `SectionStrategy` and point
`SECTION_STRATEGY` at it:

```python
from wagtail_machine_readable.generators import SectionStrategy

class NavigationSections(SectionStrategy):
    def build_sections(self, site):
        ...
```

**Custom content extraction** — subclass `ContentExtractor` (or
`MarkdownContentExtractor`, whose block handling and `convert_html` hook
are overridable) and point `EXTRACTOR` at it. Set `EXTRACTOR` to
`"wagtail_machine_readable.extractors.DefaultContentExtractor"` for the
plain-text behaviour of v0.1.

**Custom structured data** — subclass `StructuredDataBuilder` (or
`WebPageBuilder`/`ArticleBuilder`) and map page types to it via
`STRUCTURED_DATA_BUILDERS`.

## How it behaves

- **Visibility**: a page appears only if it is live, has no view
  restriction on itself or an ancestor, and is not excluded by settings,
  the class attribute or the per-page flag. The private/draft rules match
  what anonymous visitors can already see — nothing non-public leaks.
- **Multi-site**: `Site.find_for_request()` scopes every request, so each
  hostname serves its own tree with absolute URLs.
- **Caching**: responses carry `Cache-Control: public, max-age=3600` by
  default. The views are `cache_page`-compatible (cache keys include the
  host), so wrapping them or enabling Django's cache middleware works per
  site out of the box. Server-side generation caching is opt-in via
  `GENERATION_CACHE_TIMEOUT`, invalidated on publish/unpublish.
- **Output safety**: titles and descriptions are collapsed to single lines
  and escaped, so page content cannot inject sections or links into the
  document structure. JSON-LD payloads escape `<`, `>` and `&`, so page
  content cannot break out of the script element.

## How is this different from django-llms-txt?

django-llms-txt is Django-generic: you describe your content to it.
`wagtail-machine-readable` is Wagtail-native: it already understands the
page tree (sections for free), `live`/privacy rules, multi-site scoping,
`search_description`, and StreamField content. Point it at a Wagtail
project and it produces the right documents with zero configuration.

## Roadmap

v0.2 (Markdown page variants) and v0.3 (structured data and crawler
controls) have shipped. See [ROADMAP.md](ROADMAP.md) for what's next.

## Compatibility

Python 3.11–3.13 · Django 4.2/5.2 · Wagtail 6.3–7.x. The full matrix is
tested in CI.

## Contributing

```bash
git clone https://github.com/brett-allard-amp/wagtail-machine-readable.git
cd wagtail-machine-readable
python -m venv .venv && source .venv/bin/activate
pip install -e . --group dev
pre-commit install
pytest
```

Run the full matrix with `tox`. Bug reports, spec-compliance issues and
extractor improvements are all welcome — please include a failing test
where possible.

## License

[MIT](LICENSE)
