Metadata-Version: 2.4
Name: wagtail-machine-readable
Version: 0.1.0
Summary: Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.
Project-URL: Homepage, https://github.com/brett-allard-amp/wagtail-machine-readable
Project-URL: Documentation, https://github.com/brett-allard-amp/wagtail-machine-readable#readme
Project-URL: Changelog, https://github.com/brett-allard-amp/wagtail-machine-readable/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/brett-allard-amp/wagtail-machine-readable/issues
Author-email: Brett Allard <brett.allard@getamplified.co.uk>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,django,generative-engine-optimisation,llms,llms.txt,seo,wagtail
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: Django
Classifier: Framework :: Django :: 4.2
Classifier: Framework :: Django :: 5.2
Classifier: Framework :: Wagtail
Classifier: Framework :: Wagtail :: 6
Classifier: Framework :: Wagtail :: 7
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: wagtail>=6.3
Description-Content-Type: text/markdown

# wagtail-machine-readable

**Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.**

[![PyPI](https://img.shields.io/pypi/v/wagtail-machine-readable.svg)](https://pypi.org/project/wagtail-machine-readable/)
[![CI](https://github.com/brett-allard-amp/wagtail-machine-readable/actions/workflows/test.yml/badge.svg)](https://github.com/brett-allard-amp/wagtail-machine-readable/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/wagtail-machine-readable.svg)](https://pypi.org/project/wagtail-machine-readable/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

LLMs, answer engines and AI crawlers are becoming the front door to your
content — and they read [llms.txt](https://llmstxt.org/), not your carefully
tuned templates. `wagtail-machine-readable` turns your existing Wagtail page
tree into spec-compliant machine-readable outputs automatically: live pages
only, privacy restrictions respected, every Site in a multi-site install
served on its own hostname. Install it, add two lines, and `/llms.txt` works.

## What you get

- **`/llms.txt`** — a spec-compliant index of your site: H1 site name,
  blockquote description, H2 sections derived from your top-level pages,
  with `- [name](url): description` link lists.
- **`/llms-full.txt`** — the same page set with page content extracted to
  plain text/Markdown.
- **Static export** — a management command that writes the same files to
  disk for static hosting and CDN workflows.
- **Wagtail-native visibility rules** — only `live` pages, view-restricted
  (private) subtrees excluded, drafts excluded, multi-site aware.

## Quickstart (60 seconds)

```bash
pip install wagtail-machine-readable
```

```python
# settings.py
INSTALLED_APPS = [
    # …
    "wagtail_machine_readable",
]
```

```python
# urls.py — before Wagtail's catch-all page serving include
urlpatterns = [
    # …
    path("", include("wagtail_machine_readable.urls")),
    path("", include(wagtail_urls)),
]
```

That's it. Zero configuration produces a valid llms.txt:

```
# Acme

> Acme makes modular widgets.

## Pages

- [Contact](https://example.com/contact/): Get in touch.

## About

- [About](https://example.com/about/): Who we are.
- [Team](https://example.com/about/team/): The people.

## Blog

- [Blog](https://example.com/blog/): News and articles.
- [First post](https://example.com/blog/first-post/): Our first post.
- [Second post](https://example.com/blog/second-post/)
```

Sections come from the children of each Site's root page; descriptions come
from `search_description` by default. Every Wagtail `Site` gets its own
document on its own hostname.

## Static generation

```bash
python manage.py generate_machine_readable --output ./static-export
python manage.py generate_machine_readable --site example.com
```

Single-site projects write `llms.txt`/`llms-full.txt` into the output
directory; multi-site projects get one subdirectory per hostname.

## Settings

All configuration lives in a single dict. Every key has a working default —
configure only what you want to change:

```python
WAGTAIL_MACHINE_READABLE = {
    "SITE_DESCRIPTION": "Acme makes modular widgets.",
    "MAX_PAGES_PER_SECTION": 25,
    "EXCLUDE_PAGE_MODELS": ["blog.BlogTagIndexPage"],
}
```

| Key | Default | Purpose |
| --- | --- | --- |
| `SITE_DESCRIPTION` | `None` | Blockquote description. A string, or a `{hostname: description}` dict for multi-site. Falls back to the root page's description chain. |
| `SECTION_STRATEGY` | `TopLevelSectionStrategy` path | Dotted path to the class deriving H2 sections from the page tree. |
| `MAX_PAGES_PER_SECTION` | `50` | Cap entries per section (`None` = unlimited). |
| `RESPECT_SHOW_IN_MENUS` | `False` | Only include pages with `show_in_menus=True`. |
| `EXCLUDE_PAGE_MODELS` | `[]` | Page models to exclude, as `"app_label.ModelName"` strings (exact type). |
| `EXCLUDE_PAGE_IDS` | `[]` | Specific page ids to exclude. |
| `DESCRIPTION_FIELDS` | `["machine_readable_description", "search_description"]` | Per-page description fallback chain; first non-empty field wins. |
| `FULL_TEXT_ENABLED` | `True` | Serve/write `llms-full.txt`. |
| `EXTRACTOR` | `DefaultContentExtractor` path | Dotted path to the `ContentExtractor` used for full-text bodies. |
| `CACHE_MAX_AGE` | `3600` | `Cache-Control: public, max-age=N` on responses (`0` = no header). |

Misconfigured keys are caught by Django system checks at startup.

## Customisation

**Exclude a page type** — no mixin or migration needed:

```python
class InternalToolPage(Page):
    exclude_from_machine_readable = True
```

**Dedicated descriptions for AI consumers** — adopt the optional mixin:

```python
from wagtail_machine_readable.models import MachineReadableMixin

class ArticlePage(MachineReadableMixin, Page):
    promote_panels = Page.promote_panels + [
        FieldPanel("machine_readable_description"),
    ]
```

**Custom sections** — subclass `SectionStrategy` and point
`SECTION_STRATEGY` at it:

```python
from wagtail_machine_readable.generators import SectionStrategy

class NavigationSections(SectionStrategy):
    def build_sections(self, site):
        ...
```

**Custom content extraction** — subclass `ContentExtractor` and point
`EXTRACTOR` at it. The v0.2 StreamField→Markdown renderer will ship behind
this same interface.

## How it behaves

- **Visibility**: a page appears only if it is live, has no view
  restriction on itself or an ancestor, and is not excluded by settings or
  the class attribute. The private/draft rules match what anonymous
  visitors can already see — nothing non-public leaks.
- **Multi-site**: `Site.find_for_request()` scopes every request, so each
  hostname serves its own tree with absolute URLs.
- **Caching**: responses carry `Cache-Control: public, max-age=3600` by
  default. The views are `cache_page`-compatible (cache keys include the
  host), so wrapping them or enabling Django's cache middleware works per
  site out of the box.
- **Output safety**: titles and descriptions are collapsed to single lines
  and escaped, so page content cannot inject sections or links into the
  document structure.

## How is this different from django-llms-txt?

django-llms-txt is Django-generic: you describe your content to it.
`wagtail-machine-readable` is Wagtail-native: it already understands the
page tree (sections for free), `live`/privacy rules, multi-site scoping,
`search_description`, and — on the roadmap — StreamField content. Point it
at a Wagtail project and it produces the right document with zero
configuration.

## Roadmap

- **v0.2** — `.md` variants of every page and a full StreamField→Markdown
  renderer.
- **v0.3** — JSON-LD structured data and AI crawler `robots.txt` controls.

See [ROADMAP.md](ROADMAP.md) for details.

## Compatibility

Python 3.11–3.13 · Django 4.2/5.2 · Wagtail 6.3–7.x. The full matrix is
tested in CI.

## Contributing

```bash
git clone https://github.com/brett-allard-amp/wagtail-machine-readable.git
cd wagtail-machine-readable
python -m venv .venv && source .venv/bin/activate
pip install -e . --group dev
pre-commit install
pytest
```

Run the full matrix with `tox`. Bug reports, spec-compliance issues and
extractor improvements are all welcome — please include a failing test
where possible.

## License

[MIT](LICENSE)
