Metadata-Version: 2.4
Name: canonicalwebteam.store-llm
Version: 0.1.0
Summary: Flask extension serving a store's pages as Markdown at /page.md and listing them in llms.txt and llms-full.txt.
Home-page: https://github.com/canonical/canonicalwebteam.store-llm
Author: Canonical webteam
Author-email: webteam@canonical.com
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Flask
Requires-Dist: canonicalwebteam.markdown-response>=0.2.0
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

# canonicalwebteam.store-llm

Flask extension making a Canonical store (snapcraft.io, charmhub.io, ...) readable by LLMs:

- every public page is served as Markdown at `/page.md`
- `/llms.txt` lists the public pages, discovered from the routing table ([llmstxt.org](https://llmstxt.org/))
- `/llms-full.txt` concatenates their Markdown
- `<link rel="alternate" type="text/markdown">` can be advertised from the layout

The HTML to Markdown conversion is done by [canonicalwebteam.markdown-response](https://github.com/canonical/canonicalwebteam.markdown-response); this package adds the store-specific wiring around it.

## Installation

```bash
pip install canonicalwebteam.store-llm
```

## Usage

```python
from canonicalwebteam.store_llm import StoreLLM

StoreLLM(
    app,
    base_url="https://snapcraft.io",
    site_name="Snapcraft",
    summary="Snapcraft is the home of the Snap Store, where ...",
    sections={
        "": "Main pages",
        "about": "Publishing a snap",
        "docs": "Documentation",
        "store": "Store",
    },
    section_order=["Main pages", "Publishing a snap", "Documentation", "Store"],
    extra_links=[
        {
            "section": "Documentation",
            "url": "https://snapcraft.io/docs/",
            "title": "Snap documentation",
            "description": "Reference and explanation for snaps and snapd.",
        },
        {
            "section": "Optional",
            "url": "https://snapcraft.io/llms-full.txt",
            "title": "Every page in one file",
            "description": "The pages above concatenated as Markdown.",
        },
    ],
)
```

The application factory pattern works too: `store_llm = StoreLLM(base_url=..., ...)` then `store_llm.init_app(app)`.

### Options

| Option | Default | |
|---|---|---|
| `base_url` | required | Absolute URL of the site, used for the links in llms.txt |
| `site_name` | required | The H1 of llms.txt; also stripped from page titles (`About \| Snapcraft` → `About`) |
| `summary` | required | The blockquote under the H1 of llms.txt |
| `sections` | `{"": "Main pages"}` | Section label per first path segment. `""` is for top-level pages such as `/about`; `"about"` for `/about/publish`. Unknown prefixes go to `Other pages` |
| `section_order` | order of `sections` | Section headings in the order they appear. `Other pages` follows, then any unlisted section alphabetically, then `Optional` last so agents can skip it |
| `extra_links` | `[]` | Links discovery cannot find: `{"section", "url", "title", "description"}` dicts |
| `is_private` | `is_login_gated` | `callable(view) -> bool` for pages with no Markdown version. The default recognises the stores' `login_required` decorator. `None` makes every page public |
| `strip_elements` / `strip_classes` | Vanilla-aware lists | Passed to markdown-response |
| `markdown_cache_control` | `private, max-age=3600` | `Cache-Control` of Markdown responses |
| `llms_cache_control` | `public, max-age=43200, ...` | `Cache-Control` of `/llms.txt` and `/llms-full.txt` |

## What the store needs to provide

**Templates.** Discovery parses templates rather than rendering them, so every public page must define `meta_title` and `meta_description` blocks with static text, directly or through a layout it extends:

```html
{% block meta_title %}About Snaps | Snapcraft{% endblock %}
{% block meta_description %}Snaps are app packages for desktop, cloud and IoT.{% endblock %}
```

A page opts out of llms.txt with `<meta name="robots" content="noindex">`, the same signal that keeps it out of search results. Layouts and partials (file names starting with `_`) are never listed.

**Content.** Markdown is extracted from `<main id="main-content">` (falling back to `<body>`). Add `data-md-strip` to interactive chrome that reads badly as text: copy buttons, share widgets, charts.

**Alternate link.** Templates get `markdown_path`, the Markdown URL path of the current page or `None` for private pages:

```html
{% if markdown_path %}
  <link rel="alternate" type="text/markdown" href="https://snapcraft.io{{ markdown_path }}" />
  <link rel="describedby" href="https://snapcraft.io/llms.txt" />
{% endif %}
```

**Sitemap.** `app.extensions["store_llm"].sitemap_paths()` returns the same pages, so the sitemap and llms.txt never drift.

## Build step

Rendering every page for `llms-full.txt` is too slow to do on request, so generate both files at build time and the routes serve them from the static folder (falling back to rendering live when the files are missing, for development):

```bash
python3 -m canonicalwebteam.store_llm generate webapp.app:create_app
```

The argument is `module:attribute`, either a Flask app or a factory returning one. Run it from the repository root after the frontend build, for example in the Dockerfile and `rockcraft.yaml`, and ignore `static/llms.txt` and `static/llms-full.txt` in git. The command reports pages without a `meta_description` and pages that could not be rendered.

## Customising the output

The files are rendered from the `store_llm/llms.txt` and `store_llm/llms-full.txt` templates. A store can override either by adding a template with the same name to its own `templates/` folder; both receive `site_name`, and `llms.txt` receives `summary` and `sections` (`[{"section", "links": [{"title", "url", "description"}]}]`).

## How pages are discovered

A route is listed when it takes no URL parameters, answers GET, is not private, its view calls `render_template("....html")` with a literal name and does not `redirect`, and the template (or a layout it extends) has a non-empty `meta_title`. Routes rendering the same template or sharing a title are listed once, under the shortest path (`/store`, not `/explore`). The homepage is left out: the header of llms.txt already introduces the site.
