Metadata-Version: 2.5
Name: rer.linkchecker
Version: 1.0.0a3
Summary: An addon that check all links in site contents and generate a report with broken ones
Project-URL: Homepage, https://github.com/RegioneER/rer-linkchecker
Project-URL: PyPI, https://pypi.org/project/rer.linkchecker
Project-URL: Source, https://github.com/RegioneER/rer-linkchecker
Project-URL: Tracker, https://github.com/RegioneER/rer-linkchecker/issues
Author-email: RedTurtle Technology <sviluppo@redturtle.it>
License: GPL-2.0-only
License-File: LICENSE.GPL
License-File: LICENSE.md
Keywords: CMS,Plone,Python
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Framework :: Plone
Classifier: Framework :: Plone :: 6.1
Classifier: Framework :: Plone :: 6.2
Classifier: Framework :: Plone :: Addon
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.11
Requires-Dist: plone-api
Requires-Dist: plone-restapi
Requires-Dist: plone-volto
Requires-Dist: products-cmfplone
Requires-Dist: requests
Provides-Extra: test
Requires-Dist: horse-with-no-namespace; extra == 'test'
Requires-Dist: plone-app-testing; extra == 'test'
Requires-Dist: plone-restapi[test]; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-plone>=1.0.0a2; extra == 'test'
Description-Content-Type: text/markdown

# rer.linkchecker

A Plone addon that checks every internal and external link found in site contents (fields and Volto blocks) and generates a report of the broken ones.

## Features

- Adds a `portal_linkchecker` tool that crawls the whole site, collecting every link found in content fields and in Volto blocks (`resolveuid/...` links included).
- Checks internal links by resolving them against the catalog, and external links concurrently over HTTP (configurable timeout, thread pool size, and per-host throttling).
- Caches external link statuses for a configurable TTL, so repeated runs only re-check links that are due.
- Distinguishes real broken links from bot-protection responses (`403`, `429`, LinkedIn's `999`) and from `http://` links that only work over `https://` (reported so they can be fixed in place, not counted as broken).
- Reports the conditions that are not a plain http status with their own negative status, so they can be told apart in the CSV: `-1` timeout, `-2` works only over `https` (update the link), `-3` connection error.
- Does not verify TLS certificates: only reachability matters here, and many otherwise working servers omit their intermediate certificate (browsers fetch it themselves, `requests` does not), which would be reported as a broken link.
- Exposes the results as a CSV report (`PAGE, LINK, TYPE, STATUS, DESCRIPTION`) via `tool.get_rows()`.
- Ships a `check_broken_links` console script to run a check from the command line or from cron, without going through the web.

## Installation

Install `rer.linkchecker` with uv:

```shell
uv add rer.linkchecker
```

or add it to a zc.buildout-based project as a develop egg / source, alongside `plone.volto` (required: it provides the `blocks`/`blocks_layout` fields the linkchecker scans, and `plone.distribution` used by `addPloneSite`).

Then add `rer.linkchecker` to the `eggs` of your Plone instance and install the add-on from the Plone control panel (or via a GenericSetup profile) as usual.

## Usage

### The `portal_linkchecker` tool

```python
from plone import api

tool = api.portal.get_tool("portal_linkchecker")
tool.check_site()  # crawl the whole site and check every link

for uid, broken_links in tool.get_page_with_broken_links():
    ...  # [(link, status), ...] per content UID

for item in tool.get_broken_links():
    ...  # flat view: one dict per broken link, with the page it sits on

import csv

with open("broken_links.csv", "w", newline="") as fh:
    writer = csv.writer(fh, quoting=csv.QUOTE_ALL)
    for row in tool.get_rows():
        writer.writerow(row)
```

`get_broken_links()` is the flat source every report is built from, and `get_rows()` is its csv
rendering. Both include the bot-protected links (each with its own description), while
`get_page_with_broken_links()` leaves them out: it answers "how many real problems are there".
Filter either with `tool.filter_links(items, status=[404], link_type="EXTERNAL")`.

`check_site(ttl=3600 * 6, timeout=15, max_workers=10)` accepts:

- `ttl`: seconds a cached external link status stays valid (`0` forces a full recheck).
- `timeout`: per-request timeout, in seconds, for external links.
- `max_workers`: number of concurrent threads checking external links.

### The `check_broken_links` console script

Installed as a standard `console_scripts` entry point (`[project.scripts]` in `pyproject.toml`), so it lands in `bin/` both in a uv-managed virtualenv and in a zc.buildout instance. Since it runs Zope/ZODB code, it must be launched through `zconsole`/`instance run`, not called directly:

```shell
# uv-managed instance (Makefile: make check-broken-links)
./bin/zconsole run instance/etc/zope.conf ./.venv/bin/check_broken_links

# zc.buildout instance
./bin/instance run bin/check_broken_links
```

Options:

- `--ttl`, `--workers`, `--timeout`: same meaning as on `check_site()`.
- `--output-dir`: also dump the run as a csv in this directory (default: no csv).
- `--site-id`: id of the Plone site to check (default: the `PLONE_SITE_ID` env var, or `Plone`).
- `--url <url>`: verify a single url and log its status, without touching the site.
- `--content <path-or-UID>`: verify a single content's links and log them, without touching the site.

The script stores the report in the site, which is where the REST API endpoints below read it
from: that is the normal way to get at the result. With `--output-dir` it also writes
`<output-dir>/<siteid>_broken_links_<YYYYMMDD-HHMMSS>.csv`, byte for byte the csv
`@linkchecker-csv` serves — worth it only to keep an archive of past runs, since the site holds
the last one alone.

### The REST API endpoints

Two read-only endpoints on the site root expose the stored report, both guarded by the
`rer.linkchecker.ViewReport` permission (granted to `Manager`, `Site Administrator` and
`Editor`).

They **never run a check**: a full check takes minutes on a medium site and would time the
request out, so they only serve what the last run stored. Refresh the data out of band, with
the `check_broken_links` script from cron. Because the data is therefore asynchronous, both
responses carry the timestamp it dates from.

```shell
# the report as json, batched (plone.restapi conventions: b_start, b_size)
curl -u user:pass "$SITE/++api++/@linkchecker" -H 'Accept: application/json'

# the same report as a csv download
curl -u user:pass -OJ "$SITE/++api++/@linkchecker-csv"
```

Both accept the same filters:

- `status`: repeatable, e.g. `?status=404&status=-2` (negative values are the `STATUS_*`
  constants: `-1` timeout, `-2` https-only, `-3` connection error).
- `type`: `INTERNAL` or `EXTERNAL`.

An invalid filter value answers `400` rather than silently returning an empty report.

`@linkchecker` returns `last_update` and `duration` for the whole run, `items_total`, the
batched `items`, and a `summary` listing `{status, status_description, count}` sorted by count.
The summary is computed over the **unfiltered** report on purpose, so the counts a UI shows in
its filter chips do not move as filters are applied. When no check has ever run, `last_update`
is `null` and `items`/`summary` are empty, which a UI can tell apart from "nothing is broken".

Each item describes the page it sits on with the usual plone.restapi field names, so a client
can treat it as any other content reference, and names the link's own fields apart from those:

```json
{
  "@id": "http://site/bandi-e-avvisi",
  "@type": "Document",
  "title": "Bandi e avvisi",
  "UID": "1a568f09734340dfba2c1a53730b9cf6",
  "link": "https://unimc.it/careerday",
  "link_type": "EXTERNAL",
  "status": 404,
  "status_description": "Not Found",
  "last_update": "2026-07-30T03:00:12"
}
```

`@linkchecker-csv` returns the exact same csv the console script writes (same columns, same
quoting), names the file after the date of the *data* rather than of the download, and repeats
that date in the `X-Linkchecker-Last-Update` response header.

## Development

### Prerequisites

- An [operating system](https://6.docs.plone.org/install/create-project-cookieplone.html#prerequisites-for-installation) that runs all the requirements mentioned.
- [uv](https://6.docs.plone.org/install/create-project-cookieplone.html#uv)
- [Make](https://6.docs.plone.org/install/create-project-cookieplone.html#make)
- [Git](https://6.docs.plone.org/install/create-project-cookieplone.html#git)
- [Docker](https://docs.docker.com/get-started/get-docker/) (optional)

### Setup

```shell
git clone git@github.com:RegioneER/rer-linkchecker.git
cd rer-linkchecker/backend
make install
```

### Common tasks

```shell
make start                  # start a Plone instance on localhost:8080
make create-site            # create a new site from scratch
make check-broken-links     # run the linkchecker and write a csv report
make test                   # run the test suite
```

## Contribute

- [Issue tracker](https://github.com/RegioneER/rer-linkchecker/issues)
- [Source code](https://github.com/RegioneER/rer-linkchecker/)

## License

The project is licensed under GPLv2.

## Credits

Developed with the support of [Regione Emilia Romagna](http://www.regione.emilia-romagna.it/).

Regione Emilia Romagna supports the [PloneGov initiative](http://www.plonegov.it/).

## Authors

This product was developed by **RedTurtle Technology** team.

[![RedTurtle Technology](https://avatars1.githubusercontent.com/u/1087171?s=100&v=4)](http://www.redturtle.it/)
