Metadata-Version: 2.4
Name: wikiloc
Version: 1.0.0
Summary: Parse in-source locators (page, pages, chapter, quote, ...) from Wikipedia citation wikitext.
Author-email: Mario Morvan <contact@mariomorvan.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/mrmvn/wikiloc
Project-URL: Repository, https://github.com/mrmvn/wikiloc
Project-URL: Changelog, https://github.com/mrmvn/wikiloc/blob/main/CHANGELOG.md
Keywords: wikipedia,citations,wikitext,locators,verifiability
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: mwparserfromhell>=0.6; extra == "test"
Provides-Extra: extract
Requires-Dist: mwparserfromhell>=0.6; extra == "extract"
Dynamic: license-file

# wikiloc

Extract citations and parse **in-source locators** from wikitext.

An *in-source location* is the specific part of a source that supports a
statement — a page, a page range, a chapter, a quotation.... Editors record
these through dedicated citation-template parameters (`page`, `pages`, `chapter`,
`quote`, …), which we call **locators**. `wikiloc` reads the locators (and
identifiers) declared by references, using the exact set of parameters each
citation template authorises.

- Zero runtime dependencies (Python standard library only) for parsing, optional mwparserfromhell for ref extraction from wikitext.
- Three levels: a single reference, an article's reference list, or whole-article wikitext.
- Only English has been tested, though more languages can be added.

The parser is regression-tested against a golden set of 160 hand-annotated citations (English Wikipedia, 2014 and 2026 snapshots, stratified by template and locator type) in `wikiloc/tests/test_golden.py`.
For each row the test asserts the exact set of locator types, so precision and recall for locator presence and type are 100% on the set; it also asserts the `page`/`pages` value (when one is expected) and the `compute_located_pages()` count. Four rows where the parser's output differs from the human gold — noisy page values (`pages 32`, a page plus an external link), an unparseable range (`4B`), and an HTML entity (`&ndash;`) — assert the parser's value instead, each with a note recording the mismatch.

## Install

```bash
pip install wikiloc            # core (stdlib only)
pip install wikiloc[extract]   # + whole-article extraction (mwparserfromhell)
```

## Quickstart

**A single reference:**

```python
from wikiloc import parse

parse("{{cite book|title=X|isbn=978-0-13-468599-1|page=42}}")
# {'cite_type': 'cite book', 'locators': {'page': '42'}, 'ids': {'isbn': '978-0-13-468599-1'}}

parse("<ref name=\"a\">{{cite book|title=B|page=87}}</ref>{{rp|page=92}}")
# {'cite_type': 'cite book', 'locators': {'page': '92'}, 'ids': {}}
```

**A list of references from one article** (resolved with named-reference
inheritance):

```python
parse(["<ref name=a>{{cite book|page=10}}</ref>", "<ref name=a/>{{rp|13}}"])
# second dict: {'cite_type': 'cite book', 'ref_type': 'repeated_rp', 'page': '13', ...}
```

**Whole-article wikitext** (extraction + resolution; needs `[extract]`):

```python
from wikiloc import parse_article

parse_article(article_wikitext)
# [{'ref_type': 'main', 'page': '42', ...}, ...]
```

### Output

`parse()` on a single reference returns:

| key         | meaning                                                              |
|-------------|---------------------------------------------------------------------|
| `cite_type` | normalized template name (e.g. `cite book`) or `None`               |
| `locators`  | in-source locators found, keyed by their literal parameter name     |
| `ids`       | identifiers found (`isbn`, `doi`, `pmid`, `pmc`, `arxiv`)           |
| `ref_key`   | short-cite key for `{{r}}`/`{{sfn}}`/`{{sfnp}}` (only when present)  |

Locators keep their literal parameter key (e.g. `p`, `pp`); canonicalising
aliases onto `page`/`pages` is left to the caller. The one exception is
numbered `{{r}}` aliases (`p1`/`1p` → `page`), which have no literal
counterpart.

> **The return shape depends on the input.** A single reference string returns
> the grouped dict above; a list (article-level `parse()`) returns **flat**
> records with the locator keys at the top level (`{'cite_type': …, 'page': '42',
> …}`, plus `ref_type`/`ref_name`). Use `wikiloc.locators_of(record)` to read the
> locator map from either shape:
>
> ```python
> from wikiloc import parse, locators_of
> locators_of(parse("{{cite book|page=42}}"))          # {'page': '42'}
> locators_of(parse(["{{cite book|page=42}}"])[0])     # {'page': '42'}
> ```
>
> A **bundled `{{r}}`** returns a *list* of grouped dicts, one per citation
> (see *Supported reference forms*). Article-level `parse(list)`/
> `parse_article()` likewise return one flat record per citation, and
> `wikiloc.parse_citations(record)` exposes the per-citation view for a single
> reference record.

### Supported reference forms

`{{cite …}}` / `{{citation}}` (CS1/CS2), `{{rp}}`, `{{r}}`, `{{sfn}}`,
`{{sfnp}}`, Harvard short cites (`{{harvnb}}` and variants), `<ref>…</ref>`
tags (optionally followed by an `{{rp|…}}`), and untemplated (plain-text)
references. For untemplated text, only pages/page-ranges introduced by an
explicit marker (`p.`, `pp.`, `page`, `pages`, `pg`, `pgs`) are read, so years,
scores and dates are not mistaken for page numbers.

`{{r}}` accepts Template:R's full parameter set — including `at`/`loc`, the
quote family, and the numbered/named aliases. A `{{r}}` can **bundle** up to
nine references in one call (`{{r|A|B|C}}` or
`{{r|1=A|p1=10|2=B|pp2=20}}`); each bundled citation becomes its own record, so
the package keeps **one record per citation**. Bundling is not expanded when
the `{{r}}` is wrapped inside a `<ref>` tag.

### Short-cite (CITEREF) resolution

At article level, `parse()` on a list and `parse_article()` resolve shortened
footnotes (`{{sfn}}`, `{{sfnp}}`, the Harvard family) and `{{r}}` keys against
full CS1/CS2 citations.

`{{sfn}}`/`{{sfnp}}`/Harvard short cites link through their HTML anchor: the
explicit `|ref=` value or the auto-generated `CITEREF<last-names><year>` id
(built from up to four author last names — editors when there is no author —
plus the year from `|year=` or `|date=`, mirroring Module:Citation/CS1). A
matching short cite inherits the full citation's `cite_type` and identifiers
(`isbn`, `doi`, …), so enrichment follows the link.

`{{r}}` keys follow Template:R, which links to a *named reference* (`{{r|RefName}}`
is equivalent to `<ref name="RefName"/>`). A key therefore resolves first
through named-reference inheritance, then — as a heuristic — against an
explicit `|ref=` value or the auto-generated `CITEREF<key>` anchor.

```python
parse_article(
    "Text.{{sfn|Smith|2020|p=3}}\n"
    "<ref>{{cite book|last=Smith|year=2020|isbn=978-0-13-468599-1|pages=100-150}}</ref>"
)
# the sfn record inherits cite_type='cite book', isbn=… and pages='100-150'
```

`|ref=none` disables the anchor, `|ref=harv` forces the auto id, any other
`|ref=` value is the literal id, and `{{sfnref|…}}`/`{{harvid|…}}` expand to
CITEREF ids.


## Flags

Opt-in issue detectors help audit citations. Each accepts either a `parse()`
result or a flat locator record:

- `wikiloc.detect_locator_issues(parsed_ref)` — page/place locators;
- `wikiloc.detect_quote_issues(parsed_ref)` — quotation locators;
- `wikiloc.detect_issues(parsed_ref, include=("page", "quote"))` — both, merged.

### Page/place flags

- `no_locator` — the reference declares no locator at all;
- `pages_single` — a `pages`/`pp` value is a single page number (often a total-page count);
- `page_range` — a `page`/`p` value holds a range or comma list (probably belongs in `pages`);
- `page_unparseable` / `pages_unparseable` — the value could not be parsed;
- `page_huge` — a `page`/`p` number exceeds `PAGE_HUGE_THRESHOLD` (default 99999);
- `pages_range_huge` — a `pages`/`pp` range exceeds `PAGES_RANGE_HUGE_THRESHOLD` (default 999);
- `page_reversed_range` / `pages_reversed_range` — a range runs backwards (`hi < lo`);
- `page_noisy` — a parseable `page`/`p` value also carries prose (e.g. `200–201 & sketch 19`);
- `page_outside_pages` — `page`/`p` falls outside the `pages`/`pp` range;
- `multiple_page_locators` — more than one of `page`/`p`, `pages`/`pp` and `at`
  (CS1 allows exactly one; the module flags the extras as `Extra |pages= or
  |at=`). Short cites are exempt: in `{{sfn}}`/`{{sfnp}}`/Harvard, `at`/`loc`
  *supplements* `p`/`pp`, so only the page-vs-pages choice is exclusive there.

### Quotation flags

`detect_quote_issues` mirrors the above for the quote family:

- `quote_alias` / `quote_conflicting_aliases` — non-canonical spelling (`qp`,
  `qpp`, `q`, `quotation`…) or two aliases that disagree;
- `quote_unsupported_param` — a quote parameter the template does not recognise
  (e.g. `qp` on a CS1 `cite book`, though it is valid on `{{rp}}`);
- `quote_pages_single` / `quote_page_range` — value in the wrong quote-page
  vs quote-pages parameter;
- `quote_page_both`, `quote_location_and_quote_page` — mutually exclusive
  quote locators together;
- `quote_page_redundant` — `quote-page` duplicates the cited page span exactly;
- `quote_page_outside_cited` / `quote_pages_outside_cited` — the quote locator
  spills outside the cited `page`/`pages` span (CS1 says it should be a subset);
- `quote_page_unparseable` / `quote_pages_unparseable`,
  `quote_page_reversed_range` / `quote_pages_reversed_range`,
  `quote_page_huge` / `quote_pages_range_huge`;
- `quote_page_without_quote`, `quote_without_locator` — a `quote-page` with no
  quote, or a quote with no locating parameter at all;
- `quote_markup` — wikitext/HTML left inside the quote;
- `quote_excessive` — quote text longer than `QUOTE_EXCESSIVE_THRESHOLD`.

A differing `quote-page` is *not* flagged on its own — the parameter exists to
differ from `page`.

Flags are **not** run by default, so `parse()` output stays a faithful
transcription of the source. Opt in with `with_flags=True`:

```python
parse("{{cite book|title=X|pages=240}}", with_flags=True)
# {'cite_type': 'cite book', 'locators': {'pages': '240'}, 'ids': {}, 'flags': ['pages_single']}
```

For a list of references, each resolved record gets its own `flags` key.
`with_flags=True` runs the **page/place** detector only; quote flags are opted
into separately (`detect_quote_issues(record)` or `detect_issues(record)`).
Thresholds live in `wikiloc._grammar` (`PAGE_HUGE_THRESHOLD`,
`PAGES_RANGE_HUGE_THRESHOLD`, re-exported via `wikiloc.parser`) and
`QUOTE_EXCESSIVE_THRESHOLD` in `wikiloc.flags`.


## Located-page counting

`wikiloc.compute_located_pages` turns a reference's `page`/`pages`
locators into a page count. It accepts either a `parse()` result or a flat
locator record.

- a single number counts as **1 page**;
- a `page`/`p` locator counts as **1 page** even if its value is malformed
  (`detect_locator_issues` reports those);
- a range `a–b` counts as `b − a + 1` pages, with abbreviated ends expanded
  (`446–52` → pp. 446–452, 7 pages);
- a reversed range counts as 1;
- a comma-separated list of page numbers counts its items;
- section-style ranges such as `S1–S5` count their members;
- an unparseable `pages`/`pp` value is treated as absent;
- when `page` and `pages` coexist, the **minimum** is used.

Quote/chapter/`at`/`loc` and other non-paginated locators are deliberately
excluded: the package stays unopinionated about how much they narrow a source.

### What the count does not cover

`compute_located_pages` is an estimate, not an exact page count:

- only `page`/`pages` (`p`/`pp`) are counted — a page number buried in an
  `at`/`loc` value (e.g. `at=cc1278-84`) is not;
- a single `pages`/`pp` number counts as **1** page even when it is really a
  total-page count rather than a pin (see the `pages_single` flag);
- malformed values are absorbed, not surfaced: an unparseable `page`/`p` still
  counts as 1, while one bad item in a `pages`/`pp` list drops the whole list
  (treated as absent);
- a reversed range counts as 1;
- when `page` and `pages` coexist, only the minimum is returned;
- Roman numerals are accepted only in the strict 1–4999 grammar (`IIII`, `VX`
  are rejected).

Pair the count with `detect_locator_issues` when these cases matter.


## Quotation helpers

`wikiloc.quotes` is a pure, offline layer for quote locators (its issue flags
live with the other detectors, above).

**Canonicalisation.** `quote_locators(parsed_ref)` collapses the many quote
aliases onto `quote`, `quote-page`, `quote-pages` and `quote-location` (French
`extrait`/`citation` → `quote`):

```python
from wikiloc import quote_locators
quote_locators({"locators": {"q": "To be", "qp": "12"}})
# {'quote': 'To be', 'quote-page': '12'}
```

**Plain text.** `quote_plaintext()` strips wikitext/HTML from a quote, and
`quote_search_fragments()` splits it on `[...]`/`...` omissions into searchable
runs.

**Source URL deep-links.** `citation_url(wikitext)` extracts a citation's `url`
(or `chapter-url` …), and `improve_quote_url(url, quote)` appends the anchor
that jumps to the quotation — `dq=` for Google Books, `?q=` (or the legacy
`#search/`) for the Internet Archive — leaving a URL that already searches
untouched:

```python
from wikiloc import citation_url, improve_quote_url
url = citation_url("{{cite book|url=https://books.google.com/books?id=abc|title=X}}")
improve_quote_url(url, "MIT for a long time stood virtually alone as a university")
# https://books.google.com/books?id=abc&dq=MIT+for+a+long+time+stood+virtually+alone+as+a+university
```

Locating a source that is not yet linked is a network concern and stays out of
the package (see `tools/`).


## Tools

Standalone helpers built on `wikiloc` live in
[`tools/`](https://github.com/mrmvn/wikiloc/tree/main/tools); they are not part
of the published package (they are in neither the wheel nor the source
distribution, and may pull in `mwparserfromhell`/`pywikibot`). See
[`tools/README.md`](https://github.com/mrmvn/wikiloc/blob/main/tools/README.md)
and each script docstring and `--help` for details.


## Extending

The locator vocabulary and the flag rules live in `wikiloc/constants.py`:

- **New language** — add a `LOC_PARAMS[lang]` mapping of template name →
  locators, plus a `LOCATOR_ROLES[lang]` table mapping each literal key to a
  canonical role (`page`, `pages`, `place`, `quote`, …). Only `en` is populated
  today; French quote keys are bridged by `FOREIGN_QUOTE_TEXT_KEYS` until a
  full `fr` role table lands.
- **New locator** — append the parameter alias to the relevant template's list
  and give it a role in `LOCATOR_ROLES`.
- **New family rule** — add a `LOCATOR_RULES[family]` entry (which roles are
  mutually exclusive, which should be subsets) for a new citation family.
- **New identifier** — add an alias → canonical mapping in `ID_ALIASES` (and, for
  free-text detection, a pattern in `_extract_ids_from_text`).

The parser, the flag detectors (`wikiloc/flags.py`) and the quotation helpers
(`wikiloc/quotes.py`) share the page grammar and record accessors in the private
`wikiloc/_grammar.py`.

## AI-Assisted Development

Portions of this codebase were generated or refactored with AI assistance:
- [Pi](https://pi.dev/) (coding agent)
- DeepSeek V4 Flash (open-weight model)

The project concept, architecture decisions, and final review are the author's own.

## Citation

If you use `wikiloc` in academic work, please cite the accompanying paper:


```bibtex
@inproceedings{morvan2026citation,
  author    = {Morvan, Mario},
  title     = {Citation Location Needed: In-Source Locators and Wikipedia's Growing Verification Cost},
  booktitle = {The 2026 ACM/IEEE Joint Conference on Digital Libraries (JCDL '26)},
  series    = {JCDL '26},
  year      = {2026},
  month     = oct,
  location  = {Frisco, TX, USA},
  publisher = {ACM},
  address   = {New York, NY, USA},
  isbn      = {979-8-4007-2597-5},
  doi       = {10.1145/3805696.3846057},
  note      = {In press}
}
```

## License

MIT License © 2026 Mario Morvan — see
[LICENSE](https://github.com/mrmvn/wikiloc/blob/main/LICENSE) for details.
