nltk-punkt
Copyright 2026 alvations

This product includes software developed by the NLTK Project
(https://www.nltk.org/), licensed under the Apache License, Version 2.0.

--------------------------------------------------------------------------
Algorithm
--------------------------------------------------------------------------

The sentence boundary detection algorithm implemented here is described in:

    Kiss, Tibor and Strunk, Jan (2006).
    Unsupervised Multilingual Sentence Boundary Detection.
    Computational Linguistics 32(4): 485-525.

The log-likelihood ratio test it relies on is from:

    Dunning, Ted (1993).
    Accurate Methods for the Statistics of Surprise and Coincidence.
    Computational Linguistics 19(1): 61-74.

--------------------------------------------------------------------------
Pretrained models
--------------------------------------------------------------------------

The model data compiled into src/punkt/data/ is derived from the pretrained
Punkt models redistributed by the NLTK Project as the "punkt_tab" data
package, under the Apache License, Version 2.0.

Those models were prepared by Jan Strunk and Tibor Kiss using the test
corpora from Kiss & Strunk (2006), with additional models contributed by
members of the NLTK community. The conversion performed by this project is
lossless: tools/build_data.py rewrites the same abbreviation, collocation,
sentence-starter and orthographic-context tables as Python literals, without
altering any value.

Training corpora, per the NLTK punkt README:

    Czech        Multilingual Corpus 1 (ECI) - Lidove Noviny, Literarni Noviny
    Danish       Avisdata CD-Rom Ver. 1.1 (Berlingske Avisdata, Copenhagen)
    Dutch        Multilingual Corpus 1 (ECI) - Algemeen Dagblad
    English      Wall Street Journal (Penn Treebank)
    Estonian     University of Tartu corpora
    Finnish      Finnish Text Collection
    French       Multilingual Corpus 1 (ECI) - Le Monde
    German       Neue Zuercher Zeitung
    Greek        Multilingual Corpus 1 (ECI)
    Italian      Multilingual Corpus 1 (ECI)
    Malayalam    community contribution
    Norwegian    Bergen Corpus of Norwegian
    Polish       community contribution
    Portuguese   community contribution
    Russian      community contribution
    Slovene      community contribution
    Spanish      Multilingual Corpus 1 (ECI)
    Swedish      Multilingual Corpus 1 (ECI)
    Turkish      community contribution

See the NLTK punkt README for the full table, including corpus sizes and
individual contributors.

--------------------------------------------------------------------------
Reference implementation
--------------------------------------------------------------------------

This is an independent reimplementation. It was written against, and is
tested for output parity with, NLTK's nltk/tokenize/punkt.py, whose authors
are credited there as Willy, Steven Bird, Edward Loper, Joel Nothman,
Arthur Darcet and Tom Aarsen.
