Metadata-Version: 2.4
Name: sentencesplit
Version: 0.0.1
Summary: sentencesplit is a rule-based sentence boundary detection library, derived from pySBD (Python Sentence Boundary Disambiguation), that works out-of-the-box across many languages.
Keywords: natural-language-processing,nlp
Author: Nipun Sadvilkar, Yi Ding
Author-email: Nipun Sadvilkar <nipunsadvilkar@gmail.com>, Yi Ding <yi.s.ding@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: blingfire>=0.1.8 ; extra == 'benchmark'
Requires-Dist: nltk>=3.9 ; extra == 'benchmark'
Requires-Dist: spacy>=3.8 ; extra == 'benchmark'
Requires-Dist: stanza>=1.8 ; extra == 'benchmark'
Requires-Dist: syntok>=1.4.4 ; extra == 'benchmark'
Requires-Dist: python-semantic-release>=10,<11 ; extra == 'release'
Requires-Python: >=3.11
Project-URL: Repository, https://github.com/yisding/sentencesplit
Provides-Extra: benchmark
Provides-Extra: release
Description-Content-Type: text/markdown

# sentencesplit

> **Derived from [pySBD](https://github.com/nipunsadvilkar/pySBD)** (Python Sentence Boundary Disambiguation) by Nipun Sadvilkar.

sentencesplit is a rule-based sentence boundary detection library that works out-of-the-box across many languages.

This project is a direct port of ruby gem - [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter) which provides rule-based sentence boundary detection.

## Install

**Python 3.11+ required.**

    pip install sentencesplit

## Usage

-   Currently sentencesplit supports 24 languages.

```python
import sentencesplit
text = "My name is Jonas E. Smith. Please turn to p. 55."
seg = sentencesplit.Segmenter(language="en", clean=False)
print(seg.segment(text))
# ['My name is Jonas E. Smith.', 'Please turn to p. 55.']
```

```python
import sentencesplit

seg = sentencesplit.Segmenter(language="en", clean=False)

result = seg.segment_with_lookahead("The model is GPT 3.")
print(result.segments)
# ['The model is GPT 3.']
print(result.should_wait_for_more)
# True

print(seg.should_wait_for_more("This is the finale."))
# False
```

### Why lookahead uses probes

`should_wait_for_more()` answers a *counterfactual* question: "if more characters arrived, would the last boundary stay a boundary?"  
The existing regex/rule pipeline is deterministic for the text you pass in, but it cannot directly encode every continuation-sensitive case in one pass (for example `"Dr."` vs `"This is the finale."`, numeric endings, and language-specific abbreviation/starter interactions) without effectively duplicating parser state.

Instead, lookahead appends a few tiny probe suffixes and re-runs segmentation to see whether the final segment remains stable. If adding plausible continuation tokens changes that final boundary, we return `should_wait_for_more=True`; otherwise `False`.

To keep this efficient for streaming inputs, lookahead probes only the located tail segment when possible and falls back to full-text probing only when needed.

-   Use `sentencesplit` as a [spaCy](https://spacy.io/usage/processing-pipelines) pipeline component. (recommended)</br>Please refer to example [sentencesplit\_as\_spacy\_component.py](examples/sentencesplit_as_spacy_component.py)
- Use sentencesplit through [entry points](https://spacy.io/usage/saving-loading#entry-points-components)

```python
import spacy

nlp = spacy.blank('en')

# add sentencesplit component registered via package entry points
nlp.add_pipe("sentencesplit")

doc = nlp('My name is Jonas E. Smith. Please turn to p. 55.')
print(list(doc.sents))
# [My name is Jonas E. Smith., Please turn to p. 55.]

```

## Multi-language segmentation

Languages with similar writing systems (e.g. English, Spanish, French) can be combined into a single segmenter by merging their abbreviation lists. This avoids needing to detect the language of each sentence before segmenting.

```python
import sentencesplit
from sentencesplit.abbreviation_replacer import AbbreviationReplacer
from sentencesplit.lang.common import Common, Standard
from sentencesplit.lang.english import English
from sentencesplit.lang.spanish import Spanish
from sentencesplit.lang.french import French
from sentencesplit.languages import LANGUAGE_CODES

class MultiLang(Common, Standard):
    iso_code = 'multi'

    class Abbreviation(Standard.Abbreviation):
        ABBREVIATIONS = sorted(set(
            Standard.Abbreviation.ABBREVIATIONS +
            Spanish.Abbreviation.ABBREVIATIONS +
            French.Abbreviation.ABBREVIATIONS
        ))
        PREPOSITIVE_ABBREVIATIONS = sorted(set(
            Standard.Abbreviation.PREPOSITIVE_ABBREVIATIONS +
            Spanish.Abbreviation.PREPOSITIVE_ABBREVIATIONS +
            French.Abbreviation.PREPOSITIVE_ABBREVIATIONS
        ))
        NUMBER_ABBREVIATIONS = sorted(set(
            Standard.Abbreviation.NUMBER_ABBREVIATIONS +
            Spanish.Abbreviation.NUMBER_ABBREVIATIONS +
            French.Abbreviation.NUMBER_ABBREVIATIONS
        ))

    class AbbreviationReplacer(AbbreviationReplacer):
        SENTENCE_STARTERS = English.AbbreviationReplacer.SENTENCE_STARTERS

LANGUAGE_CODES['multi'] = MultiLang

seg = sentencesplit.Segmenter(language="multi", clean=False)
print(seg.segment("Hola Srta. Ledesma. How are you?"))
# ['Hola Srta. Ledesma.', 'How are you?']
```

This works well for languages that share the `Common` and `Standard` base classes and use the same sentence-ending punctuation (`.`, `!`, `?`). The same pattern can be extended to other similar languages like Italian, Dutch, or Danish. Languages with different writing systems or punctuation (e.g. Japanese, Arabic) would need a different approach.

## Releasing

Releases are published manually from GitHub Actions.

One-time setup:

1. In GitHub, create an environment named `pypi`.
2. In PyPI, add a Trusted Publisher for repo `yisding/sentencesplit`, workflow `.github/workflows/publish.yml`, and environment `pypi`.

Release steps:

1. Merge the code you want to publish into `main`.
2. Open GitHub Actions and run the `Release` workflow on `main`.
3. Choose the version bump: `patch`, `minor`, `major`, or `prerelease`.
4. Set `dry_run=true` to preview the release, then run it again with `dry_run=false` for the real release.
5. The workflow creates the version commit, tag, changelog update, and GitHub Release.
6. The new `v*` tag automatically triggers the `Publish to PyPI` workflow, which uploads the built distributions using Trusted Publishing.

`python-semantic-release` still uses Conventional Commits to generate changelog entries cleanly, so commit messages like `fix: ...`, `feat: ...`, and `feat!: ...` are still recommended even though releases are now triggered manually.

## Contributing

If you want to contribute new feature/language support or found a text that is incorrectly segmented, then please head to [CONTRIBUTING.md](CONTRIBUTING.md) to know more and follow these steps.

1.  Fork it
2.  Create your feature branch (`git checkout -b my-new-feature`)
3.  Commit your changes (`git commit -am 'Add some feature'`)
4.  Push to the branch (`git push origin my-new-feature`)
5.  Create a new Pull Request

## Citation
This project is derived from pySBD. If you use it in your projects or research, please cite the original [PySBD: Pragmatic Sentence Boundary Disambiguation](https://www.aclweb.org/anthology/2020.nlposs-1.15) paper.
```
@inproceedings{sadvilkar-neumann-2020-pysbd,
    title = "{P}y{SBD}: Pragmatic Sentence Boundary Disambiguation",
    author = "Sadvilkar, Nipun  and
      Neumann, Mark",
    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.nlposs-1.15",
    pages = "110--114",
    abstract = "We present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter which we ported to Python with additional improvements and functionality. PySBD passes 97.92{\%} of the Golden Rule Set examplars for English, an improvement of 25{\%} over the next best open source Python tool.",
}
```

## Credit

This project is derived from [pySBD](https://github.com/nipunsadvilkar/pySBD) by Nipun Sadvilkar, which itself wouldn't be possible without the great work done by the [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter) team.
