Metadata-Version: 2.4
Name: subsegmenter
Version: 0.1.2
Summary: A lightweight semantic sub-sentence segmenter using rule-optimized spaCy dependency trees.
Author-email: Stephen Meisenbacher <sjmeis@gtgd.com>
License: MIT
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: spacy<4.0.0,>=3.7.0
Description-Content-Type: text/markdown

# SubSegmenter

A lightweight Python library built to segment dense text down to atomic semantic units. Unlike traditional tokenizers (like `nltk`) that stop strictly at the sentence level, `SubSegmenter` resolves shared dependencies and breaks sentences apart at clausal and gerund breakpoints without losing context words or language.

## Core Features
- **Shared Dependency Resolution:** Expands coordinated structures smoothly (e.g., *"design and lead projects"* becomes *"design projects"*, *"lead projects"*).
- **High-Performance Execution:** Streamlined to run entirely on the CPU.
- **Fast Text Pre-Cleaning:** Built-in optimized regex filters to strip HTML markup, metadata, and other junk before parsing.
- **Domain-Agnostic:** Runs completely based on linguistic parsing. 

## Installation
```bash
pip install subsegmenter
```

Also make sure to have the appropriate `SpaCy` model downloaded:

```bash
python3 -m spacy download en_core_web_sm
```

## Quick Start
Processing any document is easy:

```python
from subsegmenter import SubSegmenter

text = "Organizes, coordinates and mentors the efforts of project engineers supporting the program. Maintains consistency with applicable standards, procedures, and implementation methodologies."

segmenter = SubSegmenter()
results = segmenter.segment(text)
```

If you would like to process a large batch of (large) documents, use the following:

```python
from subsegmenter import process_pipeline

documents = [text] * 1000

batch_outputs = process_pipeline(documents)
```