Metadata-Version: 2.4
Name: dstpr
Version: 0.1.0
Summary: A high-throughput, domain-specific text preprocessing cascading pipeline filter to rank core sentences in texts and filter away boilerplate.
Author-email: Stephen Meisenbacher <sjmeis@gtgd.com>
Maintainer-email: Stephen Meisenbacher <sjmeis@gtgd.com>
License: MIT License
        
        Copyright (c) 2026 Stephen Meisenbacher
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: numpy>=1.22.0
Requires-Dist: datasets>=2.12.0
Requires-Dist: spacy<4.0.0,>=3.8.0
Dynamic: license-file

# DSTPR: Domain-specific Text Pre-processing and Ranking

[![PyPI version](https://img.shields.io/pypi/v/dstpr.svg)](https://pypi.org/project/dstpr/)
[![License](https://img.shields.io/github/license/sjmeis/DSTPR.svg)](https://github.com/sjmeis/DSTPR/blob/main/LICENSE)

A high-throughput text preprocessing pipeline designed to filter, segment, and rank core sentences from noisy, plaintext documents. 

This package strips away boilerplate, disclaimers, and application instructions *before* moving this data onto to heavier processing pipelines.

---

## Pipeline Architecture

Instead of running heavy transformer models over every sentence in a document, `DSTPR` pipes documents through progressively stricter layers:

1. **Heuristic Cleansing & Tokenization:** Uses `PySBD` (Python Sentence Boundary Disambiguation) paired with regular expressions to fix punctuation caused by flattening, then filters out sentences containing stop phrases.
2. **Semantic Section Routing:** Utilizes a lightweight encoder (Default: `all-MiniLM-L6-v2`) to dynamically find structural transitions in input texts.
3. **Hybrid Ranking:** Scores remaining sentences using a combination of **Semantics** (cosine similarity to anchors) and **Lexical Syntax** (regex-supported detection).
4. **Parallel Execution Engine:** Wraps the entire pipeline inside efficient datasets to allow for batch processing at scale, with CPU or GPU.

---

## Installation

To install the package, simply run

```bash
pip install dstpr
```

## Usage

### High-Throughput Batch Processing

This is the recommended approach for large-scale data pipelines. 

```python
from dstpr import ParallelPreprocessingPipeline, JOB_POSTING_PROFILE

pipeline = ParallelPreprocessingPipeline(profile=JOB_POSTING_PROFILE, batch_size=64)

raw_documents = [
    "DOC 1",
    "DOC 2",
    "..."
]

cleaned_documents = pipeline.process(raw_documents, num_workers=8, threshold=0.25)
```

### Advanced Usage

If you want to integrate specific pipeline layers directly into an existing workflow, or tweak the internal parameters, you can import individual modules manually:

```python
from dstpr.cleaners import clean_and_split_chunks
from dstpr.segmenters import SemanticSectionRouter
from dstpr.rankers import HybridTaskRanker
from sentence_transformers import Transformer, SentenceTransformer

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

text = "Job posting text goes here!"

# Clean and segment sentences
sentences = clean_and_split_chunks(text)

# Route by context
router = SemanticSectionRouter(model=embedding_model)
buckets = router.route_sentences(sentences)
target_sentences = buckets['CORE'] + buckets['REQUIREMENTS']

# Grade and sort text features
ranker = HybridRanker(model=shared_model)
final = ranker.rank_and_filter(target_sentences, bi_cutoff_pct=0.25, final_threshold=0.40)
```

### Parameter Tuning

To adjust the trade-off between strict filtering and execution speed, consider tweaking these variables in `ParallelPreprocessingPipeline`:
 - `threshold` (Default: `0.4`): Controls how aggressively sentences are discarded. Raising this value toward `0.6`, for example, ensures only stronger matching sentences are passed through. Lowering it toward `0.3`, on the other hand, acts as a wider net.
 - `batch_size` (Default: `256`): This depends on your hardware! Adjust for best performance.

## Creating Custom Domain Profiles (via the provided wizard)

A `DomainProfile` is required to use `DSTPR` (see the usage example). Out of the box, `dstpr` ships with pre-configured configurations for job postings (`JOB_POSTINGS_PROFILE`). However, you can easily generate an domain-specific pipeline for **any specific domain** using our built-in interactive configuration wizard.

To spin up the profile creation walkthrough, simply open your terminal and run:

```bash
task-profile-wizard
```

All profiles generated via the terminal wizard are automatically validated and written out as JSON to a local cache: `~/.config/dstpr/profiles/`
