Metadata-Version: 2.4
Name: tajiknlp
Version: 1.0.0
Summary: Production-ready NLP library for Tajik language (Cyrillic script)
Author-email: Arabov Mullosharaf Kurbonovich <cool.araby@gmail.com>
License: MIT License
        
        Copyright (c) 2026 CodeHunterOfficial
        Copyright (c) 2026 Arabov Mullosharaf Kurbonovich
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/TajikNLPWorld/tajiknlp
Project-URL: Repository, https://github.com/TajikNLPWorld/tajiknlp
Project-URL: Issues, https://github.com/TajikNLPWorld/tajiknlp/issues
Project-URL: Documentation, https://TajikNLPWorld.github.io/tajiknlp
Project-URL: Changelog, https://github.com/TajikNLPWorld/tajiknlp/blob/main/CHANGELOG.md
Keywords: tajik,nlp,natural-language-processing,tokenization,lemmatization,low-resource,cyrillic,morphology,text-processing
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: regex>=2023.0
Requires-Dist: typing-extensions>=4.0; python_version < "3.9"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.6.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4.0; extra == "dev"
Requires-Dist: pre-commit>=3.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5; extra == "docs"
Requires-Dist: mkdocs-material>=9.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24; extra == "docs"
Requires-Dist: mkdocs-autorefs>=0.5; extra == "docs"
Provides-Extra: full
Requires-Dist: tajiknlp[dev,docs]; extra == "full"
Requires-Dist: fasttext>=0.9.2; extra == "full"
Requires-Dist: numpy>=1.20; extra == "full"
Requires-Dist: scikit-learn>=1.0; extra == "full"
Requires-Dist: pandas>=1.3; extra == "full"
Provides-Extra: train
Requires-Dist: fasttext>=0.9.2; extra == "train"
Requires-Dist: gensim>=4.3; extra == "train"
Requires-Dist: torch>=2.0; extra == "train"
Requires-Dist: transformers>=4.30; extra == "train"
Dynamic: license-file

# 🇹🇯 TajikNLP

**Production-ready NLP library for the Tajik language** (Cyrillic script).

[![PyPI version](https://img.shields.io/pypi/v/tajiknlp.svg)](https://pypi.org/project/tajiknlp/)
[![Python 3.8+](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Ruff](https://img.shields.io/badge/code%20style-ruff-261230.svg)](https://github.com/astral-sh/ruff)
[![CI](https://github.com/CodeHunterOfficial/tajiknlp/actions/workflows/ci.yml/badge.svg)](https://github.com/CodeHunterOfficial/tajiknlp/actions/workflows/ci.yml)

TajikNLP provides a complete suite of text processing tools specifically designed for the unique challenges of the Tajik language (Cyrillic alphabet). Whether you're building a search engine, a chatbot, or conducting linguistic research, TajikNLP gives you robust, customizable components that work out of the box.

---

## ✨ Features (v1.0.0)

### 🧹 Text Cleaning
- Remove URLs, emails, HTML tags, mentions, and hashtags
- Strip invisible Unicode characters and normalize whitespace
- Smart bracket cleaning (empty brackets and punctuation‑only brackets are removed)
- Remove spaces before punctuation marks

### 🔤 Normalization
- Tajik Cyrillic standardization (e.g., `љ` → `ҷ`, `ї` → `ӣ`)
- Persian/Arabic digit conversion (`۱۲۳` → `123`)
- Quote and dash normalization (`« »` → `"`, `—` → `-`)
- Optional lowercasing and custom character replacement maps

### ✂️ Tokenization
- Regex‑based word tokenizer aware of Tajik‑specific letters (`ғ`, `ӣ`, `қ`, `ӯ`, `ҳ`, `ҷ`)
- Handles hyphenated compounds (`китоб-ҳо`)
- Preserves punctuation as separate tokens (optional)
- Provides accurate character offsets for each token
- Morpheme-level tokenization for linguistic analysis

### 📝 Sentence Splitting
- Rule-based sentence boundary detection
- Protects Tajik-specific abbreviations (`т.д.`, `и.т.п.`, `ва ғ.`)
- Handles initials, decimals, dates, URLs, and emails

### 🏷️ Part‑of‑Speech Tagging
- Rule‑based tagger with dictionary lookup and morphological fallback
- Category priority system from suffix rules
- Works on raw text without prior annotation

### 🔍 Morpheme Segmentation
- Splits words into prefixes, root, and suffixes
- Uses linguistic rules from external JSON files (fully customizable)
- Returns structured morpheme data for each token

### 📚 Lemmatization
- Hybrid approach: direct dictionary lookup + iterative affix stripping
- POS‑aware suffix removal for higher accuracy
- Built‑in dictionary of irregular forms

### 🌿 Stemming
- Conservative-to-aggressive rule-based stemmer
- Safe mode for grammatical reductions
- Deep mode for derivational stripping
- Corpus-aware scoring for better accuracy

### 🛑 Stop Words Filtering
- Comprehensive stop word list (100+ items)
- Automatically removes function words while preserving content words

### 🎯 Named Entity Recognition (NER) Ready
- Character-level span alignment
- BIO tag generation for entity spans
- Offset preservation for training data preparation

### 🛠️ Utilities
- **Script detection** – identify if text is Cyrillic, Latin, Arabic, or mixed
- **Language detection** – distinguish Tajik from Russian, Persian, English, Arabic
- **Quality scoring** – evaluate how "Tajik‑like" a piece of text is
- **Validation helpers** – quickly check if a string is valid Tajik text

---

## 📦 Installation

TajikNLP requires Python 3.8 or later.

### Basic Installation

```bash
pip install tajiknlp
```

### Development Installation

```bash
pip install "tajiknlp[dev]"
```

### Full Installation (with ML dependencies)

```bash
pip install "tajiknlp[full]"
```

---

## 🚀 Quick Start

### Load the default pipeline

```python
from tajiknlp import load_pipeline

# Create a pipeline with cleaner, normalizer, tokenizer, POS tagger,
# morpheme splitter, stopword filter, and lemmatizer.
pipe = load_pipeline("default")

text = "Китобҳоямонро хондам, аммо нафаҳмидам."
doc = pipe(text)

# Inspect the results
for token in doc.tokens:
    print(f"{token.text:<15} → lemma: {token.lemma:<10} pos: {token.pos}")
```

**Output:**
```
китобҳоямонро   → lemma: китоб      pos: NOUN
хондам          → lemma: хондан     pos: VERB
,               → lemma: ,          pos: PUNCT
аммо            → lemma: аммо       pos: CONJ
нафаҳмидам      → lemma: фаҳмидан   pos: VERB
.               → lemma: .          pos: PUNCT
```

### Available Pipeline Presets

| Preset | Components |
|--------|------------|
| `minimal` | Cleaner + Tokenizer + Morpheme splitter |
| `default` | Full preprocessing + POS + Morphemes + Stopwords + Lemmatizer |
| `full` | Alias for `default` |
| `stemming` | Same as default but with stemmer instead of lemmatizer |
| `ner` | Optimized for NER (preserves case) |

### Use individual components

```python
from tajiknlp import Doc
from tajiknlp.components.tokenizers import RegexTokenizer

tokenizer = RegexTokenizer(lowercase=True, keep_punct=True)
doc = tokenizer(Doc(text="Китоб-ҳо хондам!"))
print([t.text for t in doc.tokens])
# Output: ['китоб-ҳо', 'хондам', '!']
```

### Validate and score text quality

```python
from tajiknlp import quality_score, detect_script

score = quality_score("Ман китоб хондам.")
print(f"Score: {score['score']}")        # e.g., 0.78
print(f"Valid: {score['is_valid']}")     # True
print(f"Language: {score['language']}")  # tajik

script = detect_script("Салом!")
print(script)  # Script.CYRILLIC
```

### Custom pipeline

```python
from tajiknlp import TajikPipeline
from tajiknlp.components.cleaners import TextCleaner
from tajiknlp.components.tokenizers import RegexTokenizer
from tajiknlp.components.stemmers import DictStemmer

# Build custom pipeline
pipe = TajikPipeline()
pipe.add_component(TextCleaner())
pipe.add_component(RegexTokenizer())
pipe.add_component(DictStemmer(deep_stemming=True))

doc = pipe("Ман китобҳоро хондам.")
```

---

## 📖 Documentation

Full documentation is available at **[CodeHunterOfficial.github.io/tajiknlp](https://CodeHunterOfficial.github.io/tajiknlp)**.

It includes:
- Detailed API references
- Component configuration guides
- Advanced usage examples
- NER and alignment tutorials

---

## 📁 Project Structure

```
tajiknlp/
├── src/tajiknlp/           # Main package source
│   ├── alignment/          # Span alignment utilities
│   ├── components/         # NLP components
│   │   ├── cleaners/       # Text cleaning
│   │   ├── embeddings/     # Word embeddings
│   │   ├── filters/        # Token filtering
│   │   ├── lemmatizers/    # Lemmatization
│   │   ├── normalizers/    # Text normalization
│   │   ├── sentencizers/   # Sentence splitting
│   │   ├── stemmers/       # Stemming
│   │   ├── taggers/        # POS tagging
│   │   └── tokenizers/     # Tokenization
│   ├── core/               # Base classes
│   ├── data/               # Static resources
│   ├── pipeline/           # Pipeline system
│   ├── resources/          # Resource management
│   └── utils/              # Utility functions
├── tests/                  # Test suite
├── examples/               # Usage examples
├── docs/                   # Documentation
└── pyproject.toml          # Project configuration
```

---

## 🧪 Running Tests

If you cloned the repository, you can run the test suite with:

```bash
# Install development dependencies
pip install -e ".[dev]"

# Run all tests
pytest tests/ -v

# Run with coverage report
pytest tests/ --cov=tajiknlp --cov-report=html
```

---

## 🤝 Contributing

We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on:

- Setting up the development environment
- Coding standards and style guides
- Testing requirements
- Pull request process

---

## 📄 License

This project is licensed under the MIT License – see the [LICENSE](LICENSE) file for details.

Copyright (c) 2026 CodeHunterOfficial
Copyright (c) 2026 Arabov Mullosharaf Kurbonovich

---

## 🙏 Acknowledgements

TajikNLP was inspired by the need for robust, open‑source NLP tools for low‑resource languages. Special thanks to all contributors and the Tajik linguistic community.

---

## 📊 Citation

If you use TajikNLP in your research, please cite:

```bibtex
@software{tajiknlp2026,
  author = {Arabov, Mullosharaf Kurbonovich},
  title = {TajikNLP: Production-ready NLP library for Tajik language},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/CodeHunterOfficial/tajiknlp}
}
```

---

**Made with ❤️ for the Tajik language.**
