Metadata-Version: 2.5
Name: textinsight-nlp
Version: 1.0.0
Summary: Lightweight, offline-first NLP for English and Hinglish text, with pluggable backends.
Project-URL: Homepage, https://github.com/harsh01103/TextInsight
Project-URL: Documentation, https://github.com/harsh01103/TextInsight/tree/main/docs
Project-URL: Source, https://github.com/harsh01103/TextInsight
Project-URL: Issues, https://github.com/harsh01103/TextInsight/issues
Project-URL: Changelog, https://github.com/harsh01103/TextInsight/blob/main/CHANGELOG.md
Author: Harsh Vardhan Ajmera
License-Expression: MIT
License-File: LICENSE
Keywords: code-mixed,hinglish,keywords,nlp,sentiment,summarization,text-analysis
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Natural Language :: Hindi
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: anthropic>=2.0; extra == 'all'
Requires-Dist: onnxruntime>=1.18; extra == 'all'
Requires-Dist: pandas>=2.2; extra == 'all'
Requires-Dist: scikit-learn>=1.5; extra == 'all'
Requires-Dist: tokenizers>=0.19; extra == 'all'
Requires-Dist: torch>=2.2; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Provides-Extra: demo
Requires-Dist: streamlit>=1.40; extra == 'demo'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Provides-Extra: llm
Requires-Dist: anthropic>=2.0; extra == 'llm'
Provides-Extra: onnx
Requires-Dist: onnxruntime>=1.18; extra == 'onnx'
Requires-Dist: tokenizers>=0.19; extra == 'onnx'
Provides-Extra: pandas
Requires-Dist: pandas>=2.2; extra == 'pandas'
Provides-Extra: sklearn
Requires-Dist: scikit-learn>=1.5; extra == 'sklearn'
Provides-Extra: training
Requires-Dist: pyarrow>=16; extra == 'training'
Requires-Dist: scikit-learn>=1.5; extra == 'training'
Requires-Dist: vadersentiment>=3.3.2; extra == 'training'
Provides-Extra: transformers
Requires-Dist: torch>=2.2; extra == 'transformers'
Requires-Dist: transformers>=4.40; extra == 'transformers'
Description-Content-Type: text/markdown

# TextInsight

**Lightweight, offline-first NLP for English and Hinglish text, with pluggable backends.**

[![CI](https://github.com/harsh01103/TextInsight/actions/workflows/ci.yml/badge.svg)](https://github.com/harsh01103/TextInsight/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/textinsight-nlp)](https://pypi.org/project/textinsight-nlp/)
[![Python](https://img.shields.io/pypi/pyversions/textinsight-nlp)](https://pypi.org/project/textinsight-nlp/)
![Coverage](https://img.shields.io/badge/coverage-99%25-brightgreen)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

Documentation: [docs/](docs/index.md) (build it with `mkdocs serve`). Demo app: [demo/](demo/).

## Why TextInsight?

- **Built for Hinglish** – handles code-mixed text such as *"yeh phone mast hai but battery bekaar hai"*: spelling variants, Hindi word order for negation, and a model trained on real Hinglish tweets.
- **Lightweight by default** – small base install; heavy models are optional extras.
- **No surprises** – importing the library never downloads anything.
- **Pluggable backends** – swap lexicon, classical ML, ONNX, transformer or LLM implementations with one argument.
- **Honest benchmarks** – every backend ships with measured results.

## Installation

```bash
pip install textinsight-nlp
```

The package installs as `textinsight-nlp` and is imported as `textinsight`.

## Quick start

```python
import textinsight as ti

ti.clean("Visit https://example.com now!", lowercase=True)  # 'visit now!'
ti.tokenize("Don't miss it :) #sale")  # ["Don't", 'miss', 'it', ':)', '#sale']
ti.sentences("Dr. Rao aaye. yeh accha hai!")  # ['Dr. Rao aaye.', 'yeh accha hai!']
ti.normalize_hinglish("bohot acha hai")  # 'bahut accha hai'
ti.detect_language("yeh phone mast hai but battery bekaar").language  # 'hi-en'
ti.sentiment("bahut accha product hai")  # SentimentResult(label='positive', ...)
ti.sentiment("yeh accha nahi hai", backend="lexicon")  # handles Hindi negation

doc = (
    "Phone ki battery life acchi hai. Camera quality bhi mast hai. Battery life do din chalti hai."
)
ti.keywords(doc, top_n=2).texts  # ['battery life', 'camera quality']
ti.summarize(doc, max_sentences=1).summary  # the most representative sentence

ti.aspects("camera mast hai but battery bekaar hai").labels
# {'camera': 'positive', 'battery': 'negative'}
ti.stats(doc).words  # also readability, diversity and reading time
ti.redact("Mail a.b@example.com or call 9876543210").text
# 'Mail [EMAIL] or call [PHONE]'

report = ti.analyze(reviews, tasks=["sentiment", "aspects", "keywords"])
report.to_html("report.html")  # one self-contained file

ti.list_backends()  # registered backends
ti.config.offline = True  # never download models
```

Language labels are `en`, `hi` (mainly Devanagari), `hi-en` (code-mixed or romanized Hindi)
and `und` (undetermined).

## Benchmarks

Sentiment on the held-out test split of [SentiMix](https://doi.org/10.5281/zenodo.3974927)
(3,000 real Hinglish tweets):

| Method | Accuracy | Macro F1 | Weighted F1 |
|---|---|---|---|
| Majority class | 0.367 | 0.179 | 0.197 |
| VADER | 0.492 | 0.470 | 0.475 |
| TextInsight `lexicon` | 0.561 | 0.540 | 0.546 |
| **TextInsight `classical`** (default, 529 KB, no dependencies) | **0.702** | **0.707** | **0.703** |
| **TextInsight `hybrid`** (optional transformer blend) | **0.723** | **0.728** | **0.725** |

Cross-validated macro F1 on the other 17,000 tweets is lower and more conservative: 0.639 for
`classical` and 0.666 for `hybrid`. Word-level language detection on the test tweets: 72.7%.

Short, plain sentences such as reviews use a lexicon fallback, because a model trained on tweets
reads them as neutral too often. On 60 hand-written review-style Hinglish sentences kept back for
this check, macro F1 rose from 0.581 to 0.819 for `classical` and from 0.499 to 0.834 for `hybrid`.
That set is small and not independent, so treat it as a behaviour check.

The test-split scores above are for tweets only. See the
[model card](docs/model_cards/sentiment_classical.md) for method, limitations and bias.

To reproduce them, install the `training` extra, download the dataset and run:

```bash
textinsight benchmark sentiment --data-dir path/to/Hinglish --split test
```

### Keywords and summaries

Keywords on the [Inspec](https://aclanthology.org/W03-1028/) test split (500 abstracts):

| Method | Precision@10 | Recall@10 | F1@10 |
|---|---|---|---|
| TF-IDF | 0.249 | 0.297 | 0.255 |
| RAKE | 0.253 | 0.302 | 0.259 |
| **TextRank** (default) | **0.275** | **0.318** | **0.278** |

Summaries on the CNN/DailyMail test split (11,490 articles, three sentences each):

| Method | ROUGE-1 F1 | ROUGE-2 F1 | ROUGE-L F1 |
|---|---|---|---|
| Lead-3 (first three sentences) | 0.402 | 0.175 | 0.251 |
| **TextRank** (default) | 0.378 | 0.158 | 0.241 |
| Frequency | 0.320 | 0.113 | 0.212 |

TextRank does not beat Lead-3 on news, where the main facts come first. On 20 hand-written
Hinglish paragraphs, TextRank keywords scored 0.375 F1 at 5 keywords, against 0.279 without
Hinglish handling. That set is small and not independent. See the
[method card](docs/model_cards/keywords_summarize.md).

### Aspects and PII

On 20 hand-written reviews in English and Hinglish, `aspects()` found the right aspect and
sentiment with 0.877 F1. PII rules scored 1.0 precision and recall on generated values, which
only shows that they work on the formats they were written for. Neither is an independent
benchmark. See the [method card](docs/model_cards/aspects_stats_pii.md) for known limits:
for example, `redact()` cannot find names, and treats any 10-digit number starting with 6 to 9
as a phone number.

## Most accurate option

```bash
pip install "textinsight-nlp[transformers]"
```

```python
ti.sentiment("zabardast movie thi, maza aa gaya", backend="hybrid")
```

First use downloads the 438 MB HingBERT encoder, so `hybrid` runs only when you ask for it.

## Credits

The bundled sentiment model derives from SentiMix (CC-BY-4.0): Patwa et al., "SemEval-2020
Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets", SemEval-2020. The `hybrid` backend
uses HingBERT (CC-BY-4.0): Joshi, "L3Cube-HingCorpus and HingBERT", 2022. Benchmarks use Inspec
(Hulth, 2003) and CNN/DailyMail (Hermann et al., 2015), which are not redistributed here.

## Command line

```bash
textinsight sentiment "yeh phone mast hai"
```

```bash
textinsight --help
```

Commands: `sentiment`, `aspects`, `keywords`, `summarize`, `stats`, `redact`, `language`,
`clean`, `analyze`, `info`, `models` and `benchmark`. Run `textinsight --version` or
`textinsight info` to see versions and installed backends.

```bash
textinsight analyze reviews.csv --column review --tasks sentiment,aspects,keywords --out report.html
```

## Optional backends

| Backend | Install | Use |
|---|---|---|
| `hybrid` sentiment | `textinsight-nlp[transformers]` | Most accurate on Hinglish; downloads 438 MB |
| `onnx` sentiment | `textinsight-nlp[onnx]` | Fast English-only model; downloads 68 MB |
| `abstractive` summaries | `textinsight-nlp[transformers]` | Writes new English sentences; downloads 242 MB |
| `llm` sentiment and aspects | `textinsight-nlp[llm]` | Uses Claude; sends your text to Anthropic and needs your own API access |
| pandas and scikit-learn | `textinsight-nlp[pandas]`, `textinsight-nlp[sklearn]` | `df["review"].ti.sentiment()` and a feature transformer |

None of these run unless you ask for them by name. The trade-offs are measured in
[docs/backends.md](docs/backends.md).

## Development

```bash
python -m venv .venv
.venv\Scripts\activate          # Windows  (source .venv/bin/activate on macOS/Linux)
pip install -e ".[dev]"
pre-commit install

pytest                          # tests + coverage
ruff check . && ruff format --check .
mypy
```

## Configuration

| Setting | Environment variable | Default |
|---|---|---|
| `config.cache_dir` | `TEXTINSIGHT_CACHE_DIR` | OS user cache folder + `/textinsight` |
| `config.offline` | `TEXTINSIGHT_OFFLINE` | `False` |
| `config.device` | `TEXTINSIGHT_DEVICE` | `"cpu"` |
| `config.max_input_chars` | `TEXTINSIGHT_MAX_INPUT_CHARS` | `1_000_000` |

## License

MIT
