Metadata-Version: 2.4
Name: normeval
Version: 0.1.3
Summary: A scientific suite for evaluating text normalization algorithms.
Author-email: Md Abdullah Al Kafi <kafi.cse@diu.edu.bd>
Project-URL: Homepage, https://github.com/abkafi1234/normeval
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scipy
Requires-Dist: scikit-learn
Requires-Dist: python-Levenshtein
Requires-Dist: matplotlib
Dynamic: license-file

# normeval: Scientific Evaluation Suite for Text Normalization

`normeval` is a robust, mathematically grounded Python framework for evaluating text normalization algorithms. It goes beyond simple string matching by combining semantic preservation, vocabulary compression, character-level fidelity, and statistically rigorous downstream impact analysis.

---

## 🔬 Motivation: Why do we need this?

Evaluating text normalization (such as stemming, lemmatization, spell-correction, or noise reduction) is traditionally a fragmented process. Researchers often rely solely on downstream accuracy, which is highly susceptible to dataset bias and random initialization.

A **"good"** normalization algorithm must balance two competing objectives:

1.  **Compression**: It must collapse noisy variations of words into a single representation to reduce the feature space.
2.  **Preservation**: It must not destroy the underlying semantic meaning or map unrelated words together.

`normeval` provides a holistic, unified suite to measure this balance, ensuring your normalization techniques are rigorously tested before being deployed in NLP pipelines.

---

## 📦 Installation

Install directly from PyPI:

```bash
pip install normeval
```
### 🚀 Quick StartPythonfrom normeval import NormalizationEvaluator

```python
from sklearn.linear_model import LogisticRegression
from sentence_transformers import SentenceTransformer

# 1. Your parallel datasets
texts_original = [
    "The cats are running!!",
    "A running cat..."
]

texts_normalized = [
    "the cat run",
    "a run cat"
]

labels = [1, 0]

# 2. Define your models
classifiers = [LogisticRegression()]
embedder = SentenceTransformer('all-MiniLM-L6-v2')

# 3. Initialize Evaluator
evaluator = NormalizationEvaluator(
    texts_original,
    texts_normalized,
    labels=labels,
    classifiers=classifiers,
    embedding_model=embedder
)

# 4. Run the full suite
results = evaluator.evaluate_all(lang="Global")

print(results)
```
# 📊 Metrics (Function Deep-Dive)

`normeval` evaluates text normalization across **macroscopic**, **semantic**, and **downstream** dimensions.

---

## 1. Compression Ratio (CR)

- **Function:** `calculate_cr()`
- **What it is:** Measures the macroscopic reduction in vocabulary size.
- **The Math:**  
  `CR = |V_original| / |V_normalized|`
- **Interpretation:**
  - CR = 1.0 → No compression occurred (bijective mapping).
  - CR > 1.0 → Multiple noisy variants successfully collapsed into fewer normalized forms.

---

## 2. Information Retention Score (IRS)

- **Function:** `calculate_irs(batch_size=32)`
- **What it is:** Measures how much semantic meaning survived the normalization process.
- **Mechanism:** Converts both original and normalized texts into dense vector embeddings and computes paired cosine similarity.
- **Interpretation:**
  - Range: [-1, 1]
  - A score near 1.0 indicates semantic meaning was preserved successfully.

---

## 3. Algorithm Effectiveness Score (AES)

- **Function:** `calculate_aes(cr, irs)`
- **What it is:** A harmonic mean balancing Compression (CR) and Preservation (IRS).
- **The Math:**
  `VRG = 1-1/CR`
  `AES = (2 × IRS × VRG) / (IRS + VRG)`
- **Interpretation:** Punishes algorithms that are:
  - Too aggressive (High CR, low IRS)
  - Too passive (High IRS, low CR)

---

## 4. Average Normalized Levenshtein Distance (ANLD)

- **Function:** `calculate_anld()`
- **What it is:** A micro-level safety metric measuring character-level fidelity.
- **The Math:**  
  `ANLD = (1 / |V|) Σ [ LD(w, σ(w)) / |w| ]`
- Where:
  - `LD` = Levenshtein Distance  
  - `σ(w)` = normalized form of word `w`

---

## 5. Model Performance Delta (MPD) & Statistical Significance

- **Function:** `calculate_mpd(n_splits=5, average_method='weighted')`
- **What it is:** Measures the impact on classification performance using Stratified N-Fold CV.
- **Statistical Rigor:** Applies the **Wilcoxon Signed-Rank Test** across CV folds (p < 0.05).
- **Leakage Prevention:** `TfidfVectorizer` is fit inside each CV fold to strictly prevent train/test leakage.

## 🏗️ Project Goals
Reproducible evaluation pipelines.

Statistically grounded benchmarking.

Semantic preservation analysis.

Language-agnostic evaluation.

## 🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, open an issue first to discuss proposed improvements.

## 📄 License
This project is licensed under the MIT License.

⭐ Citation
If you use normeval in your research, please cite: underway 
