Metadata-Version: 2.4
Name: xidootopic
Version: 0.1.0
Summary: Graph-based topic modeling using embeddings, Leiden clustering and semantic merging
Project-URL: Source, https://github.com/JoseAdrianRodriguezGonzalez/xidooTOPIC
Project-URL: Issues, https://github.com/JoseAdrianRodriguezGonzalez/xidooTOPIC/issues
Author: Angel Diaz Pacheco
License-Expression: MIT
Keywords: clustering,embeddings,graph,leiden,nlp,topic-modeling
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: faiss-cpu
Requires-Dist: keybert
Requires-Dist: leidenalg
Requires-Dist: nltk
Requires-Dist: numpy>=1.23
Requires-Dist: python-igraph
Requires-Dist: scikit-learn>=1.2
Requires-Dist: scipy>=1.9
Requires-Dist: sentence-transformers
Requires-Dist: spacy
Requires-Dist: transformers
Requires-Dist: umap-learn
Provides-Extra: dev
Requires-Dist: coverage>=7.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: llm
Requires-Dist: llama-cpp-python; extra == 'llm'
Provides-Extra: nlp
Requires-Dist: keybert; extra == 'nlp'
Requires-Dist: nltk; extra == 'nlp'
Requires-Dist: spacy; extra == 'nlp'
Description-Content-Type: text/markdown

# XidooTopic

**Graph-based topic modeling using embeddings, Leiden clustering, and semantic merging**

---

## 🚀 Overview

XidooTopic is a modular framework for unsupervised topic modeling built on top of modern NLP embeddings and graph-based clustering. It transforms raw text corpora into coherent, semantically meaningful topics through a pipeline that combines:

- Transformer-based embeddings  
- Dimensionality reduction (UMAP)  
- Outlier detection (Isolation Forest)  
- Graph construction (k-NN via FAISS)  
- Community detection (Leiden algorithm)  
- Semantic cluster merging  
- Keyword extraction (KeyBERT)  

---

## 📦 Installation

```bash
pip install xidootopic
```

Optional dependencies:

```bash
pip install xidootopic[nlp]
pip install xidootopic[llm]
```

---

## 🧠 Pipeline Summary

The method follows a structured six-stage pipeline:

1. Text Preprocessing  
   Raw reviews are cleaned, normalized, and filtered to remove noise and low-information samples.

2. Embedding Generation  
   Each document is mapped into a dense semantic vector space using transformer-based models.

3. Dimensionality Reduction  
   UMAP reduces embeddings to a low-dimensional manifold while preserving local structure.

4. Outlier Detection  
   Isolation Forest removes anomalous samples that may distort clustering structure.

5. Graph Construction & Clustering  
   A k-NN graph is built using FAISS and clustered using the Leiden algorithm with resolution optimization.

6. Postprocessing  
   Clusters are refined via:
   - Removal of small clusters  
   - Semantic merging of similar clusters  
   - Keyword extraction with KeyBERT  

---

## 📊 Core Ideas Behind the Method

### Isolation Forest

[!NOTE]
Isolation Forest assumes anomalies are easier to isolate than normal points. It builds random partition trees and assigns higher anomaly scores to points that require fewer splits to isolate.

This makes it highly effective for high-dimensional embeddings where distance-based methods degrade.

---

### Leiden Clustering

[!NOTE]
Leiden improves over Louvain by guaranteeing connected communities and better modularity optimization.

It operates in three phases:

- Local node movement  
- Partition refinement  
- Graph aggregation  

This results in more stable and coherent topic clusters.

---

### Semantic Cluster Merging

After initial clustering, clusters are merged based on:

- Cosine similarity between centroids  
- Cluster cohesion  
- Adaptive thresholds combining global and local structure  

[!NOTE]
This step reduces redundancy and improves topic interpretability by merging semantically overlapping clusters.

---

### Keyword Extraction (KeyBERT)

[!NOTE]
KeyBERT selects keywords based on embedding similarity rather than frequency.

Unlike TF-IDF, KeyBERT captures semantic relevance, allowing rare but meaningful terms to surface as keywords.

Implementation details:

- Only top 35% most representative documents per cluster are used  
- MMR is applied to ensure diversity  
- Results are lemmatized and aggregated  

---

## 🧪 Features

- Modular architecture for experimentation  
- Graph-based clustering pipeline  
- Automatic resolution tuning for Leiden  
- Semantic cluster merging  
- Robust outlier filtering  
- KeyBERT-based topic representation  

---

## 📁 Project Structure

```text
src/
├── embedding/
├── preprocessing/
├── modeling/
├── xidootopic/
```

---

## ⚙️ Configuration Example

```python
from xidootopic.xidootopic import xidootopic

model = xidootopic(
    preprocessor=TextCleaner(),
    chunker=TextChunker(),
    embedder=TextEmbedder(...),
    reducer=UMAPReducer(),
    outlier_detector=IsolationForestOutlierRemover(),
    graph_builder=KNNGraphBuilder(),
    clusterer=LeidenClusterer(),
    cluster_merger=ClusterMerger(),
    keyword_model=KeyWordExtractor(...)
)

result = model.fit_transform(texts)
```

---

## 📌 Key Contributions

- Graph-based topic modeling pipeline  
- Hybrid global/local cluster merging strategy  
- Efficient embedding-based keyword extraction  
- Fully modular design for research and experimentation  

---

## 📚 Methodological Foundations

- Isolation Forest — Liu et al. (2008, 2012)  
- Leiden Algorithm — Traag et al. (2019)  
- KeyBERT — Grootendorst (2020)  
- UMAP — McInnes et al. (2018)  

---

## 🧾 Summary of Process

- Clean text data  
- Generate embeddings  
- Reduce dimensionality  
- Remove outliers  
- Build similarity graph  
- Apply Leiden clustering  
- Optimize resolution  
- Filter small clusters  
- Merge semantically similar clusters  
- Extract keywords and representations  

---

## 📌 License

MIT License  

---

## ✨ Status

[!NOTE]
This project is currently in beta stage, designed for research and experimentation in topic modeling and semantic clustering.
```
