Metadata-Version: 2.4
Name: polyglotminer
Version: 1.1.0
Summary: Empirical research framework for Git repository analysis and cross-language migration mining
Home-page: https://github.com/SariYanouche/PolyglotMiner
Author: Sari Yanouche
License: MIT
Project-URL: Source, https://github.com/SariYanouche/PolyglotMiner
Project-URL: Tracker, https://github.com/SariYanouche/PolyglotMiner/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Version Control :: Git
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: gitpython>=3.1.40
Requires-Dist: jinja2>=3.1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: rich-argparse>=1.4.0
Requires-Dist: pygments>=2.15.0
Provides-Extra: gpu
Requires-Dist: torch>=2.0.0; extra == "gpu"
Requires-Dist: transformers>=4.30.0; extra == "gpu"
Requires-Dist: numpy>=1.24.0; extra == "gpu"
Requires-Dist: einops>=0.6.0; extra == "gpu"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<div align="center">

<img src=".github/assets/logo.png" alt="PolyglotMiner Logo" width="120">

# PolyglotMiner

### Empirical Git Mining Toolkit for Longitudinal Software Evolution & Cross-Language Migration Analysis

[![Release](https://img.shields.io/badge/release-v1.1.0-4F46E5?style=flat-square&logo=github&logoColor=white)](https://github.com/SariYanouche/PolyglotMiner/releases/tag/v1.1.0)
[![PyPI Version](https://img.shields.io/badge/pypi-v1.1.0-4F46E5?style=flat-square&logo=pypi&logoColor=white)](https://pypi.org/project/polyglotminer/)
[![Python Version](https://img.shields.io/badge/python-3.10%2B-3776AB?style=flat-square&logo=python&logoColor=white)](https://www.python.org/)
[![Hardware Acceleration](https://img.shields.io/badge/hardware-CUDA%20%7C%20Apple%20MPS%20%7C%20CPU-76B900?style=flat-square&logo=nvidia&logoColor=white)](https://developer.nvidia.com/cuda-zone)
[![Storage: SQLite](https://img.shields.io/badge/storage-SQLite%203%20(WAL)-003B57?style=flat-square&logo=sqlite&logoColor=white)](https://www.sqlite.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-059669.svg?style=flat-square)](LICENSE)
[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SariYanouche/PolyglotMiner/blob/main/notebooks/polyglotminer_colab_gpu.ipynb)

<p align="center">
  <b>PolyglotMiner</b> is an empirical software engineering research framework engineered to investigate longitudinal multi-language software evolution, quantify language turnover, and detect cross-language migrations across Git histories at scale.
</p>

[Quick Start](#-quick-start) •
[Motivation & Scope](#-motivation--research-scope) •
[Detection Architecture](#-detection-architecture-v110) •
[CLI Reference](#-cli-command-reference) •
[Python Research API](#-python-research-api) •
[Interactive Web Studio](#-interactive-web-studio) •
[Citation](#-citation)

</div>

---

<div align="center">
  <img src=".github/assets/dashboard.png" alt="PolyglotMiner Evolution CLI" width="95%">
</div>

---

## 🔬 Motivation & Research Scope

While multi-language software systems are ubiquitous, studying their evolutionary dynamics empirically presents major methodological challenges:
1. **Phased Migrations (*Strangler Fig* Pattern)**: Developers rarely rewrite and delete legacy code in atomic single-commit operations; migrations span multiple commits, weeks, or release cycles with concurrent co-existence.
2. **Structural Restructuring ($1 \to N$ and $N \to 1$)**: Rewriting across languages frequently involves architectural consolidation (e.g., C/C++ `.h + .c` pairs into unified Rust/Zig modules) or decomposition (splitting legacy monoliths into modular components).
3. **Severe False-Positive Hazards**: Multi-language repositories frequently experience polyglot co-evolution, C/C++ header extraction, build reclassifications, and third-party vendoring that distort naive migration metrics.

**PolyglotMiner v1.1.0** addresses these challenges through zero-checkout streaming tree traversal, a multi-stage deterministic-to-neural matching waterfall, sliding-window Strangler Fig tracking, and strict false-positive elimination guards.

---

## ⚡ Quick Start

### 1. Installation

```bash
# Core framework (CPU, O(1) tree traversal, Code DNA & contract invariants)
pip install polyglotminer

# With GPU acceleration (CUDA/MPS dense vector embeddings via PyTorch)
pip install "polyglotminer[gpu]"
```

Verify your environment and diagnostics:
```bash
polyglotminer doctor
```

### 2. Immediate Examples

```bash
# 1. Profile a repository's current language composition and lines of code
polyglotminer scan facebook/react --open

# 2. Track longitudinal language trajectories across 10 historical milestones
polyglotminer history jackwener/OpenCLI --samples 10 --gpu --open

# 3. Mine commit-level file conversions and refactorings
polyglotminer mine carlphilipp/chicago-commutes --gpu --open

# 4. Launch the local interactive Web Analytics Studio
polyglotminer ui --port 8080
```

---

## 🧠 Detection Architecture (v1.1.0)

PolyglotMiner combines high-throughput deterministic matching with semantic recovery in a **Hierarchical Waterfall**:

```
[Candidate Commit / Multi-Commit Window: Deleted & Added Files]
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Pre-Filter Guards: Exclude Vendored Code, 100% Identical   │
  │ Renames, C/C++ Header Splits, and Test-Prod Cross-Pairings │
  └──────────────────────────────┬──────────────────────────────┘
                                 │
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 1: Exact Basename & Header+Impl Consolidation         │ ──► O(1) Deterministic
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 2: Normalized Stem (camelCase / snake_case) & Suffixes│ ──► Morphological Norm
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 3: Subtoken Jaccard Overlap & Edit Distance           │ ──► Subtoken Similarity
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 4: Contract Invariants (API Routes, DB Tables, RPC)   │ ──► Behavioral Invariant
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 5: Structural Solvers (1:N Decomposition / N:1 Merge) │ ──► Structural Token Bags
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 6: Strangler Fig Tracker (Multi-Commit Windows)       │ ──► Phased Lifecycle Arcs
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files & --gpu)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 7: Hardware-Accelerated Dense Neural Embeddings       │ ──► 8,192-token Cosine Sim
  │ (Default: jinaai/jina-embeddings-v2-base-code)             │
  └─────────────────────────────────────────────────────────────┘
```

<div align="center">
  <img src=".github/assets/migration_replacement.png" alt="Migration Replacement Match" width="90%">
</div>

### Empirical Migration Archetypes

| Archetype | Evolutionary Behavior & Diagnostic Criteria |
| :--- | :--- |
| **Direct Replacement** | Atomic 1-to-1 file translation (`file.js` $\to$ `file.ts`) within a single commit. |
| **Strangler Fig Migration** | Phased asynchronous replacement across multiple commits ($k \ge 1$) with temporary co-existence. |
| **1-to-$N$ Decomposition** | Monolithic source module split into $N$ cohesive target components while preserving token invariants. |
| **$N$-to-1 Consolidation** | $N$ source files (e.g. `.h + .c` or legacy helpers) consolidated into 1 unified target implementation. |
| **Macro Displacement** | Longitudinal trajectory where the incoming language displaces the legacy language ($r \le -0.70$). |
| **Heterogeneous Interop** | Stable multi-language co-existence across historical milestones without replacement. |

---

## 💻 CLI Command Reference

Both `polyglotminer` and its alias `pgminer` provide a modular research CLI:

| Command | Alias | Category | Description |
| :--- | :---: | :--- | :--- |
| **`history`** | `h` | Evolution | Sample milestone checkpoints and compute longitudinal trajectories |
| **`mine`** | `m` | Mining | Extract multi-language co-change commits and file replacements |
| **`campaign`** | `c` | Mining | High-throughput multiprocess candidate harvesting across GitHub strata |
| **`scan`** | `s` | Profiling | Point-in-time language distribution, file counts, and physical LOC |
| **`diff`** | `d` | Churn | Pairwise 2-commit file turnover, line deltas, and replacements |
| **`ui`** | `u` | Studio | Launch local interactive web studio dashboard on port 8080 |
| **`bookmark`**| `b` | Datasets | Curate, tag, annotate, and export benchmark research repositories |
| **`list`** | `l` | Workspace | List stored snapshots, evolution runs, and mined datasets |
| **`rebuild`** | `r` | Workspace | Rebuild HTML dashboards and reports from SQLite datasets |
| **`clean`** | `x` | Workspace | Purge stored reports, specific runs, or analysis caches |
| **`doctor`** | `k` | Diagnostics | Environment diagnostics for Git, Python runtime, and GPU acceleration |

### Key Research Workflows

#### 1. Longitudinal Trajectory Analysis (`history`)
```bash
# Analyze a target repository across 10 historical milestones
polyglotminer history jackwener/OpenCLI --samples 10 --gpu --open

# Deep physical LOC traversal between specific release tags
polyglotminer history vuejs/vue --from v2.6.0 --to v3.0.0 --full-loc
```

#### 2. Mining Commit Refactorings (`mine`)
```bash
# Extract multi-language co-change commits with neural matching
polyglotminer mine carlphilipp/chicago-commutes --gpu --open

# Target specific migration transitions with minimum LOC churn threshold
polyglotminer mine . --source-lang Java --target-lang Kotlin --min-loc 10 --export-csv transitions.csv
```

#### 3. Large-Scale Empirical Discovery Campaigns (`campaign`)
```bash
# Harvest and mine 50 candidate repositories matching specific migration transitions
polyglotminer campaign -p "Python -> Rust, Java -> Kotlin" --limit 50 --workers 4 --gpu

# Continuous unattended empirical harvester with commit, star, and file bounds
polyglotminer campaign --min-stars 50 --min-commits 100 --max-commits 5000 --min-files 100 --workers 4 --no-logs
```

---

## 🐍 Python Research API

PolyglotMiner can be integrated directly into Python data pipelines, Jupyter notebooks, or pandas workflows:

```python
from polyglotminer.scanner import scan_repository
from polyglotminer.evolution import analyze_evolution
from polyglotminer.cochange import mine_cochange_commits

# 1. Profile language distribution
snapshot = scan_repository("facebook/react")
print(f"Total lines: {snapshot.total_lines:,}")
for lang, stats in snapshot.languages.items():
    print(f"  {lang:<12} {stats.code_lines:>8,} LOC ({stats.percentage:.1f}%)")

# 2. Mine cross-language file transitions programmatically
transitions = mine_cochange_commits(
    repo_path="jackwener/OpenCLI",
    source_lang="Python",
    target_lang="Rust"
)
for t in transitions:
    for pair in t.migration_pairs:
        print(f"[{t.commit_hash[:7]}] {pair.source_file} -> {pair.target_file} ({pair.match_type})")
```

---

## 📊 Interactive Web Studio

Launch the local Zero-Trust web analytics studio at `http://localhost:8080`:

```bash
polyglotminer ui --port 8080
```

<div align="center">
  <img src=".github/assets/web_studio_trajectory.png" alt="PolyglotMiner Interactive Analytics Studio" width="90%">
</div>

* **Longitudinal Visualizations**: Interactive dual-axis milestone trajectories, language churn breakdowns, and crossover inflection points.
* **Component Restructuring Inspection**: Drill down into $1 \to N$ decompositions, $N \to 1$ consolidations, and multi-commit Strangler Fig lifecycle arcs.
* **Academic Export Suite**: 1-click export of publication-ready LaTeX `booktabs` tables, high-resolution figures (300 DPI), and raw CSV datasets.
* **Zero-Latency Persistence**: Backed by embedded SQLite with Write-Ahead Logging (`WAL`), serving sub-5ms queries across thousands of mined repositories.

---

## ☁️ Google Colab & Hardware Acceleration

A ready-to-run Google Colab notebook is available with NVIDIA GPU acceleration and Google Drive persistence:

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/SariYanouche/PolyglotMiner/blob/main/notebooks/polyglotminer_colab_gpu.ipynb)

See [`notebooks/README.md`](notebooks/README.md) for step-by-step instructions on running campaigns with cloud acceleration.

---

## 🧪 Verification & Testing

PolyglotMiner maintains an extensive test suite verifying matching passes, Git parsing, database concurrency, and false-positive guards:

```bash
# Run full unit test suite
python -m unittest discover tests

# Run specific domain test suites
python -m unittest tests/test_cli_smoke.py
python -m unittest tests/test_strangler_fig.py
python -m unittest tests/test_v117_consolidation_decomposition_fixes.py
python -m unittest tests/test_false_positive_migrations.py
```

---

## 📚 Citation

If you use **PolyglotMiner** in academic research, empirical software engineering studies, or replication packages, please cite:

```bibtex
@software{polyglotminer2026,
  author    = {Sari, Yanouche},
  title     = {PolyglotMiner: An Empirical Research Framework for Software Evolution and Cross-Language Migration Mining},
  year      = {2026},
  version   = {1.1.0},
  url       = {https://github.com/SariYanouche/PolyglotMiner}
}
```

---

## 📄 License

This project is licensed under the [MIT License](LICENSE).
