Metadata-Version: 2.4
Name: polyglotminer
Version: 1.0.1
Summary: Empirical research framework for Git repository analysis and cross-language migration mining
Home-page: https://github.com/SariYanouche/PolyglotMiner
Author: Sari Yanouche
License: MIT
Project-URL: Source, https://github.com/SariYanouche/PolyglotMiner
Project-URL: Tracker, https://github.com/SariYanouche/PolyglotMiner/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Version Control :: Git
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: gitpython>=3.1.40
Requires-Dist: jinja2>=3.1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: rich-argparse>=1.4.0
Requires-Dist: pygments>=2.15.0
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<div align="center">

# PolyglotMiner

### Empirical Git Repository & Migration Intelligence Platform

[![PyPI](https://img.shields.io/pypi/v/polyglotminer?color=4F46E5&label=PyPI)](https://pypi.org/project/polyglotminer/)
[![Python Version](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat&logo=python&logoColor=white)](https://www.python.org/)
[![Version](https://img.shields.io/badge/Release-v1.0.1-4F46E5?style=flat)](https://github.com/SariYanouche/PolyglotMiner/releases)
[![License: MIT](https://img.shields.io/badge/License-MIT-059669.svg)](LICENSE)
[![Storage: SQLite](https://img.shields.io/badge/Storage-SQLite%203%20%28WAL%29-003B57?style=flat&logo=sqlite&logoColor=white)](https://www.sqlite.org/)
[![UI: Web Studio](https://img.shields.io/badge/Interface-Interactive%20Studio-0284C7?style=flat)](http://localhost:8080)

**PolyglotMiner** is a high-throughput research platform and CLI tool suite for analyzing longitudinal codebase evolution, multi-language composition dynamics, and cross-language software migrations across Git repositories at scale.

[Installation & Quick Start](#installation--quick-start) •
[Overview & Methodology](#overview--methodology) •
[CLI Command Reference](#cli-command-reference) •
[Web Studio & API](#web-analytics-studio) •
[Architecture](#database-architecture) •
[Citation](#citation)

</div>

---

## Installation & Quick Start

### 1. Install via pip (Recommended)

```bash
pip install polyglotminer
```

Both `polyglotminer` and its short alias `pgminer` are immediately available in your terminal.

```bash
# Verify environment health
polyglotminer doctor

# Launch the interactive terminal wizard (TUI)
polyglotminer
```

<details>
<summary><b>Or install from source / Git (for development)</b></summary>

```bash
git clone https://github.com/SariYanouche/PolyglotMiner.git
cd PolyglotMiner
python -m venv .venv
source .venv/bin/activate       # On Windows: .\.venv\Scripts\Activate.ps1
pip install -e .
```
</details>

### 2. Configure Autocompletion & Auth (Optional)

```bash
# Enable tab autocompletion for your shell (PowerShell, Bash, Zsh, Fish)
polyglotminer completion --install

# (Optional) High-volume candidate harvesting without GitHub rate-limits (60 -> 5,000 req/hr)
gh auth login   # or: export GITHUB_TOKEN="ghp_..."
```

---

## Overview & Methodology

PolyglotMiner traverses repository histories without checking out physical file trees or downloading file blobs, achieving mining throughputs exceeding **70+ repositories per minute**.

* **$O(1)$ Memory Blobless Tree Diffing**: Traverses commit trees via native `git ls-tree -r` streaming with zero blob network calls.
* **Native Subprocess Streaming**: Direct `git rev-list` and `git log --name-status` pipelines eliminate commit-object memory allocation overhead.
* **Dual Execution Modes**: Fast **Empirical Module Mode** (default, file-turnover analysis) or **Physical LOC Mode** (`--full-loc`, batched `git cat-file --batch` streaming).
* **Noise & False-Positive Elimination**: Excludes C/C++/Objective-C header pairings (`.h` vs `.c`/`.cpp`/`.m`), identical renames/moves, DevOps/CI glue scripts (`.sh`, `.bat`, Dockerfiles), and vendor directories (`node_modules/`, `Pods/`, `vendor/`).

### Empirical Migration Archetypes

| Archetype | Defining Empirical Criteria |
| :--- | :--- |
| **Direct File Replacement** | Verified 1-to-1 file conversion (e.g. `Parser.java` $\to$ `Parser.kt`) in co-changes |
| **Incremental Migration** | Macro inverse displacement with Pearson correlation $r \le -0.25$ and source contraction $\ge 25\%$ |
| **Heterogeneous Interop** | Long-term concurrent multi-language co-existence ($r \approx 0$) |
| **Polyglot Co-Growth** | Concurrent expansion of both languages across milestones ($r > 0$) |

$$\text{Milestone Pearson Correlation: } r = \frac{\sum (s_i - \bar{s})(t_i - \bar{t})}{\sqrt{\sum (s_i - \bar{s})^2 \sum (t_i - \bar{t})^2}} \le -0.25$$

---

## CLI Command Reference

Both `polyglotminer` and its short alias `pgminer` are registered executable commands.

| Command | Aliases | Description | Quick Example |
| :--- | :--- | :--- | :--- |
| **`scan`** | `s` | Point-in-time language distribution & LOC metrics | `polyglotminer scan . --open` |
| **`history`** | `h`, `evol` | Longitudinal equidistant milestone trajectories | `polyglotminer history repo -k 10 --open` |
| **`compare`** | `c`, `diff` | Pairwise 2-commit delta & churn inspector | `polyglotminer compare . HEAD~10 HEAD` |
| **`mine`** | `cochange` | Mined multi-language commits & file replacements | `polyglotminer mine repo --min-loc 10` |
| **`campaign`** | `harvest` | Multiprocess batch discovery & mining pipeline | `polyglotminer campaign -p "C->C++,JS->TS"` |
| **`ui`** | `serve` | Launch interactive Web Analytics Studio | `polyglotminer ui --port 8080` |
| **`saved`** | `bookmarks`| List and export curated benchmark corpus | `polyglotminer saved --export-csv corpus.csv` |
| **`bookmark`** | `star` | Tag or annotate a repository in SQLite | `polyglotminer bookmark repo -c "Java->Kotlin"` |
| **`rebuild`** | `b` | Instant HTML dashboard re-compilation | `polyglotminer rebuild --open` |
| **`list`** | `ls` | List stored snapshots, runs, and repositories | `polyglotminer list` |
| **`clean`** | `rm` | Delete reports, snapshots, or database records | `polyglotminer clean my-repo --yes` |
| **`doctor`** | `check` | System diagnostics & dependency health-check | `polyglotminer doctor` |
| **`completion`**| | Auto-detect shell and install tab-completion | `polyglotminer completion --install` |

### Key Command Examples

```bash
# 1. Point-in-time profiling (local directory or remote repository)
polyglotminer scan https://github.com/facebook/react --commit v18.2.0 --open

# 2. Longitudinal evolution trajectory across 10 milestone intervals
polyglotminer history handmux/handmux --samples 10 --open

# 3. Directed migration campaign (target specific transitions e.g. C -> C++ or JS -> TS)
polyglotminer campaign -p "C->C++,JS->TS" --workers 4 --limit 50 --open

# 4. Continuous overnight mining loop with stratified star sampling
polyglotminer campaign --min-stars 50 --max-stars 5000 --seed 42 --workers 2 --loop

# 5. Mine multi-language co-changes with minimum LOC delta threshold
polyglotminer mine carlphilipp/chicago-commutes --source-lang Java --target-lang Kotlin --export-csv pairs.csv

# 6. Pairwise 2-commit turnover comparison
polyglotminer compare vuejs/vue v2.6.0 v3.0.0 --open
```

<div align="center">
<img src=".github/assets/cli_history_demo.png" alt="PolyglotMiner CLI Terminal Evolution Output" width="95%">
</div>

---

## Web Analytics Studio

Launch the dedicated read-only analytics suite with `polyglotminer ui` (default: `http://localhost:8080`).

<div align="center">
  <img src=".github/assets/web_studio_trajectory.png" alt="PolyglotMiner Interactive Analytics Studio" width="75%">
</div>

* **Graph Navigator & Spotlight Visualizations**: Instant navigation across transition ecosystems, turnover archetypes, star tier distributions, and commit depth brackets.
* **Live Heuristic Tuning**: Interactive threshold sliders (candidate pairs threshold, minimum confidence score, archetype filtering) with instant client-side recalculation.
* **Instant SSR & Skeleton Shimmers**: Virtualized tables and lightweight SQLite streaming (<10MB RAM) for lag-free exploration of large datasets.
* **Academic Export Suite**: 1-click export of publication-ready high-resolution figures (300 DPI PNG), LaTeX `booktabs` tables, and CSV datasets.
* **REST API**: Integrates `/api/summary`, `/api/corpus`, `/api/repositories`, `/api/repositories/saved`, `/api/cochanges`, and dynamic `/<owner>/<repo>/evolution_latest.html` routing.

---

## Database Architecture

Analytical state is stored in an embedded SQLite database (`reports/polyglotminer.db`) with Write-Ahead Logging (`WAL`):

| Table | Primary Key | Description |
| :--- | :--- | :--- |
| **`repositories`** | `name_with_owner` | Discovered corpus metadata, stars, composition, bookmarks, and status |
| **`snapshots`** | `id` | Point-in-time scan metrics, file counts, code/comment/blank lines |
| **`evolution_runs`** | `run_id` | Longitudinal trajectories, milestone sample counts, and migration pairs |
| **`cochange_commits`** | `id` | Commits modifying $\ge 2$ languages with net directional churn deltas |
| **`migration_pairs`** | `id` | 1-to-1 file replacements, deleted/added snippets, and similarity confidence |

* **Process Recycling**: Campaign workers recycle processes every 25 repositories (`maxtasksperchild=25`) for 100% OS memory reclamation.
* **Watchdog & Failure Resilience**: 75-second per-repository watchdog timeout. Failed or timeout repositories are permanently tracked in SQLite so resume queues never stall.

---

## Verification & Tests

```bash
# Run full automated test suite
python -m unittest discover tests

# Run specific verification suites
python -m unittest tests/test_cli_smoke.py
python -m unittest tests/test_directed_campaign.py
python -m unittest tests/test_header_migration_exclusion.py
```

---

## Repository Structure

```text
PolyglotMiner/
├── polyglotminer/                  # Core research package
│   ├── __init__.py                 # Version and package metadata (v1.0.1)
│   ├── cli.py                      # Unified CLI parser and router
│   ├── scanner.py                  # Single-commit line-counting and tokenizer
│   ├── evolution.py                # Milestone sampler and trajectory engine
│   ├── cochange.py                 # Multi-language commit and snippet extractor
│   ├── campaign.py                 # Multiprocess mass-mining campaign engine
│   ├── cache.py                    # Git clone cache pool (.cache/repos/)
│   ├── languages.py                # Language specifications and color maps
│   ├── config.py                   # Vendor and asset exclusion rules
│   ├── db.py                       # SQLite persistence layer (WAL mode)
│   ├── storage.py                  # Storage orchestration
│   ├── server.py                   # Embedded REST API and Web Studio server
│   ├── doctor.py                   # System diagnostics & dependency check
│   ├── completion.py               # Shell autocompletion generator
│   ├── wizard.py                   # Interactive quick-start terminal wizard (TUI)
│   ├── models.py                   # Analytical data structures
│   ├── reporters/                  # CLI and HTML formatters
│   └── templates/                  # Web Studio Jinja2 templates
├── tests/                          # Automated verification test suite
├── main.py                         # Universal entry point
├── requirements.txt                # Package dependencies
├── setup.py                        # Setuptools build configuration
└── README.md                       # Documentation
```

---

## Citation

If you use **PolyglotMiner** in academic publications, please cite:

```bibtex
@software{polyglotminer2026,
  author = {Sari, Yanouche},
  title = {PolyglotMiner: An Empirical Research Platform for Longitudinal Software Evolution and Cross-Language Migration Mining},
  year = {2026},
  version = {1.0.1},
  url = {https://github.com/SariYanouche/PolyglotMiner}
}
```
