Metadata-Version: 2.4
Name: corpus-lens
Version: 0.1.0
Summary: Drop-in TF-IDF + rhetorical-fingerprint + embedding analysis for any research corpus, with a static-HTML dashboard.
Author: Abhishek Shivakumar
License: MIT
Project-URL: Homepage, https://github.com/godofecht/corpus-lens
Project-URL: Documentation, https://github.com/godofecht/corpus-lens/blob/main/README.md
Project-URL: Issues, https://github.com/godofecht/corpus-lens/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: embed
Requires-Dist: numpy>=1.20; extra == "embed"
Requires-Dist: scikit-learn>=1.0; extra == "embed"
Requires-Dist: umap-learn>=0.5; extra == "embed"
Requires-Dist: pacmap>=0.7; extra == "embed"
Provides-Extra: pdf
Requires-Dist: pypdf>=4.0; extra == "pdf"
Provides-Extra: all
Requires-Dist: numpy>=1.20; extra == "all"
Requires-Dist: scikit-learn>=1.0; extra == "all"
Requires-Dist: umap-learn>=0.5; extra == "all"
Requires-Dist: pacmap>=0.7; extra == "all"
Requires-Dist: pypdf>=4.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"

# corpus-lens

**Drop-in TF-IDF + rhetorical-fingerprint + embedding analysis for any research corpus, with a static-HTML dashboard.**

You have a folder of papers, blog posts, working notes, lecture transcripts, or any other writing. You want to know:

- What is each paper actually about? (keywords + bigrams)
- What is its rhetorical posture? (hedging / assertion / empirical / theoretical / critical / quantitative density)
- Which papers are similar to which? (cosine on TF-IDF, top-K)
- Where does each paper sit in concept space? (UMAP / PaCMAP / PCA)
- How does it compare to public baselines? (arxiv categories, specific researchers)
- What clusters emerge? (force-directed similarity network)

`corpus-lens` gives you all of that from a single command, with a self-contained HTML dashboard you can open in any browser, host as a static site, or embed in your existing docs.

Zero hard dependencies for the core analysis (stdlib only). Embeddings need `numpy` + `sklearn` + optionally `umap-learn` / `pacmap`. No vector DB, no Docker, no LLM API calls, no telemetry.

## Quickstart

```bash
pip install corpus-lens[all]

# Discover what's in your input directory + emit an editable manifest
corpus-lens prepare ./my-papers --output ./.lens

# Run the analysis (TF-IDF, fingerprint, similarity, keyword index)
corpus-lens analyze ./my-papers --output ./.lens

# Add 2D embeddings (UMAP / PaCMAP / PCA)
corpus-lens embed --output ./.lens

# Add external comparison baselines from arxiv
corpus-lens compare --output ./.lens \
    --against arxiv:cs.LG --against arxiv-author:Friston

# Build the dashboard and open it
corpus-lens dashboard --output ./.lens --title "My Lab" --serve
```

That's it. The dashboard at `http://127.0.0.1:8765/dashboard.html` will show:

- **Per-paper** view: top-20 TF-IDF keywords, distinctive vocab, bigrams, rhetorical fingerprint, top-5 most-similar papers
- **Map**: UMAP / PaCMAP / PCA scatter, coloured by programme / maturity / year
- **Network**: force-directed similarity graph
- **Timeline**: lane chart of papers over time, per programme
- **Heatmap**: programme &times; rhetorical-dimension grid
- **Plots**: configurable scatter, programme bars, length histogram
- **Compare**: side-by-side with arxiv baselines
- **Search by term**: full inverted index
- **Leaderboard**: sort papers by any metric

## Input formats

`corpus-lens` accepts your corpus in three forms.

### 1. Directory of files (most common)

Drop `.md`, `.html`, `.htm`, `.txt`, or `.pdf` files into a directory. `corpus-lens prepare` will discover them, extract titles / dates / metadata, and emit a manifest you can edit. PDFs require `pip install corpus-lens[pdf]`.

### 2. Frontmatter-annotated files

Add YAML-ish frontmatter to your markdown / HTML to override extracted metadata:

```markdown
---
title: My Paper Title
date: 2026-06-18
programme: Statistical Mechanics
maturity: preprint
tags: koopman, perturbation, spectral theory
---

# My Paper Title

The rest of the content...
```

### 3. JSON manifest

For non-file-based corpora (database dumps, API exports), write a manifest directly:

```json
{
  "items": [
    {
      "id": "paper-001",
      "title": "Whatever",
      "date": "2026-06-18",
      "programme": "Some Programme",
      "maturity": "preprint",
      "tags": ["tag1", "tag2"],
      "text": "The full plain-text body of the paper..."
    }
  ]
}
```

Then `corpus-lens analyze --manifest ./manifest.json --output ./.lens`.

See [`docs/INPUT_FORMATS.md`](docs/INPUT_FORMATS.md) for full schema details.

## What the rhetorical fingerprint measures

Each paper gets a six-axis fingerprint in markers per 1000 tokens:

| Axis | What it counts | Why it matters |
|---|---|---|
| `hedging` | "may", "might", "could", "perhaps", "suggests", ... | Epistemic caution, openness |
| `assertion` | "proves", "establishes", "demonstrates", "definitively", ... | Confidence, finality |
| `empirical` | "experiment", "dataset", "measure", "baseline", ... | Data-driven posture |
| `theoretical` | "theorem", "lemma", "proof", "axiom", ... | Formal-derivation posture |
| `critical` | "refutes", "fails", "mistaken", "naive", ... | Argumentative / refutational tone |
| `quantitative` | numbers with units (10ms, 95%, 3.2kHz) | Numeric specificity |

The lexicons are tunable; pass `--lexicons` to override or extend via the Python API. See [`docs/INTERPRETATION.md`](docs/INTERPRETATION.md) for guidance on reading the fingerprint.

## Python API

The CLI is a thin wrapper. The full API is small and importable:

```python
from corpuslens import extract, analyze, embed, compare, dashboard

items = extract.from_directory("./my-papers")
result = analyze.run(items)
embeddings = embed.run(result, methods=["pca", "umap", "pacmap"])
baselines = compare.run(["arxiv:cs.LG", "arxiv-author:Friston"], our_analysis=result)
dashboard.build("./.lens", title="My Lab")
```

Each function takes plain dicts/lists and returns plain dicts/lists. No frameworks, no inheritance, no DI.

## Preparation guide

To get the most out of `corpus-lens`, spend ten minutes on preparation. See [`docs/PREPARATION.md`](docs/PREPARATION.md) for the full guide. The short version:

1. **One file per paper.** Don't pre-concatenate. The unit of analysis is the paper.
2. **Use frontmatter.** Title and date in YAML at the top of every file, even if the extractor would auto-detect them.
3. **Pick a programme label.** A 1-3 word category for each paper. Loose groupings are fine; this is just colour-coding in the dashboard.
4. **Strip boilerplate.** Page headers, footers, navigation, references sections, and author bios skew the keyword analysis. The HTML extractor drops these automatically; for markdown, put boilerplate in fenced blocks (`\`\`\``) so it's excluded.
5. **Decide on maturity tags.** `draft`, `preprint`, `published`, `working` are useful axes for the dashboard's colour-by-maturity view.

## Comparison baselines

`corpus-lens compare` pulls abstracts from arxiv and applies the same fingerprint so you can position your corpus against published baselines.

```bash
corpus-lens compare --against arxiv:cs.LG          # ML
corpus-lens compare --against arxiv:math.PR        # probability
corpus-lens compare --against arxiv:q-bio.NC       # neuroscience
corpus-lens compare --against arxiv:cond-mat.stat-mech  # stat mech
corpus-lens compare --against arxiv-author:Friston
corpus-lens compare --against arxiv-author:Bengio
corpus-lens compare --against arxiv-query:cat:math.DS+AND+abs:Koopman
```

Each baseline becomes a row in the dashboard's Compare tab. Hover for the per-1k-token density of each rhetorical axis.

## Output files

After `analyze + embed + compare`, your `.lens/` directory contains:

```
.lens/
  manifest.json             # editable input description
  corpus_analysis.json      # per-paper meta + keywords + fingerprint
  corpus_keyword_index.json # inverted index: term -> [{paper, tf}]
  corpus_similarity.json    # paper -> top-K most-similar
  corpus_embedding.json     # 2D coordinates per paper, per method
  comparison_corpora.json   # external baselines
  dashboard.html            # the dashboard (loads the JSON siblings)
```

All JSON. All plain. Open in any browser, host as a static site (GitHub Pages, Netlify), or pipe into another tool.

## Hosting the dashboard

The dashboard is a single HTML file plus its JSON siblings. Drop the whole `.lens/` directory anywhere that serves static files:

```bash
# GitHub Pages
gh repo create my-corpus --public
cp -r .lens/* docs/
gh-pages -d docs

# Netlify drop: drag .lens/ onto netlify.app/drop
# Vercel: vercel deploy ./.lens
# nginx / Apache: copy .lens/ to your webroot
```

No build step. No server-side code.

## License

MIT.
