Metadata-Version: 2.5
Name: splitscope
Version: 0.2.0
Summary: Cluster a train set and measure how well it covers your eval set (CPU hashing or LLM embeddings; KMeans, MiniBatchKMeans, FAISS).
License: MIT
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: numpy
Requires-Dist: scikit-learn
Requires-Dist: scipy
Requires-Dist: tqdm
Provides-Extra: all
Requires-Dist: faiss-cpu; extra == 'all'
Requires-Dist: torch; extra == 'all'
Requires-Dist: transformers; extra == 'all'
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Provides-Extra: faiss
Requires-Dist: faiss-cpu; extra == 'faiss'
Provides-Extra: faiss-gpu
Requires-Dist: faiss-gpu; extra == 'faiss-gpu'
Provides-Extra: llm
Requires-Dist: torch; extra == 'llm'
Requires-Dist: transformers; extra == 'llm'
Description-Content-Type: text/markdown

# splitscope

**Does your training set actually cover your eval set?**

`splitscope` clusters a train dataset, projects an eval dataset onto those clusters, and tells you how well the eval examples fit the train distribution. Use it to spot distribution gaps, decide how aggressively you can filter or subsample training data, and understand which parts of your data matter for evaluation.

- **Fast CPU features:** hashed word + char n-grams, TF-IDF, optional LSA. No GPU needed.
- **LLM embeddings:** pooled hidden states from any Hugging Face model (`mean` or `last` pooling).
- **Multiple clustering backends:** KMeans, MiniBatchKMeans, FAISS (CPU or GPU).
- **Library and CLI.**

## Installation

```bash
pip install splitscope                # CPU features + sklearn KMeans / MiniBatchKMeans
pip install "splitscope[faiss]"       # + FAISS (CPU)
pip install "splitscope[faiss-gpu]"   # + FAISS (CUDA)
pip install "splitscope[llm]"         # + LLM embeddings (torch, transformers)
pip install "splitscope[all]"         # everything (CPU FAISS)
```

Optional dependencies are never installed automatically at import time. If a backend is missing, you get a clear error with the install command.

## Quick start

### Python

```python
import splitscope as ss

train = ss.load_texts("train.jsonl")
ev = ss.load_texts("test.jsonl")

result = ss.analyze(
    train, ev,
    features="hashing",       # "hashing" | "llm"
    clusterer="minibatch",    # "kmeans" | "minibatch" | "faiss"
    n_clusters=150,
)
print(result.summary())
```

### Command line

```bash
# CPU only
splitscope --train train.jsonl --eval test.jsonl --clusterer minibatch

# LLM embeddings + FAISS
splitscope --train train.jsonl --eval test.jsonl \
           --features llm --model Qwen/Qwen3-4B \
           --llm-max-length auto --clusterer faiss
```

## Input format

`.jsonl` (one JSON object per line) or `.json` (a list of objects). By default, each record's text is built from these fields:

| Field | Role |
|---|---|
| `instruct` | prompt part 1 |
| `input` | prompt part 2 |
| `output` | completion |

Text = `instruct + input + output`. If neither prompt field exists, all values in the record are concatenated. Change the field names with `TextConfig` or the `--instruction-col`, `--input-col` and `--output-col` flags:

```python
cfg = ss.TextConfig(instruction_col="question", input_col=None, output_col="answer")
train = ss.load_texts("train.jsonl", cfg)
```

## How it works

1. **Featurize** train texts (hashing or LLM embeddings).
2. **Cluster** the train set only.
3. **Project** eval texts into the same feature space and assign each to its nearest train centroid.
4. **Compare** each eval point's distance to its centroid against that cluster's train distance distribution:
   - **1σ band:** within mean ± std of train distances.
   - **Robust band:** within a MAD-based robust z-score of 3.5.
5. **Recommend** a data-reduction strategy based on how much of eval falls inside the train core.

## Output

`analyze()` returns an `AnalysisResult`:

| Attribute | Meaning |
|---|---|
| `train_labels` | cluster id for each train example |
| `eval_labels` | nearest train cluster for each eval example |
| `eval_min_dists` | distance from each eval example to its centroid |
| `stats` | per-cluster `ClusterStats` (size, mean/std/median/MAD, representative index) |
| `frac_one_sigma` | fraction of eval inside the train 1σ band |
| `frac_robust` | fraction of eval inside the train robust band |
| `summary()` | human-readable report |
| `recommendation()` | suggested filtering strategy |

## Advanced usage

Fit once, analyze several eval sets:

```python
from splitscope import ClusterAnalyzer

analyzer = ClusterAnalyzer(
    features="llm",
    clusterer="faiss",
    n_clusters=200,
    features_kwargs=dict(model_name="Qwen/Qwen3-4B", pooling="last",
                         max_length="auto", batch_size=8),
)
analyzer.fit(train)
res_a = analyzer.analyze(eval_a)
res_b = analyzer.analyze(eval_b)
```

Inspect a cluster:

```python
cid = res_a.eval_cluster_counts.most_common(1)[0][0]
s = res_a.stats[cid]
print(s.n, s.mean, s.std)
print(train[s.representative_idx])
```

## Choosing a backend

| Goal | Suggestion |
|---|---|
| Small or medium data, best quality | `clusterer="kmeans"` |
| Large data on CPU | `clusterer="minibatch"` + `features="hashing"` |
| Very large data or many clusters | `clusterer="faiss"` (uses GPU automatically if available) |
| Semantic similarity rather than lexical | `features="llm"` |

Notes:

- `hashing` features default to LSA (96 dims). Without it, centroids are dense over a very wide space, which is slow and memory-hungry.
- With `features="llm"`, `max_length="auto"` uses the P95 token length of the **train** texts. Longer examples are truncated.
- LLM features are pooled hidden states from a base model, not retrieval-trained embeddings. A dedicated embedding model usually gives sharper clusters.

## CLI reference

| Flag | Default | Description |
|---|---|---|
| `--train`, `--eval` | required | dataset paths (`.json` / `.jsonl`) |
| `--n-clusters` | 150 | number of clusters |
| `--clusterer` | `kmeans` | `kmeans`, `minibatch`, `faiss` |
| `--features` | `hashing` | `hashing`, `llm` |
| `--no-lsa` | off | disable LSA for hashing features |
| `--model` | none | Hugging Face model (required for `llm`) |
| `--pooling` | `last` | `mean` or `last` |
| `--llm-batch-size` | 8 | embedding batch size |
| `--llm-max-length` | 2048 | token limit, or `auto` for P95 |
| `--max-train`, `--max-eval` | none | cap number of samples |
| `--instruction-col`, `--input-col`, `--output-col` | `instruct`, `input`, `output` | field names |

## Development

```bash
git clone https://github.com/antony-ro/splitscope
cd splitscope
pip install -e ".[dev]"
pytest
```

## License

MIT