Metadata-Version: 2.5
Name: splitscope
Version: 0.2.3
Summary: Cluster a train set and measure how well it covers your eval set (CPU hashing or LLM embeddings; KMeans, MiniBatchKMeans, FAISS).
License: MIT
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: numpy
Requires-Dist: scikit-learn
Requires-Dist: scipy
Requires-Dist: tqdm
Provides-Extra: all
Requires-Dist: faiss-cpu; extra == 'all'
Requires-Dist: torch; extra == 'all'
Requires-Dist: transformers; extra == 'all'
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Provides-Extra: faiss
Requires-Dist: faiss-cpu; extra == 'faiss'
Provides-Extra: faiss-gpu
Requires-Dist: faiss-gpu; extra == 'faiss-gpu'
Provides-Extra: llm
Requires-Dist: torch; extra == 'llm'
Requires-Dist: transformers; extra == 'llm'
Description-Content-Type: text/markdown

# splitscope

**Does your training set actually cover your eval set?**

`splitscope` clusters a train dataset, projects an eval dataset onto those clusters, and tells you how well the eval examples fit the train distribution. Use it to spot distribution gaps, decide how aggressively you can filter or subsample training data, and understand which parts of your data matter for evaluation.

- **Fast CPU features:** hashed word + char n-grams, TF-IDF, optional LSA. No GPU needed.
- **LLM embeddings:** pooled hidden states from any Hugging Face model (`mean` or `last` pooling).
- **Multiple clustering backends:** KMeans, MiniBatchKMeans, FAISS (CPU or GPU).
- **Cluster-balanced train/eval splitting** of a single dataset, with outlier handling and an optional cache.
- **Library and CLI.**

## Installation

```bash
pip install splitscope                # CPU features + sklearn KMeans / MiniBatchKMeans
pip install "splitscope[faiss]"       # + FAISS (CPU)
pip install "splitscope[faiss-gpu]"   # + FAISS (CUDA)
pip install "splitscope[llm]"         # + LLM embeddings (torch, transformers)
pip install "splitscope[all]"         # everything (CPU FAISS)
```

Optional dependencies are never installed automatically at import time. If a backend is missing, you get a clear error with the install command.

## Quick start

### Python

```python
import splitscope as ss

train = ss.load_texts("train.jsonl")
ev = ss.load_texts("test.jsonl")

result = ss.analyze(
    train, ev,
    features="hashing",       # "hashing" | "llm"
    clusterer="minibatch",    # "kmeans" | "minibatch" | "faiss"
    n_clusters=150,
)
print(result.summary())
```

### Command line

```bash
# analyze: how well does train cover eval? (CPU only)
splitscope analyze --train train.jsonl --eval test.jsonl --clusterer minibatch

# analyze with LLM embeddings + FAISS
splitscope analyze --train train.jsonl --eval test.jsonl \
                   --features llm --model Qwen/Qwen3-4B \
                   --llm-max-length auto --clusterer faiss

# split: one dataset -> train.jsonl + eval.jsonl
splitscope split --data data.jsonl --eval-ratio 0.1 --out-dir splits/
```

## Input format

`.jsonl` (one JSON object per line) or `.json` (a list of objects). By default, each record's text is built from these fields:

| Field | Role |
|---|---|
| `instruct` | prompt part 1 |
| `input` | prompt part 2 |
| `output` | completion |

Text = `instruct + input + output`. If neither prompt field exists, all values in the record are concatenated. Change the field names with `TextConfig` or the `--instruction-col`, `--input-col` and `--output-col` flags:

```python
cfg = ss.TextConfig(instruction_col="question", input_col=None, output_col="answer")
train = ss.load_texts("train.jsonl", cfg)
```

## How it works

1. **Featurize** train texts (hashing or LLM embeddings).
2. **Cluster** the train set only.
3. **Project** eval texts into the same feature space and assign each to its nearest train centroid.
4. **Compare** each eval point's distance to its centroid against that cluster's train distance distribution:
   - **1σ band:** within mean ± std of train distances.
   - **Robust band:** within a MAD-based robust z-score of 3.5.
5. **Recommend** a data-reduction strategy based on how much of eval falls inside the train core.

## Output

`analyze()` returns an `AnalysisResult`:

| Attribute | Meaning |
|---|---|
| `train_labels` | cluster id for each train example |
| `eval_labels` | nearest train cluster for each eval example |
| `eval_min_dists` | distance from each eval example to its centroid |
| `stats` | per-cluster `ClusterStats` (size, mean/std/median/MAD, representative index) |
| `frac_one_sigma` | fraction of eval inside the train 1σ band |
| `frac_robust` | fraction of eval inside the train robust band |
| `summary()` | human-readable report |
| `recommendation()` | suggested filtering strategy |

## Splitting a dataset

Have one dataset and no eval set yet? `split_dataset` clusters it and carves out an eval set whose cluster distribution matches the train set.

```python
import splitscope as ss

cfg = ss.SplitConfig(eval_ratio=0.1, outliers="drop", clusterer="minibatch")

# SplitResult only
result = ss.split_dataset("data.jsonl", cfg)

# SplitResult + the train rows, the eval rows, or both (always in this order)
result, train = ss.split_dataset("data.jsonl", cfg, return_train=True)
result, eval_ = ss.split_dataset("data.jsonl", cfg, return_eval=True)
result, train, eval_ = ss.split_dataset("data.jsonl", cfg, return_train=True, return_eval=True)

print(result.summary())
result.save("splits/", which="both")   # "train" | "eval" | "both"
```

How it works:

- **Duplicates stay together.** Rows are grouped by a content hash first, so an exact duplicate can never land in both train and eval.
- **Balanced coverage.** Each cluster receives eval slots in proportion to its size. Every cluster keeps at least one row in train.
- **Which rows become eval:** `strategy="quantile"` (default) picks rows evenly spaced by distance to the centroid, so eval covers the core and the tails of each cluster. `strategy="random"` is stratified random sampling.
- **Outliers** (far from their centroid by a robust z-score, or in a cluster smaller than `min_cluster_size`) follow the `outliers` policy: `drop`, `train`, `eval` or `keep` (no outlier detection).
- **Reference eval (optional).** Pass example eval sets as `reference_eval=[...]` to shift eval slots toward the clusters those examples fall into. `reference_weight` (0 to 1) controls how strongly.

Rows dropped as outliers are available as `result.dropped`.

### Caching

Results are cached under `~/.cache/splitscope` (override with `cache_dir=` or the `SPLITSCOPE_CACHE` environment variable). The cache key combines an order-independent hash of the dataset rows with all split parameters, so the same data and settings return the stored assignment without re-clustering. Only row hashes and labels are stored, not your data.

```python
ss.split_dataset("data.jsonl", cfg, cache=False)   # skip the cache entirely
```

## Advanced usage

Fit once, analyze several eval sets:

```python
from splitscope import ClusterAnalyzer

analyzer = ClusterAnalyzer(
    features="llm",
    clusterer="faiss",
    n_clusters=200,
    features_kwargs=dict(model_name="Qwen/Qwen3-4B", pooling="last",
                         max_length="auto", batch_size=8),
)
analyzer.fit(train)
res_a = analyzer.analyze(eval_a)
res_b = analyzer.analyze(eval_b)
```

Inspect a cluster:

```python
cid = res_a.eval_cluster_counts.most_common(1)[0][0]
s = res_a.stats[cid]
print(s.n, s.mean, s.std)
print(train[s.representative_idx])
```

## Choosing a backend

| Goal | Suggestion |
|---|---|
| Small or medium data, best quality | `clusterer="kmeans"` |
| Large data on CPU | `clusterer="minibatch"` + `features="hashing"` |
| Very large data or many clusters | `clusterer="faiss"` (uses GPU automatically if available) |
| Semantic similarity rather than lexical | `features="llm"` |

Notes:

- `hashing` features default to LSA (96 dims). Without it, centroids are dense over a very wide space, which is slow and memory-hungry.
- With `features="llm"`, `max_length="auto"` uses the P95 token length of the **train** texts. Longer examples are truncated.
- LLM features are pooled hidden states from a base model, not retrieval-trained embeddings. A dedicated embedding model usually gives sharper clusters.

## CLI reference

### `splitscope split`

| Flag | Default | Description |
|---|---|---|
| `--data` | required | dataset path (`.json` / `.jsonl`) |
| `--out-dir` | `splits` | where to write the results |
| `--which` | `both` | write `train`, `eval` or `both` |
| `--format` | `jsonl` | `jsonl` or `json` |
| `--eval-ratio` | 0.1 | fraction of rows for eval |
| `--n-clusters` | `auto` | integer or `auto` |
| `--outliers` | `drop` | `drop`, `train`, `eval`, `keep` |
| `--outlier-z` | 3.5 | robust z-score threshold |
| `--min-cluster-size` | 1 | smaller clusters count as outliers |
| `--strategy` | `quantile` | `quantile` or `random` |
| `--reference-eval` | none | example eval file(s) |
| `--reference-weight` | 0.5 | 0 ignores the reference, 1 follows it |
| `--seed` | 42 | random seed |
| `--max-samples` | none | cap number of rows |
| `--save-dropped` | off | also write dropped outliers |
| `--no-cache`, `--cache-dir` | cache on | disable or relocate the cache |

It also accepts the feature and clusterer flags below (`--features`, `--clusterer`, `--model`, ...).

### `splitscope analyze`

| Flag | Default | Description |
|---|---|---|
| `--train`, `--eval` | required | dataset paths (`.json` / `.jsonl`) |
| `--n-clusters` | 150 | number of clusters |
| `--clusterer` | `minibatch` | `kmeans`, `minibatch`, `faiss` |
| `--features` | `hashing` | `hashing`, `llm` |
| `--no-lsa` | off | disable LSA for hashing features |
| `--model` | none | Hugging Face model (required for `llm`) |
| `--pooling` | `last` | `mean` or `last` |
| `--llm-batch-size` | 8 | embedding batch size |
| `--llm-max-length` | 2048 | token limit, or `auto` for P95 |
| `--max-train`, `--max-eval` | none | cap number of samples |
| `--instruction-col`, `--input-col`, `--output-col` | `instruct`, `input`, `output` | field names |

## License

MIT