Metadata-Version: 2.4
Name: andamento
Version: 0.3.0
Summary: Model-relative training data diagnostics for fine-tuned ML models
Author-email: Boris Blosse <borisblosse@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Autarkis/andamento
Project-URL: Documentation, https://github.com/Autarkis/andamento/tree/main/docs
Project-URL: Repository, https://github.com/Autarkis/andamento
Project-URL: Issues, https://github.com/Autarkis/andamento/issues
Keywords: machine-learning,fine-tuning,training-data,topology,diagnostics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: scikit-learn>=1.3
Requires-Dist: sentence-transformers>=2.2
Requires-Dist: jinja2>=3.1
Requires-Dist: tqdm>=4.65
Requires-Dist: datasets>=2.14
Requires-Dist: rich>=13.0
Requires-Dist: pydantic>=2
Requires-Dist: tomli>=1.1.0; python_version < "3.11"
Provides-Extra: models
Requires-Dist: torch>=2.0; extra == "models"
Requires-Dist: transformers>=4.35; extra == "models"
Requires-Dist: peft>=0.6; extra == "models"
Provides-Extra: quantize
Requires-Dist: bitsandbytes>=0.44.0; sys_platform == "linux" and extra == "quantize"
Provides-Extra: server
Requires-Dist: fastapi>=0.100; extra == "server"
Requires-Dist: uvicorn>=0.20; extra == "server"
Requires-Dist: andamento[models]; extra == "server"
Provides-Extra: remote
Requires-Dist: requests>=2.28; extra == "remote"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-xdist>=3.0; extra == "dev"
Requires-Dist: ruff>=0.1; extra == "dev"
Requires-Dist: pyright>=1.1; extra == "dev"
Provides-Extra: all
Requires-Dist: andamento[dev,models,quantize,remote,server]; extra == "all"
Dynamic: license-file

<p align="center">
  <h1 align="center">Andamento</h1>
  <p align="center">Model-relative training data diagnostics for fine-tuned ML models</p>
</p>

<p align="center">
  <a href="https://pypi.org/project/andamento/"><img src="https://img.shields.io/pypi/v/andamento?color=blue" alt="PyPI"></a>
  <a href="https://github.com/Autarkis/andamento/actions"><img src="https://img.shields.io/github/actions/workflow/status/Autarkis/andamento/ci.yml?branch=main" alt="CI"></a>
  <a href="https://github.com/Autarkis/andamento/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-green" alt="License"></a>
  <a href="https://pypi.org/project/andamento/"><img src="https://img.shields.io/pypi/pyversions/andamento" alt="Python"></a>
</p>

> Training data effectiveness is relational, not intrinsic.
> The same dataset produces different diagnostics depending on
> which base model you measure against.

![Andamento dashboard — Airoboros 3.2 audit showing 71.9% helped, 18.8% hurt, repair queue ranked by impact](docs/assets/dashboard_overview.png)

---

## The Problem

You fine-tuned a model. Benchmarks improved overall. But specific capabilities
degraded and you don't know why.

Count-based analysis says "add more data." Andamento surfaces **which regions
look wasted, harmful, or high-impact under a specific base model** so you know
where to inspect first.

## Quickstart

```bash
pip install 'andamento[models]'

# Zero-config demo: downloads a small model (~1 GB), runs curvature on 400 examples
andamento quickstart
```

An interactive HTML report opens showing which task categories helped, which hurt,
and why. No data, no models, no configuration needed.

```bash
# Then try on your own data:
andamento audit your_data.jsonl                    # Structural audit (no GPU)
andamento curvature your_data.jsonl \
    --base mistralai/Mistral-7B-v0.1 \
    --ft ./my-fine-tuned-model                     # Full curvature analysis
```

**Recommended workflow:** `quickstart` → `validate` → `plan` → `audit` → `curvature`

## Choose Your Analysis Depth

Andamento has four named analysis paths. Pick the one that matches what you need:

### Path 1: Structural Audit

**What you need:** CPU only. No models, no GPU.
**What you get:** Coverage gaps, near-duplicate clusters, topic coherence, size imbalance, topology.

```bash
andamento validate your_data.jsonl        # Check format
andamento audit your_data.jsonl           # Structural findings + HTML report
```

**When to use:** First pass on any dataset. Answers "is this data structurally sound?" before spending GPU time.

### Path 2: Curvature Diagnosis

**What you need:** GPU + base model + fine-tuned model (or LoRA adapter).
**What you get:** Everything in Path 1, plus: per-cell curvature (helped/hurt/wasted), bootstrap CIs, repair queue ranked by impact, waste classification, content vs format decomposition.

```bash
andamento curvature your_data.jsonl --base org/model --ft org/fine-tuned
```

**When to use:** You want to know which regions of your training data are helping, which are hurting, and what to fix first.

### Path 3: Curvature + Benchmark Coverage

**What you need:** GPU + models + a benchmark dataset (evaluation JSONL).
**What you get:** Everything in Path 2, plus: benchmark alignment per cell, benchmark-dark detection (training regions with no external validation), delta agreement between curvature and benchmark performance.

```bash
andamento curvature your_data.jsonl --base org/model --ft org/fine-tuned \
    --benchmark eval_set.jsonl
```

**When to use:** You have benchmark results and want to cross-check whether curvature agrees with downstream performance.

### Path 4: Remote Analysis

**What you need:** A GPU machine (cloud VM, Colab, lab server) + your laptop.
**What you get:** Everything in Paths 2-3, but heavy computation runs on the GPU machine.

```bash
# On the GPU machine:
pip install 'andamento[server]' && andamento serve --port 8080

# From your laptop:
andamento curvature data.jsonl --base X --ft Y \
    --provider http --provider-url http://gpu:8080
```

**When to use:** Your data is small enough to send over the network, but your laptop has no GPU.

---

## Why Not Just Use Cleanlab / FiftyOne / Manual Inspection?

Those tools measure data *quality* -- duplicates, label errors, outliers. Andamento measures data *effectiveness* -- whether a clean, well-labeled example actually produces a measurable loss change relative to a specific base model.

## What It Measures

Two failure modes invisible to standard evaluation:

1. **Over-tessellation** -- cells where the curvature CI includes zero (no measurable training signal). Fine-tuning produced no measurable change, usually because the base model already handled this region.
2. **Negative curvature** -- cells where fine-tuning increased loss (CI entirely negative). The training data made the model worse here.

Plus: near-duplicate detection with per-cell adaptive thresholds (a separate diagnostic layer that catches corpus-internal redundancy curvature can't see), and a pre-flight calibration that adapts decision thresholds to your specific model and data.

## ⚠ The most dangerous failure mode: false all-clear

There's a regime where curvature will tell you everything is fine when it isn't. **If your adapter is rank-constrained (LoRA, QLoRA) and the training data is semantically homogeneous** (a cultural-voice corpus, a function-calling corpus, anything where every example serves the same functional role), the adapter can run out of capacity *while every cell shows positive curvature*. Curvature reports loss-per-region; it doesn't see gradient-subspace collisions inside the adapter. We hit this exactly once in the case studies (System B — every cell helped while the adapter regressed because its 64 rank dimensions were collapsed onto 17 distinct learning directions).

Two partial guardrails ship in the tool:

- **`global_uniformity_suspect`** flags reports where one curvature class dominates >80% of cells with abnormally low between-cell variation. **Treat this flag as a stop sign**, not a feature: it means you cannot trust an all-clear from curvature alone. Verify with `--gradient`.
- **`--gradient`** opts in to per-cell gradient direction analysis (~2× cost). Surfaces effective adapter rank, pairwise cosines, and collision pairs.

If `global_uniformity_suspect` fires on your run, do not take "every cell is positive" as confirmation. The only definitive answer is gradient-subspace inspection.

## Key Observation: Effectiveness Depends on the Base Model

The same dataset produces different diagnostics depending on which base model you measure against:

| | Raw base (Mistral-7B-v0.1) | Instruct base (Mistral-7B-Instruct-v0.2) |
|---|---|---|
| Helped | 100% | 71.9% |
| Wasted | 0% | 9.4% |
| Hurt | 0% | 18.8% |

Against a raw pretrained model, every example produces positive curvature.
Against the instruct model, 28.2% shows zero or negative curvature -- regions where the base model's existing distribution already covers these examples. (Note: this is a cross-model comparison. The fine-tune was trained from the raw base, not the instruct model.)

A dataset isn't effective or wasteful in isolation. It depends on what the base model already does.

## How It Works

Andamento partitions your training data into cells, then measures how
fine-tuning changed the model's loss in each region.

```
curvature = loss_base - loss_ft
```

- **Positive** = training helped (loss went down)
- **Negative** = training hurt (loss went up)
- **CI includes zero** = training was wasted (indistinguishable from noise)

Classification uses bootstrap 95% CIs per cell, not a global threshold. Each cell has its own effective decision boundary based on its sample size and loss variance. An optional pre-flight calibrates a fallback threshold from the base model's loss distribution when CIs can't be computed.

The **repair queue** ranks cells by `|curvature| x example_count` -- highest
impact fixes first.

## The Repair Loop

```bash
andamento curvature v1.jsonl --base X --ft Y --output v1/   # Diagnose
# Fix training data based on repair queue
andamento curvature v2.jsonl --base X --ft Y --output v2/   # Re-diagnose
andamento compare --before v1/report.json --after v2/report.json    # Track progress
```

## What It Found (Airoboros Case Study)

We ran Andamento on [Airoboros 3.2](https://huggingface.co/datasets/jondurbin/airoboros-3.2) (58K examples, Mistral 7B). Against an instruction-tuned base model, the diagnostic found:

- **6 categories where the fine-tune increased loss** relative to the reference (creative writing + coding + chain-of-thought — the instruct model already handles these)
- **3 categories with no distinguishable loss shift** (curvature CI includes zero)
- **92% of `multiturn` examples are near-duplicates** (2,740 examples, 212 unique)
- **22 of 32 categories are novel_effective** — genuine training value
- **2 incoherent topological components** — category boundaries that don't map to real structural differences

Downstream intervention check: 6 pruning strategies × LoRA r=32 on raw Mistral-7B-v0.1, 1 epoch, lr 2×10⁻⁴, 10 tasks, 3 random seeds (42/123/456), bootstrap 95% CIs paired at the per-item level. This is not a claim that benchmarks are ground truth for curvature; it tests whether the diagnostic signals can guide useful interventions. Headline results vs. random pruning of the same size:

- **Pruned + dedup → +10.41pp on GSM8K** (CI [+9.00, +11.89]). Curvature flags loss-space waste; near-duplicate detection finds corpus-internal redundancy. Layering them is what produces this; neither alone does. Recovers about half of the −30.5pp regression that full fine-tuning costs.
- **Topology-informed pruning (10% removed) → +1.95pp on TruthfulQA** (CI [+1.18, +2.74]) and +1.92pp on Winogrande (CI [+0.45, +3.39]). H₀ persistence on the cell adjacency graph identifies which cell boundaries are structural; only structurally distinct negatives get cut.
- **Category pruning alone**: not significant on TruthfulQA. **Quadrant-informed pruning**: significantly hurts (random wins).
- **The honest tradeoff**: 5 of 6 pruning variants significantly hurt GSM8K. Pruning is targeted, not uniformly better.

See [Methodology](docs/methodology.md), [Interpreting Results](docs/interpreting_results.md), and the [reproducibility docs](reproducibility/README.md) for the full context, caveats, and reproduction steps.

## Evidence and Scope

Andamento is an exploratory diagnostic workbench, not a certified data-quality oracle. The directly measured quantity is the paired loss delta between two fixed models. The interpretation layer — cells, topology, repair queues, and pruning choices — is evidence-informed but not universal truth.

The strongest public anchor is the reproducible Airoboros/Mistral audit. The broader methodology is informed by additional case studies, including a public Pragma function-calling corpus and two proprietary systems that are documented as regime checks and failure cases. These case studies show where the diagnostic helped, where it diverged from benchmarks, and where curvature alone failed. They do not prove generalization across every model family, language, domain, or scale.

If Andamento gives a surprising result on your corpus, treat it as a prioritized inspection signal. Reports, failures, and replications are welcome; the goal of the open-source release is to grow shared evidence around when model-relative curvature is useful and when it is not.

## Commands

**Setup & validation:**

| Command | Path | Description |
|---------|------|-------------|
| `andamento validate` | All | Check dataset format and schema before running anything |
| `andamento plan` | All | Dry-run preflight: shows what will happen, catches bad paths |
| `andamento doctor` | All | Check environment, GPU, packages, providers |
| `andamento quickstart` | -- | Zero-config demo: auto-downloads model, runs curvature |

**Analysis:**

| Command | Path | Description |
|---------|------|-------------|
| `andamento audit` | 1 | Structural audit: coverage, duplication, coherence (no GPU) |
| `andamento curvature` | 2-4 | Full curvature analysis: what's helping, hurting, and wasted |
| `andamento analyze` | 2-4 | Power-user entry point (all flags, checkpoint trajectories) |
| `andamento saturate` | 2 | Compare curvature across multiple base models |

**Post-analysis:**

| Command | Path | Description |
|---------|------|-------------|
| `andamento compare` | 2-4 | Compare two reports to track repair progress |
| `andamento dashboard` | All | View a saved report interactively |
| `andamento serve` | 4 | Start GPU compute server for remote analysis |

<details>
<summary><b>Key Flags</b></summary>

| Flag | Description |
|------|-------------|
| `--base` / `--ft` | Base and fine-tuned model (HuggingFace ID or local path) |
| `--prefilter` | Pre-flight: base perplexity, calibrated threshold, sequence length stats |
| `--gradient` | Gradient analysis: effective adapter rank, pairwise cosines (~2x cost) |
| `--tessellate-by` | Metadata-based tessellation (e.g., `domain,tier`) |
| `--infer-task-type` | Infer task types from instruction text (translation, coding, math, etc.) |
| `--threshold` | Curvature fallback threshold in nats (overridden by calibration) |
| `--stability-check` | Test repair queue robustness under threshold perturbation |
| `--format-type json\|xml\|yaml` | Decompose curvature into format vs content tokens |
| `--loss-temperature` | Temperature sweep for curvature robustness (e.g., `"0.7,1.3"`) |
| `--no-cache` | Skip stage checkpoints, recompute everything |
| `--provider http` | Offload to remote `andamento serve` instance |
| `--provider-url URL` | URL for HTTP provider (persists in config.toml) |
| `--provider-api-key KEY` | API key for authenticated remote provider |
| `--benchmark PATH` | Benchmark dataset for alignment coverage |
| `--loss-samples auto` | Adaptive sqrt scaling (default), or integer for fixed |
| `--load-in-4bit` | 4-bit quantization (default) |
| `--open / --no-open` | Auto-open HTML report in browser |

See [CLI Reference](docs/cli_reference.md) for all flags.

</details>

## Installation

Pick the path that matches what you're trying to do:

### "I just want a structural audit (no GPU)"

```bash
pip install andamento
andamento validate your_data.jsonl
andamento audit your_data.jsonl
```

Coverage, duplication, coherence, topology — all run on CPU. Downloads a ~90MB embedding model on first run (sentence-transformers). If your data has metadata fields (category, source, etc.), embedding is skipped entirely.

### "I have HuggingFace models and a local GPU"

```bash
pip install 'andamento[models]'
andamento plan your_data.jsonl --base org/model --ft org/fine-tuned
andamento curvature your_data.jsonl --base org/model --ft org/fine-tuned
```

Loads models via Transformers. Quantizes to 4-bit by default (fits 7B models in ~6GB VRAM). 4-bit/8-bit quantization requires `bitsandbytes` (Linux only): `pip install 'andamento[quantize]'`. On macOS/Windows, use `--no-quantize` (needs more VRAM).

### "I have a LoRA adapter (local directory)"

```bash
pip install 'andamento[models]'
andamento curvature data.jsonl --base org/base-model --ft ./my-lora-adapter/
```

Andamento auto-detects `adapter_config.json`. The `--base` flag must point at the full base model, not the adapter.

### "I have no GPU"

```bash
pip install 'andamento[remote]'
# On a GPU machine (yours, a cloud VM, Colab):
pip install 'andamento[server]' && andamento serve --port 8080

# From your laptop:
andamento curvature data.jsonl --base X --ft Y \
    --provider http --provider-url http://gpu-machine:8080
```

All heavy computation runs remotely. Your laptop only needs CPU.

### Cloud deployment

On the GPU machine:

```bash
pip install 'andamento[server]'
andamento serve --port 8080 --host 0.0.0.0 --api-key YOUR_SECRET
```

**Security:**
- Always use `--api-key` when binding to `0.0.0.0`. Without it, anyone with network access can load models and run inference on your GPU.
- For internet-facing deployments, put the server behind a reverse proxy (nginx, Caddy) with TLS. `andamento serve` does not handle HTTPS itself.
- On private networks (VPC, tailnet, SSH tunnel), `--api-key` alone is sufficient.

**HuggingFace auth for gated models:**

```bash
# On the GPU machine, before starting the server:
huggingface-cli login
# or
export HF_TOKEN=hf_...
```

**Persistent HF cache** (Docker / ephemeral VMs):

```bash
# Mount the host cache so model downloads survive container restarts
docker run -v ~/.cache/huggingface:/root/.cache/huggingface ...
```

**Preflight check** (from your laptop):

```bash
andamento doctor --remote http://gpu-machine:8080 --api-key YOUR_SECRET
```

This checks GPU, CUDA, installed packages, HuggingFace auth, and cache directory on the remote server.

### "I use GGUF / Ollama / llama.cpp"

Your dataset works fine — `andamento audit` and `andamento validate` need no model at all. For curvature analysis, point `--base` at the original HuggingFace model your GGUF was quantized from. You don't need to stop using Ollama for inference; Andamento just needs the HF weights for a one-time diagnostic pass.

**How to find the HF source model:**

1. Go to the GGUF repo on HuggingFace (e.g., `TheBloke/Mistral-7B-v0.1-GGUF`)
2. Look at the model card — it almost always links the source under "Original model" or "Base model"
3. That link is your `--base` (e.g., `mistralai/Mistral-7B-v0.1`)

Common examples:

| You're running | Use as --base |
|---|---|
| `mistral:7b` in Ollama | `mistralai/Mistral-7B-v0.1` |
| `llama3:8b` in Ollama | `meta-llama/Meta-Llama-3-8B` |
| `TheBloke/...-GGUF` | Check "Original model" link on the model card |
| Your own GGUF quant | Use the HF ID you quantized from |

```bash
# Structural audit — works immediately, no model needed:
andamento audit your_data.jsonl

# Curvature — point at the HF source, not the GGUF:
pip install 'andamento[models]'
andamento curvature your_data.jsonl \
    --base mistralai/Mistral-7B-v0.1 \
    --ft ./your-fine-tune
```

`andamento plan` will catch GGUF paths and Ollama names before you download anything, and suggest the HF equivalent.

### Quick reference

```bash
pip install andamento                  # Structural audit only (CPU)
pip install 'andamento[models]'        # + Curvature analysis (GPU)
pip install 'andamento[remote]'        # + HTTP provider client
pip install 'andamento[server]'        # + GPU server (andamento serve)
pip install 'andamento[quantize]'      # + 4-bit/8-bit (bitsandbytes, Linux)
pip install 'andamento[all]'           # Everything
```

## Input Formats

- **JSONL** (messages, text, or prompt/completion) -- auto-detected
- **CSV/TSV** -- `--input-format csv`
- **HuggingFace datasets** -- pass dataset ID, use `--split` for custom splits

## Reproduce the Airoboros Audit

The Airoboros numbers above come from public data and public models. Step-by-step reproducibility scripts are included:

```bash
pip install 'andamento[models]'

# Full analysis with all diagnostics
andamento curvature jondurbin/airoboros-3.2 \
    --base mistralai/Mistral-7B-Instruct-v0.2 \
    --ft jondurbin/airoboros-m-7b-3.1.2 \
    --tessellate-by category \
    --prefilter --gradient --stability-check

# See reproducibility/README.md for step-by-step scripts
```

## Python API

Every CLI command has a typed Python equivalent:

```python
from andamento import run_audit, run_validate, run_compare

result = run_audit("your_data.jsonl")
print(result.verdict, result.n_cells)

v = run_validate("your_data.jsonl")
print(v.valid, v.n_examples)

cmp = run_compare("v1/report.json", "v2/report.json")
for d in cmp.summary_deltas:
    print(f"{d.label}: {d.delta:+.1f}")
```

All result types are frozen dataclasses. See [CLI Reference — Python API](docs/cli_reference.md#python-api) for full documentation.

## Documentation

| Doc | Description |
|-----|-------------|
| [Methodology](docs/methodology.md) | Measurement model, assumptions, and limits |
| [Interpreting Results](docs/interpreting_results.md) | How to read the report and act on it |
| [CLI Reference](docs/cli_reference.md) | Full flag documentation, examples, Python API |

## Privacy

By default, all analysis runs locally and your training data never leaves your machine. Network activity is limited to downloading public artifacts:

- **Model weights** from HuggingFace Hub (when using a HF model ID for `--base` or `--ft`)
- **Embedding model** (~90MB, sentence-transformers, first run only — skipped with `--tessellate-by`)
- **HuggingFace datasets** (when you pass a HF dataset ID instead of a local file)
- **`--provider http`** sends raw example texts and chat messages to your own remote `andamento serve` instance (your server, your network — nothing leaves it)

No telemetry. No phoning home. No third-party API calls. No cloud dependencies.

## Attribution

This release is centered on the package, articles, and reproducibility docs. If Andamento helps your work, please link the repository and, when useful, the methodology or reproducibility docs:

- Repository: https://github.com/Autarkis/andamento
- Methodology: [docs/methodology.md](docs/methodology.md)
- Reproducibility: [reproducibility/README.md](reproducibility/README.md)

## License

Apache 2.0. See [LICENSE](LICENSE).
