Metadata-Version: 2.4
Name: AnlordAbliterator
Version: 1.2.0.post1
Summary: Anlord Abliterator — automated LLM abliteration and evaluation
Home-page: https://github.com/justbedwarsplay/AnlordAbliterator
Author: Anlord Abliterator Development Team
Author-email: Anlord Abliterator Development Team <anlixide@gmail.com>
License: AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/justbedwarsplay/AnlordAbliterator
Project-URL: Repository, https://github.com/justbedwarsplay/AnlordAbliterator
Project-URL: Issues, https://github.com/justbedwarsplay/AnlordAbliterator/issues
Keywords: llm,abliteration,benchmark,evaluation
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: GPU
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.2.0
Requires-Dist: torchvision>=0.17.0
Requires-Dist: accelerate>=1.0.0
Requires-Dist: transformers[kernels]>=4.46.0
Requires-Dist: huggingface-hub>=0.24.0
Requires-Dist: bitsandbytes>=0.44.0
Requires-Dist: peft>=0.13.0
Requires-Dist: optuna>=4.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: datasets>=2.18.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: py-cpuinfo>=9.0.0
Requires-Dist: rich>=13.7.0
Requires-Dist: questionary>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: tomli-w>=1.0.0
Requires-Dist: nvidia-ml-py>=12.0.0
Requires-Dist: requests<3.0,>=2.32.0
Requires-Dist: chardet<6.0,>=5.0.0
Provides-Extra: dev
Requires-Dist: ruff>=0.14.5; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Provides-Extra: lmeval
Requires-Dist: lm-eval[hf]<0.5,>=0.4.9; extra == "lmeval"
Provides-Extra: native
Requires-Dist: datasets>=2.18.0; extra == "native"
Provides-Extra: research
Requires-Dist: geom-median>=1.2; extra == "research"
Requires-Dist: scikit-learn>=1.3; extra == "research"
Requires-Dist: pacmap>=0.8; extra == "research"
Requires-Dist: imageio>=2.31; extra == "research"
Requires-Dist: matplotlib>=3.8; extra == "research"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# AnlordAbliterator

<p align="center">
  <img src="https://img.shields.io/badge/Python-3.10%2B-3776AB?style=for-the-badge&logo=python&logoColor=white" />
  <img src="https://img.shields.io/badge/PyTorch-2.2%2B-EE4C2C?style=for-the-badge&logo=pytorch&logoColor=white" />
  <img src="https://img.shields.io/badge/CUDA-11.8%20%2F%2012.x-76B900?style=for-the-badge&logo=nvidia&logoColor=white" />
  <img src="https://img.shields.io/badge/License-AGPL--3.0-00A86B?style=for-the-badge" />
</p>

<p align="center">
  <strong>Automated LLM abliteration. Honest before/after evaluation. Beautiful reports.</strong>
</p>

<p align="center">
  Download with progress → Baseline benchmarks → Abliteration → Merge LoRA → Re-evaluate → Compare → HTML / JSON / CSV
</p>

<p align="center">
  <a href="#quick-start">Quick Start</a> •
  <a href="#features">Features</a> •
  <a href="#benchmarks">Benchmarks</a> •
  <a href="#output">Output</a> •
  <a href="#gguf-quantization">GGUF</a> •
  <a href="#publishing">Publishing</a>
</p>

---

## Features

| | What it does |
|---|---|
| **🔧 Native abliteration** | In-process Optuna search over refusal direction & layer weights, Pareto selection (`refusals ↓` + `KL ↓`), merges LoRA into base model |
| **🧩 Scorer plugins** | Extensible objectives: built-in `Refusals`, `KL divergence`, `BenchmarkScore` (lm-eval) — or load your own from a `.py` file / import path. Multiple instances, per-scorer settings, `minimize` / `maximize` / `none` per objective |
| **📐 Residual analysis** | `--print-residual-geometry`: per-layer cosine similarities, norms & silhouettes (means + geometric medians); `--plot-residuals`: PaCMAP projections per layer + animated GIF (optional `research` extra) |
| **♻️ Reproduction mode** | Every run saves full ablation parameters + a shareable `reproduce/` bundle (settings, scores, package versions, SHA-256 hashes, Optuna journal). `--reproduce <file \| user/model \| HF URL>` re-applies a published ablation and verifies weight hashes byte-for-byte |
| **🎚️ Component filter** | `--components attn` ablates only attention and leaves MLP untouched (prefix matching, recorded in bundles, reproducible) |
| **📤 Export strategy** | `--export-strategy merge` (default, full model) or `adapter` (LoRA only) — the actually used strategy is recorded everywhere |
| **⬇️ Visible downloads** | `huggingface_hub.snapshot_download` with progress bars into `cache/hub` |
| **🔐 Ephemeral HF auth** | Optional `HF_TOKEN` kept only in the current process (`questionary` masked prompt). `Enter` = anonymous. `--no-hf-token-prompt` for CI |
| **📊 Native runner (default)** | Custom evaluators on `datasets` + `transformers` — no `lm-eval` required, works offline, tolerant to `trust_remote_code` deprecation |
| **🧪 Extended suite** | **29 tasks / 17 families** — 6 classic + GPQA, MMLU-Pro, Ifstruct, ParseBench, ExtractBench, ScreenSpot-Pro, MMMU-Pro. All selectable via `--tasks` or checkbox |
| **⚖️ Before → After** | Same benchmarks run on baseline and abliterated models, `comparison.json` with deltas |
| **💻 Hardware-aware** | `psutil` + `pynvml` monitoring. `HardwarePlanner` auto-selects load-time `quantization` / `device_map` / `max_memory` / `batch_size` for your GPU (see note below) |
| **📝 Reports** | Interactive `report.html` + `report.json` + `report.csv` + per-benchmark `*.json` |
| **🔄 Resume** | `--resume` (on by default) — skips completed abliteration + benchmark shards by `model_id + config` hash. Corrupted `*.tmp` files are ignored |
| **🪟 Windows-first** | `questionary` + `rich` prompts, no `curses`, handles `triton-windows` / `causal-conv1d` gracefully |

> **Note on `--quantization`:** this flag in AnlordAbliterator controls only **how the model is loaded into memory** during abliteration/evaluation (VRAM saving via `bitsandbytes` + CPU offload). It does **not** quantize the saved abliterated model — the output in `models/abliterated/` is always full-precision `safetensors` (BF16/F16). For distribution quants (GGUF `Q4_K_M` etc.) see [GGUF Quantization](#gguf-quantization).

## Architecture

```
Hugging Face model (any causal LM)
        │
        ▼ snapshot_download → cache/hub
   HardwarePlanner ──▶ dtype / device_map / max_memory / batch_size
        │
        ├─► Baseline benchmarks (optional) ──► results/baseline/*.json
        │
        ▼ Abliteration ── trials × (reset → abliterate → KL + refusal count)
        │              └─ Pareto front → best trial → merge LoRA
        │                              └─► models/abliterated/
        │
        ├─► Abliterated benchmarks ──► results/abliterated/*.json
        │
        ▼ Comparison ──► results/comparison.json
        │
        ▼ Reports ──► reports/report.{html,json,csv}
```

Works with any supported architecture (Qwen, Llama, Mistral, Gemma, etc.).

## Requirements

- Python 3.10+
- PyTorch 2.2+ with CUDA 11.8/12.x recommended (CPU also works)
- 16GB+ RAM; 8GB VRAM is enough for 0.8B–7B models (larger models auto-use load-time `bnb_4bit` + CPU offload)
- `transformers>=4.46`, `datasets>=2.18`, `accelerate`, `bitsandbytes`, `peft`, `optuna`

See `pyproject.toml` / `requirements.txt` for the full list.

## Installation

**Windows (PowerShell):**
```powershell
python -m venv anlord_env
anlord_env\Scripts\activate
python -m pip install -U pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -e .
# optional: residual geometry + PaCMAP plots
pip install -e .[research]
# optional: lm-eval harness for exact leaderboard reproduction
pip install -e .[lmeval]
```

**Linux / WSL / Colab:**
```bash
sudo apt update && sudo apt install -y python3.10-venv build-essential git
python3 -m venv anlord_env && source anlord_env/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -e .
```

Verify:
```bash
python -m src.anlord --info
pytest tests/ -v
```

The distribution installs as **AnlordAbliterator** (pip name) and provides the
`AnlordAbliterator` console command; the importable package is `anlord`.
Most people run it from the project root (the folder above `src`):

```bash
python -m src.anlord --info
```

## Quick Start

### Interactive (recommended on Windows)

```bash
python -m src.anlord
```

You will be asked for:
1. **HF token** (masked, `Enter` to skip) — needed for gated datasets like `Idavidrein/gpqa`
2. **Model ID** (e.g. `Qwen/Qwen2.5-7B`, `meta-llama/Llama-3.2-3B`, or a local path)
3. **Output dir** (where `results/`, `reports/`, `models/`, `cache/` live)
4. **Tasks** — checkbox (`Space` to toggle, `Enter` to confirm)
5. **Mode** — `quick` (100 samples) or `full`

### CLI

```bash
# Classic 6 + 8 extended (14 tasks) — recommended
python -m src.anlord --model Qwen/Qwen2.5-7B --output ./output --tasks all
# all = extended = mmlu,gsm8k,hellaswag,arc_challenge,winogrande,truthfulqa,
#                  gpqa,gpqa_diamond,mmlu_pro,ifstruct,parsebench,extractbench,screenspot_pro,mmmu_pro

# Only the new benchmarks
python -m src.anlord --model Qwen/Qwen2.5-7B --output ./output \
  --tasks gpqa_diamond,mmlu_pro,ifstruct,parsebench,extractbench,screenspot_pro,mmmu_pro

# Everything (29 tasks including mmlu subsets)
python -m src.anlord --model Qwen/Qwen2.5-7B --output ./output --tasks full

# Quick smoke test
python -m src.anlord --model HuggingFaceTB/SmolLM2-135M --output ./tmp --mode quick --limit 20 --tasks standard

# Gated datasets need a token
# PowerShell:  $env:HF_TOKEN="hf_xxx"
# bash:        export HF_TOKEN=hf_xxx
# or paste it when prompted (https://huggingface.co/settings/tokens → Read)
```

## Benchmarks

Choose via `--tasks` or the interactive checkbox:

| Flag | Expands to |
|---|---|
| `all` / `*` | `extended` — 14 tasks (6 classic + 8 new) |
| `full` | all 29 tasks |
| `standard` | 6 classic only |
| `parsing` / `vision` / `gpqa` / `academic` … | group |
| `mmlu,gsm8k,gpqa_diamond` | explicit comma list |

### Task table (Native runner)

| ID | Display | Dataset | Split | Few-shot | Metric |
|---|---|---|---|---|---|
| `mmlu` | MMLU | `cais/mmlu` | test (14k) | 5 | acc |
| `gsm8k` | GSM8K | `openai/gsm8k` | test | 5 | exact_match |
| `hellaswag` | HellaSwag | `Rowan/hellaswag` | validation | 10 | acc_norm |
| `arc_challenge` | ARC-Challenge | `allenai/ai2_arc` | test | 0 | acc_norm |
| `winogrande` | Winogrande | `allenai/winogrande` | validation | 0 | acc |
| `truthfulqa` | TruthfulQA MC2 | `truthful_qa` | validation | 0 | acc |
| `gpqa` / `gpqa_diamond` | GPQA Diamond | `Idavidrein/gpqa:gpqa_diamond` | train (198) | 0 | acc |
| `gpqa_main` | GPQA Main | `Idavidrein/gpqa:gpqa_main` | train | 0 | acc |
| `gpqa_extended` | GPQA Extended | `Idavidrein/gpqa:gpqa_extended` | train | 0 | acc |
| `mmlu_pro` | MMLU-Pro | `TIGER-Lab/MMLU-Pro` | test (12k) | 5 | acc |
| `ifstruct` | Ifstruct v1.0 | `LiquidAI/ifstruct-v1.0` | test (2k) | 0 | acc |
| `parsebench` | ParseBench | `llamaindex/ParseBench` | test | 0 | F1 |
| `extractbench` | ExtractBench | `llamaindex/ExtractBench` | test | 0 | F1 |
| `screenspot_pro` | ScreenSpot-Pro | `likaixin/ScreenSpot-Pro` | test | 0 | acc |
| `mmmu_pro` | MMMU-Pro | `MMMU/MMMU_Pro` | test | 0 | acc |
| `mmlu_*` (×3), `arc_easy`, `math`, `squad_v2`, `race`, `humaneval`, `mbpp`, `ifeval` | subsets / extras | various | — | 0–5 | acc / F1 / pass@1 |

> The native GPQA/MMLU-Pro scorers use log-prob over answer letters (` A`/` B`/…) with length normalization. On 198-sample Diamond this differs ±2% from the `lm-eval` harness — delta `baseline → abliterated` stays fair because both sides use the same scorer.

## CLI Options

```
Model:
  --model, -m            HF ID or local path
  --model-commit         pin a commit hash
  --output, -o           output root (contains results/reports/models/cache)
  --cache-dir            HF cache dir (default ./cache)
  --prefetch-model / --no-prefetch-model

Auth:
  --hf-token-prompt / --no-hf-token-prompt

Abliteration:
  --trials, -t           trials (default 100)
  --timeout              seconds (default 43200 = 12h)
  --eval-prompts         prompts for KL/refusal (default 100)
  --backend              native | auto (default native)
  --row-normalization    none | pre | full (default full)
  --no-orthogonalize     disable projected abliteration
  --subspace-rank        1 = single direction, 3-5 = multi-vector subspace via SVD
  --capability-proxy     3rd Optuna objective: tiny MMLU proxy (refusals ↓ KL ↓ MMLU ↑)
  --components           comma-separated include list, e.g. '--components attn' = attention-only
  --scorers              scorer plugins as JSON or a JSON file path

Residual analysis (optional 'research' extra):
  --print-residual-geometry   per-layer geometry table (also saved to file + JSON)
  --plot-residuals            PaCMAP plots per layer + animated GIF
  --residual-plot-path        plots directory (default: <output>/plots)

Export / reproduction:
  --export-strategy      merge (default) | adapter (LoRA only)
  --max-shard-size       safetensors shard size (default 5GB)
  --reproducibility-info full | basic | none  (reproduction bundle contents)
  --reproduce            reproduce.json path, 'user/model' repo id, or HF model URL
  --ignore-mismatches    reproduce despite environment differences (default: warn)
  --print-debug-information

Benchmarks:
  --tasks, --benchmarks, -b   see table above
  --mode                 quick | full
  --limit, -l            samples per benchmark (quick default 100)
  --num-fewshot          override few-shot
  --native-benchmarks / --no-native-benchmarks  (default native)

Hardware (load-time only, not saved):
  --dtype                auto | float16 | bfloat16 | float32
  --device               cuda | cpu | mps
  --device-map           auto | cuda | cpu
  --quantization         auto | none | bnb_4bit | bnb_8bit  (how to fit model in VRAM)
  --batch-size           inference batch for abliteration (refusal counting); omit to
                         autotune from VRAM headroom (default: autotune)
  --max-memory           e.g. '{"0":"7GB","cpu":"11GB"}'

Pipeline:
  --skip-baseline --skip-abliteration --skip-benchmarks
  --resume / --no-resume
  --baseline-evaluate    force a separate Abliteration baseline pass
  --yes                  auto-confirm prompts (kept for compat)

Reproducibility:
  --seed, -s             random seed (default 42)

Other:
  --info --verbose, -v --quiet, -q --log-file
```

## Output

```
output/
├── cache/hub/                          # HF snapshot (safetensors + tokenizer)
├── models/
│   ├── abliterated/  # ← your merged model (ready to use)
│   │   ├── config.json
│   │   ├── tokenizer.json
│   │   ├── model.safetensors (or shards)
│   │   ├── native_abliteration_metrics.json
│   │   ├── abliteration_reproduction.json   # full ablation parameters (--reproduce ready)
│   │   ├── abliteration_pareto_front.json   # all Pareto trials + refusals/KL
│   │   ├── pareto_trials/trial_N.json       # each Pareto trial as a reproduce file
│   │   └── reproduce/                       # shareable bundle (if fully pinned)
│   │       ├── reproduce.json · SHA256SUMS
│   │       ├── requirements.txt · config.json · README.md
│   │       └── <model>.jsonl                # Optuna journal copy
│   ├── abliteration_study/                  # Optuna study (for resume)
│   └── abliteration_baseline/               # baseline metrics
├── plots/<model>/                           # residual analysis (if enabled)
│   ├── layer_001.png … · animation.gif
│   ├── residual_geometry.txt
│   └── residual_geometry.json
├── results/
│   ├── baseline/*.json
│   ├── abliterated/*.json
│   ├── comparison.json
│   ├── abliteration.json
│   └── reproduction_hashes.json             # reproduction mode only
├── reports/
│   ├── report.html
│   ├── report.json
│   └── report.csv
├── run_config.json
└── environment.json
```

The abliterated model is already merged (LoRA weights folded in). Load it directly:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("output/models/abliterated", trust_remote_code=True, device_map="auto")
tok = AutoTokenizer.from_pretrained("output/models/abliterated", trust_remote_code=True)
```

## Reproduce a published model

Every run saves the complete ablation recipe next to the model. To re-apply someone else's
ablation (or your own on another machine) and verify the weights byte-for-byte:

```bash
# from a local reproduce.json / saved trial file
python -m src.anlord --reproduce "C:/models/Qwen3.5-0.8B/models/abliterated/reproduce/reproduce.json"
# straight from Hugging Face
python -m src.anlord --reproduce username/model-name
# after `pip install AnlordAbliterator` the same run is just:
AnlordAbliterator --reproduce username/model-name
```

The pipeline restores the original settings and parameters, re-applies the ablation, exports
the model and checks the SHA-256 hashes of the weight files (`reproduction_hashes.json`).
Environment differences are reported in a table; `--ignore-mismatches` proceeds anyway.

## Scorer plugins

Objectives are plugins. Defaults: refusals (keyword rate) + KL divergence, both minimized.
Add your own via the `--scorers` JSON (or the `scorers` / `scorer_settings` settings):

```json
{
  "scorers": [
    {"plugin": "anlord.scorers.keyword_rate.KeywordRate", "optimization": "minimize", "instance_name": "refusals"},
    {"plugin": "anlord.scorers.kl_divergence.KLDivergence", "optimization": "minimize"},
    {"plugin": "anlord.scorers.benchmark_score.BenchmarkScore", "optimization": "maximize",
     "instance_name": "piqa"},
    {"plugin": "C:/plugins/my_scorer.py:MyScorer", "optimization": "none"}
  ],
  "scorer_settings": {
    "KeywordRate_refusals": {"score_name": "Refusals"},
    "BenchmarkScore_piqa": {"score_name": "PIQA acc_norm", "task": "piqa", "metric": "acc_norm,none"}
  }
}
```

Built-in plugins live under the `anlord.scorers.*` namespace; external ones are referenced as
`path/to/plugin.py:ClassName` or `module.submodule.ClassName`. Only runs whose model and
datasets are pinned Hugging Face paths and whose scorers are all reproducible built-ins get a
reproduction bundle.

## GGUF Quantization

AnlordAbliterator saves the abliterated model as full-precision `safetensors`. To distribute smaller files for `llama.cpp` / Ollama / LM Studio, convert to GGUF after the run.

### Default (no quant)

The default output is `model.safetensors` in BF16/F16 — no quantization, full quality. Keep it for Hugging Face.

| GGUF type | Bits | Size vs F16 | Quality | Use |
|---|---|---|---|---|
| `f16` | 16 | 1.0× | lossless | base for re-quant, or direct GGUF use |
| `bf16` | 16 | 1.0× | lossless | same as f16 |
| `q8_0` | 8 | ~0.50× | near-lossless | best quant quality, good default |
| `q6_k` | 6 | ~0.39× | very high | K-quant |
| `q5_k_m` | 5 | ~0.33× | high | K-quant medium, sweet spot |
| `q5_k_s` | 5 | ~0.32× | high- | K-quant small |
| `q5_0` | 5 | ~0.33× | medium | legacy 5-bit |
| `q4_k_m` | 4 | ~0.26× | good | **smallest recommended** — ~3.8× smaller than F16 |
| `q4_k_s` | 4 | ~0.25× | okay | smaller, more loss |
| `q4_0` | 4 | ~0.26× | okay | legacy 4-bit |
| `q3_*` / `q2_*` | 3/2 | ~0.20× | low | not recommended, beyond Q4_K_M |

## Troubleshooting

- **`Dataset is gated` (GPQA)** → set `HF_TOKEN` (`$env:HF_TOKEN="hf_..."` / `export HF_TOKEN=...`) or paste it at the prompt.
- **CUDA OOM** → `--quantization bnb_4bit` or use a smaller base model; `HardwarePlanner` does this automatically with `auto` (load-time only).
- **KL > 0.5** → model is damaged — try fewer `--trials`, different `--eval-prompts`, or pick another Pareto trial from `results/ablation.json`.
- **Resume stuck on zeros** → `python -m src.anlord --no-resume` once to clear corrupted `*.tmp` shards.

## License

AGPL-3.0-or-later. Each base model has its own license — respect it when redistributing abliterated weights.

## Acknowledgments

- [Hugging Face Transformers & Datasets](https://huggingface.co)
- [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) — optional

<p align="center"><em>AnlordAbliterator — make models honest, keep reports beautiful.</em></p>
