Metadata-Version: 2.4
Name: hdd-analyzer
Version: 1.2.0
Summary: Triage old hard drives for salvageable files with cost-capped LLM classification and a ranked report.
Keywords: hard-drive,triage,classification,jev,typesafe,ocr,data-recovery,forensics
Author: Aaron Meese
Author-email: Aaron Meese <aaron@meese.dev>
License-Expression: BSD-3-Clause
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Filesystems
Classifier: Topic :: Utilities
Requires-Dist: chardet>=7.6.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: pypdf>=6.19.0
Requires-Dist: pypdfium2>=5.13.0
Requires-Dist: python-dotenv>=1.2.3
Requires-Dist: typesafe-sdk>=0.7.0
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/ajmeese7/hdd-analyzer
Project-URL: Repository, https://github.com/ajmeese7/hdd-analyzer
Project-URL: Issues, https://github.com/ajmeese7/hdd-analyzer/issues
Description-Content-Type: text/markdown

# hdd-analyzer

<p align="center">
  <img src="https://raw.githubusercontent.com/ajmeese7/hdd-analyzer/master/promo/hdd-analyzer.gif" alt="hdd-analyzer walking a drive, scanning files with an LLM, and ranking the keepers">
</p>

Old hard drives pile up faster than anyone can manually sort them, and most of what is on them is "dark data": junk, caches, and installers with a handful of genuinely irreplaceable files buried inside. hdd-analyzer walks a drive for free, estimates the token and dollar budget to have [Jev](https://typesafe.ai) (TypeSafe's [System One](https://docs.typesafe.ai/concepts/system-one) decision model, also served through [OpenRouter](https://openrouter.ai/typesafe/jev-1.13) and [Vercel AI Gateway](https://vercel.com/ai-gateway/models/jev)) read every file, then runs a capped classification pass and produces a ranked report so a human can decide what is worth keeping before the drive gets wiped. Cheap, fast inference has made this kind of exhaustive per-file triage practical in a way it was not a few years ago.

## What is Jev?

Jev is [TypeSafe](https://typesafe.ai)'s System One decision model: instead of free-text generation, it answers a fixed set of yes/no questions (nouls), scored questions, and multiple-choice questions against a piece of state, returning calibrated probabilities instead of prose, which makes it well suited to structured classification like this tool's per-file rubric. hdd-analyzer talks to Jev directly through TypeSafe's API, through [OpenRouter's Decisions endpoint](https://openrouter.ai/typesafe/jev-1.13), or through [Vercel AI Gateway's TypeSafe-compatible endpoint](https://vercel.com/docs/ai-gateway/sdks-and-apis/typesafe), auto-detected from your API key.

## Install

```
pip install hdd-analyzer
```

Or as an isolated tool with `uv`:

```
uv tool install hdd-analyzer
```

Requires Python 3.13+. To run the latest unreleased code instead, install from GitHub with `pip install git+https://github.com/ajmeese7/hdd-analyzer.git`.

## Configuration

Create a `.env` file in the working directory you'll run `hdd-analyzer` from; it is loaded automatically.

| Variable | Required | Description |
| --- | --- | --- |
| `TYPESAFE_API_KEY` | yes | A native TypeSafe key, an OpenRouter key (prefix `sk-or-`), or a Vercel AI Gateway key (prefix `vck_`). The provider is auto-detected from the prefix. |
| `JEV_PROVIDER` | no | `typesafe`, `openrouter`, or `vercel`, overrides the key-prefix auto-detection. |

Vercel AI Gateway's free tier rate-limits each model to 30 requests per minute; pass `scan --rpm 28` there, or buy any amount of gateway credits to move to the paid tier, which lifts the limit.
| `TESSERACT_CMD` | no | Full path to `tesseract.exe` if it is not on PATH. Needed only for the optional `ocr` command. |

```
TYPESAFE_API_KEY=your-key-here
```

Do not double-quote `TESSERACT_CMD` in `.env`. `python-dotenv` treats a double-quoted value as an escaped string, so `"\tesseract.exe"` becomes a literal tab followed by `esseract.exe`. Leave it unquoted or single-quoted.

## Quickstart

Everything below assumes an old drive mounted at `D:\` (or `/mnt/olddrive` on Linux/macOS) with the interesting content under `Users`.

```
hdd-analyzer walk D:\ --run olddrive --include Users
hdd-analyzer estimate --run olddrive
hdd-analyzer scan --run olddrive --cap 5
hdd-analyzer report --run olddrive
hdd-analyzer manifest --run olddrive
```

`walk` inventories the drive for free. `estimate` prices out a full scan with no network calls. `scan` extracts local content and sends one classification call per file to Jev, prompting for confirmation and stopping at the cap. `report` renders `runs/olddrive/report.md`, `report.csv`, and `report.html`, ranked overall and per category. Open `report.html` in a browser for a navigable directory tree that shows where the notable files cluster, so you know where to look first when going through the drive by hand. `manifest` turns the results into a salvage copy list in `runs/olddrive/manifest/`, ready to hand to `robocopy` or `rsync` before the drive is wiped.

## How it stays cheap and safe

- `walk` is a pure filesystem pass: no network calls, no API spend.
- Duplicate files are deduped by content hash (text/code/doc) or by size and filename (everything else), so Jev never classifies the same file twice.
- Dependency caches, build output, and other junk directories (`node_modules`, `.git`, `AppData\Local`, and dozens more) are skipped before they ever hit the candidate list.
- `scan --prefilter` asks Jev about each directory listing first (only directories with 10 or more files) and skips the files of directories it judges not worth reading. Measured on a 50k-file Windows profile: half the calls, with 214 of 218 content-verified credential hits (every one valued 2.0 or higher), every file valued 2.5 or higher, and 97.7% of notable rows still found; on a curated Documents archive it skips nothing and costs under 1%. Decisions land in `runs/NAME/prefilter.jsonl` so you can see exactly what was skipped.
- `estimate` and `scan` print the candidate count, estimated tokens, and estimated dollar cost, and `scan` asks for interactive `y` confirmation before spending anything (skip with `--yes`).
- `scan` enforces a hard spend cap (`--cap`, default $5) before dispatching each batch, using actual spend so far plus a worst-case estimate of the in-flight batch.
- A preflight canary call and a circuit breaker abort the scan immediately on authentication or billing errors (HTTP 401/403/402), instead of burning through the candidate list on a broken key.
- `scan` is resumable: re-running it skips files whose dedupe key is already in `results.jsonl`, and failed files retry automatically.
- Every result is labeled `content` (Jev read the actual file), `ocr` (Jev read a Tesseract transcription), or `name-only` (Jev only saw the file name and metadata), and reports list content-verified hits first so a suggestive filename like `passport.pdf` with no extractable text can't be mistaken for a verified hit.
- The tool never writes outside its own `runs/` directory, and drives are only ever opened read-only.

## Optional: OCR for images and scanned PDFs

`scan` never runs OCR itself; images and no-text PDFs get judged by Jev on filename alone. The separate `ocr` command is local, free, and makes zero API calls: it revisits an existing run's results, OCRs the highest-ranked name-only images and no-text PDFs with Tesseract, and writes `runs/NAME/ocr.jsonl`. Then `scan --from-ocr` re-classifies those rows using the OCR excerpt instead of re-running extraction.

Install Tesseract first:

```
winget install UB-Mannheim.TesseractOCR
```

On Linux or macOS:

```
sudo apt install tesseract-ocr
brew install tesseract
```

Then:

```
hdd-analyzer ocr --run olddrive
hdd-analyzer scan --run olddrive --from-ocr --cap 0.25
hdd-analyzer report --run olddrive
```

## Commands

- `walk ROOT --run NAME [--include SUBPATH ...]` - free, deterministic inventory walk. Writes `runs/NAME/inventory.jsonl`.
- `estimate --run NAME` - token and dollar estimate for a scan. No network calls.
- `scan --run NAME [--cap USD] [--limit N] [--yes] [--dry-run] [--concurrency N] [--rpm N] [--prefilter] [--only-name-only] [--outdated-rubric] [--exclude-ext EXT[,EXT...]] [--from-ocr]` - local extraction plus one Jev call per file, under a hard spend cap. Appends to `runs/NAME/results.jsonl` and is resumable. `--rpm` paces requests for rate-limited providers (Vercel AI Gateway's free tier allows 30 per minute; use `--rpm 28`). `--outdated-rubric` re-scores every row still carrying an older rubric version, since scores from different rubric versions are not comparable. `--prefilter` triages directories before any per-file call (see above).
- `annotate --run NAME [--all] [--top N] [--min-prob P]` - backfills `metadata_only`/`extraction_status` onto existing results by re-running local extraction only, zero API calls.
- `ocr --run NAME [--top N] [--all] [--min-value V] [--limit N]` - local, free OCR pass over an existing run's name-only image and no-text-PDF rows.
- `report --run NAME [--top N] [--min-prob P] [--min-value V]` - renders `runs/NAME/report.md`, `report.csv`, and `report.html`, ranked overall and per category. `--min-value` and `--min-prob` set which files count as notable in the HTML report; the defaults match `manifest`.
- `manifest --run NAME [--min-value V] [--min-prob P] [--out DIR]` - the salvage deliverable: read-only over `results.jsonl`/`inventory.jsonl`, writes copy lists and summaries to `runs/NAME/manifest/`.

## Output files

- `runs/NAME/inventory.jsonl` - one row per walked file: path, size, mtime, extension, category, dedupe key.
- `runs/NAME/walk-errors.log` - files the walker could not read (permissions, long paths).
- `runs/NAME/results.jsonl` - one row per scanned file: all Jev probabilities, value score, extraction status, tokens used.
- `runs/NAME/prefilter.jsonl` - with `scan --prefilter`, one row per directory asked about: its `worth_scanning` probability, `kind`, and file count. Directories below 0.05 were skipped.
- `runs/NAME/ocr.jsonl` - OCR excerpts for name-only images and no-text PDFs; contains real file content, keep it out of version control.
- `runs/NAME/report.md` / `report.csv` - the ranked report, overall and per category.
- `runs/NAME/report.html` - self-contained interactive report: summary tiles, the directories holding the most notable files, a collapsible directory tree with notable/scanned counts per subtree, and a searchable, filterable table of every notable file. Contains paths and scores only, never file content.
- `runs/NAME/manifest/copy-list.txt`, `copy-list-verified.txt`, `copy-list-name-only.txt` - one absolute source path per line, ready for `robocopy` or `rsync`.
- `runs/NAME/manifest/credentials.md` - every row that scored high on credentials, paths only, with a reminder to rotate anything still valid.
- `runs/NAME/manifest/by-category.md` - a table per category plus a top-30 directory rollup.
- `runs/NAME/manifest/summary.txt` - counts, total bytes, and a documented copy-tool invocation.

## Privacy

`scan` sends excerpts of file content to whichever API provider you configure (TypeSafe, OpenRouter, or Vercel AI Gateway) so Jev can classify them. `runs/ocr.jsonl` and `runs/results.jsonl` hold those excerpts locally, which can include credentials or other personal information pulled straight from your files; `runs/` is gitignored for this reason and should never be committed or shared as-is.

## Development

```
uv sync
uv run pytest
```

## License

BSD 3-Clause. See [LICENSE](LICENSE).
