Metadata-Version: 2.4
Name: toktrim
Version: 0.2.0
Summary: Trim noisy logs, diffs, and chat transcripts before they become LLM tokens
Author-email: headWw <headWw@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/headWw/toktrim
Project-URL: Repository, https://github.com/headWw/toktrim
Project-URL: Issues, https://github.com/headWw/toktrim/issues
Project-URL: Changelog, https://github.com/headWw/toktrim/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/headWw/toktrim/tree/main/docs
Keywords: llm,cli,tokens,prompt,terminal
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: openai
Requires-Dist: openai>=2.0.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.50; extra == "anthropic"
Provides-Extra: bench
Requires-Dist: tiktoken>=0.5; extra == "bench"
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.110; extra == "dashboard"
Requires-Dist: uvicorn>=0.27; extra == "dashboard"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: httpx>=0.27; extra == "test"
Dynamic: license-file

# toktrim

Trim noisy logs, diffs, and chat transcripts before they become LLM tokens.

## What is it

`toktrim` is a small command-line tool that takes the kind of bulk text you tend to paste into an LLM — a 5,000-line CI log, a multi-file `git diff`, a long chat transcript — and rewrites it so the important parts stay and the noise drops out. The result is shorter, cheaper, and usually clearer for the model to read.

It is intentionally *not* a coding agent. It does one job: shrink context. You can pipe its output into your existing workflow, into a provider call (`toktrim review`, `toktrim debug`, `toktrim ask`), or wire it into [Claude Code](https://claude.com/claude-code) so trimming happens automatically.

The trims are lossy by design. Every mode preserves what matters most — errors, changed lines, the most recent turns — and drops what almost certainly does not — ANSI escape codes, hundreds of identical warnings, unchanged file regions, stale chat history. A bundled benchmark harness measures recall against fixed test cases so the trade-off is visible, not vibes-based.

## Before and after

A 156-line build log with 150 repeated `[WARN]` lines and one real failure at the end:

```
[OK] build started
[OK] compile module a
[OK] compile module b
[WARN] deprecated symbol legacyFn at module 0
[WARN] deprecated symbol legacyFn at module 1
... 148 more identical warnings ...
[WARN] deprecated symbol legacyFn at module 149
[FAIL] TypeError: cannot access undefined.id
[FAIL] build aborted
```

Run through `toktrim logs`:

```
[OK] build started
[OK] compile module a
[OK] compile module b
[WARN] deprecated symbol legacyFn at module 0
... (head retained: 37 lines)
[76 log lines omitted]
... (tail retained: 40 lines, including:)
[FAIL] TypeError: cannot access undefined.id
[FAIL] build aborted
```

`2,766 → 929` tokens. **66% saved.** ANSI codes stripped, warning storm collapsed, both `[FAIL]` lines preserved verbatim. (Source: `toktrim benchmark`, fixture `ansi_heavy_log`, `cl100k_base`.)

## Install

```bash
pip install toktrim                  # base install
pipx install toktrim                 # recommended for a CLI tool
pip install 'toktrim[openai]'        # add the OpenAI provider
pip install 'toktrim[anthropic]'     # add the Anthropic provider
pip install 'toktrim[openai,anthropic,bench,dashboard]'   # everything
```

For provider-backed commands, set the matching key:

```bash
export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
```

## Five-minute tour

The local transforms are pure: text in, smaller text out. No network, no API key needed.

### `toktrim auto` — the one to remember

Hand it a path. It detects whether the file is a log, a diff, or a transcript, and trims it accordingly. Tiny files pass through unchanged.

```bash
toktrim auto /tmp/build.log
toktrim auto failing-test-output.txt --show-mode
```

### `toktrim logs`

Strips ANSI escapes, collapses repeated lines, and keeps a head and tail of the file with everything in between marked `[N log lines omitted]`.

```bash
pytest 2>&1 | toktrim logs
toktrim logs --input failing.log --max-lines 120
```

### `toktrim diff`

Keeps file headers, hunk headers, and changed lines. Drops most surrounding context (a couple of lines stay for readability). Caps at 12 hunks by default.

```bash
git diff | toktrim diff
toktrim diff --input patch.diff --context-lines 2
```

### `toktrim convo`

Keeps the most recent turns of a transcript verbatim and converts older content into a short bulleted summary.

```bash
toktrim convo --input transcript.txt
cat transcript.txt | toktrim convo --recent-lines 30
```

### `toktrim stats`

Show the savings without printing the trimmed text.

```bash
git diff | toktrim stats --mode diff
pytest 2>&1 | toktrim stats --mode logs --json
```

`stats` uses `tiktoken` if installed; otherwise it falls back to a byte-based estimate.

## Sending to an LLM

Three opinionated developer verbs that combine "trim" + "send" + "store the result locally."

### `toktrim review` — review a diff

```bash
git diff | toktrim review --provider openai --show-stats
```

Trims the diff first, then asks the model for a code review. Add `--dry-run` to see the request without sending.

### `toktrim debug` — root-cause from a log

```bash
pytest 2>&1 | toktrim debug --provider openai --prompt "why is this failing?"
```

### `toktrim ask` — anything else

```bash
toktrim ask --provider openai --prompt "summarize the tradeoffs of this approach"
```

Switch providers per call with `--provider openai|anthropic`. When `--model` is omitted, toktrim picks a sensible default per provider. Reasoning effort defaults are also provider-specific (Anthropic defaults to `medium` and enables adaptive thinking; OpenAI defaults to `low`).

There is also a low-level `toktrim send` primitive that requires you to spell out `--mode` and `--prompt` explicitly. It's the foundation the developer verbs are built on; reach for it if you want full control.

## Local sessions and the dashboard

Every successful run stores a normalized message log and a metrics record under your user state directory (or `./.toktrim-state/` if that's read-only). Sessions persist across provider switches, so a conversation that started on OpenAI can continue on Anthropic without losing history.

```bash
toktrim sessions list
toktrim sessions show <session_id>
```

If you installed the `dashboard` extra, `toktrim dashboard` serves a small local web UI over the same data:

```bash
pip install 'toktrim[dashboard]'
toktrim dashboard          # http://127.0.0.1:8765
```

## Claude Code integration

If you use [Claude Code](https://claude.com/claude-code), `toktrim` ships an auto-invoking skill that trims logs and diffs on your behalf whenever you reference one. A `PreToolUse(Read)` hook also intercepts agent-self-produced bulk files (e.g., `pdflatex > /tmp/x.log` followed by Read) so trimming applies even when you don't reference the file in chat. A `PostToolUse(Bash)` hook goes further and trims verbose command output (pytest, builds, linters) in place — leaving file reads, patches, and structured JSON byte-for-byte intact.

```bash
toktrim install-claude-code
toktrim doctor                  # verify the install
toktrim uninstall-claude-code   # reverse it (idempotent)
```

The integration is opt-in, scoped narrowly, and includes safety guarantees against prompt-injection abuse. See [`docs/CLAUDE_CODE.md`](https://github.com/headWw/toktrim/blob/main/docs/CLAUDE_CODE.md) for the full setup, per-Read enforcement details, tunable thresholds, security model, and `--skill-mode` lockdown details.

## Security

`toktrim` is a local CLI. The main attack surface is `toktrim auto` being pointed at a sensitive path by a malicious or prompt-injected caller. The tool refuses well-known sensitive patterns (SSH keys, AWS creds, `.env` files, Claude Code's own state, and more) after symlink resolution, and `--skill-mode` pins the safety knobs so they cannot be widened by passed flags.

This is one layer, not a guarantee. Continue to protect true secrets with filesystem permissions (`chmod 0600`).

For the full threat model, mitigation list, and how to report a vulnerability, see [`SECURITY.md`](https://github.com/headWw/toktrim/blob/main/SECURITY.md).

## Configuration

`toktrim` follows the standard CLI convention: per-user knobs are environment variables. Each is read at invocation time, so changing a value takes effect on the next call.

| Variable | Default | Purpose |
|----------|---------|---------|
| `TOKTRIM_MIN_AUTO_TOKENS` | `500` | Token-count floor below which `toktrim auto` returns `raw` instead of trimming. Raise to skip more small inputs; lower (or set `0`) to trim aggressively. |
| `TOKTRIM_AUTO_ALLOW_SENSITIVE` | unset | Set to any truthy value to bypass the sensitive-path denylist. Honored only outside `--skill-mode`; the Claude Code skill ignores it. |
| `TOKTRIM_STATE_DIR` | `$XDG_STATE_HOME/toktrim` or `~/.local/state/toktrim` | Override where session history and run metrics are written. |
| `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` | unset | Required only for provider-backed commands (`send`, `review`, `debug`, `ask`). |

Quick example — raise the floor for a single invocation:

```bash
TOKTRIM_MIN_AUTO_TOKENS=2000 git diff | toktrim auto --show-mode
```

> **Migration note (0.2.0):** the `TOKTRIM_MIN_AUTO_TOKENS` default was raised from `200` to `500`. Inputs in the 200–500 token range that previously trimmed will now route to `raw` (skip trim) by default. Set `TOKTRIM_MIN_AUTO_TOKENS=200` to restore the prior behavior.

## Quality benchmarks

Trims are lossy, so the project measures the trade-off explicitly. The bundled `toktrim benchmark` command runs every transform against fixture inputs and reports three metrics per fixture:

- **savings** — token reduction vs the original input (using `cl100k_base`).
- **recall** — fraction of declared `must_retain` facts that survive the trim.
- **prefix stability** — how much of the trimmed output's token prefix stays identical when an elidable region of the input is mutated. Measures prompt-cache friendliness.

```bash
pip install 'toktrim[bench]'
toktrim benchmark
toktrim benchmark --fail-under 90 --fail-savings-under 50 --fail-prefix-under 80
```

The committed baseline lives in `src/toktrim/benchmarks/baseline.json`. See [`docs/benchmarks.md`](https://github.com/headWw/toktrim/blob/main/docs/benchmarks.md) for how fixtures work and how to add new ones.

## Roadmap

- More task profiles such as `fix`, `explain`, and `commit-msg`
- Provider adapters beyond OpenAI and Anthropic
- Git-aware file scoping beyond raw diff input
- Cache-stable prompt ordering for providers that support prompt caching
- Benchmark fixtures for additional CI logs, stack traces, and multi-turn coding chats

## License and contributing

MIT. Issues and pull requests welcome at [github.com/headWw/toktrim](https://github.com/headWw/toktrim) — see [CONTRIBUTING.md](https://github.com/headWw/toktrim/blob/main/CONTRIBUTING.md) to get set up.
