Metadata-Version: 2.4
Name: token-diet-cli
Version: 0.2.0
Summary: Diagnostic tools and methodology to audit and optimize token footprint in AI coding workflows
Author-email: joseoper <joseoper@users.noreply.github.com>
License: MIT
Keywords: tokens,llm,claude,gemini,codex,opencode,developer-tools
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# token-diet

Audit token footprint and detect silent context leaks in AI agent workflows.

Zero-dependency CLI for **Claude Code**, **Cursor**, **Codex**, **Gemini**, and **OpenCode**.

---

## Why?

AI coding assistants inject instruction files (`AGENTS.md`, `CLAUDE.md`, `.cursorrules`) into the system prompt on **every single turn**. A bloated 25 KB prompt file forces the model to re-process ~6,000 tokens on every message, draining rate limits and breaking prompt cache discounts.

Worse, some local tools install hidden `pre-commit` hooks that send your entire git diff to an LLM on every commit.

`token-diet` scans your repository or workstation in milliseconds to:
- **Audit prompt size:** Flags instruction files that exceed context budgets (`[OK]`, `[WARN]`, `[ALERT]`).
- **Catch rogue git hooks:** Detects and disables silent LLM review hooks in the active hooks directory (`--fix-hooks`), including linked worktrees, submodules, and repositories using `core.hooksPath`.
- **Locate the bloat:** Breaks a Markdown directive down per level-2 heading so you know which section to trim (`--sections`).
- **Audit MCP overhead:** Reports the context cost of every declared MCP server in `opencode.json`, `.mcp.json`, and `mcp.json` (`--mcp`).
- **Guard CI pipelines:** Fails PR builds if prompt expansions exceed token thresholds or regress past a saved baseline (`--strict`, `--max-tokens`, `--baseline`, `--max-delta`).
- **Zero dependencies:** Pure Python standard library (Python >= 3.8). Runs in < 10ms.

---

## Installation

### Via Homebrew (macOS & Linux)
```bash
brew install joseoper/tap/token-diet
# Or:
brew tap joseoper/tap
brew install token-diet
```

### Via `pipx` (Recommended for Isolated CLI)
```bash
pipx install token-diet-cli
```

### Via `pip` / `uv`
```bash
# Using pip:
pip install token-diet-cli

# Using uv:
uv tool install token-diet-cli
```

### Running Directly from Source (No Installation Needed)
```bash
git clone https://github.com/joseoper/token-diet.git
cd token-diet
PYTHONPATH=src python3 -m token_diet
```

---

## Usage

### 1. Instant Health Check in Current Directory
Simply run `token-diet` in any project repository. It automatically scans local directives and checks for dangerous pre-commit hooks:

```bash
token-diet
```

**Example Output:**
```text
================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive                               |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
/Users/developer/my-project/AGENTS.md          |       962 |       240 |     [OK]
/Users/developer/my-project/.cursorrules       |     1,240 |       310 |     [OK]
================================================================================
Total Bytes:     2,202 B
Total Tokens:    ~550 estimated base tokens per turn
Files Detected:  2 of 2
```

### 2. Automated Alerts: Context Bloat & Invisible Pre-Commit Hooks
When a prompt file is oversized or a git hook silently burns tokens on every commit, `token-diet` issues an instant visual alert:

```bash
token-diet
```

**Example Alert Output:**
```text
================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive                               |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
/Users/developer/my-project/AGENTS.md          |    24,500 |     6,125 |  [ALERT]
================================================================================
Total Bytes:     24,500 B
Total Tokens:    ~6,125 estimated base tokens per turn
Files Detected:  1 of 1

🚨 IRREGULAR HOOK DETECTED:
  • [pre-commit] Pre-commit hook invokes AI/LLM review tooling (gga, opencode). This causes invisible token consumption on every commit.
    Path: /Users/developer/my-project/.git/hooks/pre-commit
  💡 Tip: Run 'token-diet --fix-hooks' to automatically neutralize silent AI invocations.

⚠️  CONTEXT BLOAT DETECTED:
  Files marked [ALERT] exceed 16 KB (>4k tokens) and severely penalize context windows.
  Recommendation: Decouple procedures into On-Demand Skills using the LUT Router pattern.
```

### 3. Auto-Fix Silent Pre-Commit Hooks (`--fix-hooks`)
Neutralize unmonitored AI review tooling in `.git/hooks/pre-commit` in a single command with an automatic backup (`.git/hooks/pre-commit.bak`):

```bash
token-diet --fix-hooks
```

**Example Output:**
```text
🛠️  AUTO-REMEDIATION: PRE-COMMIT HOOKS NEUTRALIZED
  • Backup saved: /Users/developer/my-project/.git/hooks/pre-commit.bak
  • Neutralized 1 AI review invocation(s).
  • Safe pre-commit hook is active without silent token bleed.
```

### 4. Audit System-Wide Global Prompts (`--global`)
Scan default assistant instruction files across your entire workstation (`~/.claude/CLAUDE.md`, `~/.codex/AGENTS.md`, `~/.gemini/GEMINI.md`, etc.):

```bash
token-diet --global
```

### 5. Audit Specific Paths or Folders
```bash
# Audit a single file
token-diet AGENTS.md

# Audit entire directories recursively
token-diet docs/
```

### 6. Automated CI / Pre-Commit Gate
Integrate into your CI pipeline or Git pre-commit workflow to prevent prompt regressions:

```bash
# Fail CI if prompt footprint exceeds 2,500 tokens:
token-diet --max-tokens 2500 AGENTS.md

# Fail CI if any file triggers a [WARN] or [ALERT]:
token-diet --strict docs/
```

### 7. Structured Telemetry Output (`--json`)
Emit structured JSON for automated pipelines, threshold assertions, or telemetry dashboards:

```bash
token-diet --json
```

### 8. Locate Prompt Bloat per Heading (`--sections`)
A monolithic directive rarely bloats evenly. This breakdown attributes bytes and estimated tokens to every level-2 (`##`) heading, so you know exactly which section to compress or move into an on-demand skill:

```bash
token-diet --sections AGENTS.md
```

```
SECTION BREAKDOWN: MARKDOWN HEADING FOOTPRINT
================================================================================
AGENTS.md  (23,914 B / ~5,978 tokens)
  Line | Heading                                  |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
     1 | (preamble)                               |       412 |       103 |              [OK]
    12 | Core Directives                          |    19,220 |     4,805 |           [ALERT]
    98 | Additional Protocols                      |     4,282 |     1,070 |            [WARN]
--------------------------------------------------------------------------------
```

Content before the first `##` heading is grouped as `(preamble)`, and deeper headings (`###`) stay attributed to their parent section. Combine with `--json` for machine-readable output.

### 9. Audit MCP Server Overhead (`--mcp`)
Every declared MCP server consumes context, and some configs inline large tool schemas. `token-diet` discovers local `opencode.json`, `.mcp.json`, `mcp.json`, `.cursor/mcp.json`, and `.vscode/mcp.json` files and reports the payload cost per server:

```bash
token-diet --mcp
```

```
MCP SERVER PAYLOAD AUDIT
================================================================================
opencode.json  [opencode]  (2 server(s))
Server                         | Transport | Tools |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
engram                         | local     |    12 |    18,430 |     4,607 |           [ALERT]
web-search                     | remote    |     0 |       142 |        35 |              [OK]
--------------------------------------------------------------------------------
```

Config-declared `tools` blocks are counted and included in each server's payload. Schemas fetched by a server at runtime are not visible to a local audit; the reported figure is the local injection cost floor.

### 10. Fail on Regression vs a Saved Baseline (`--baseline`, `--max-delta`)
Record a baseline once, then gate every pull request against it. This catches slow, incremental growth that a fixed threshold never notices:

```bash
# Save the baseline on your default branch
token-diet --json > token-baseline.json

# Fail if the total grows more than 5% in a pull request
token-diet --baseline token-baseline.json --max-delta 5
```

`--baseline` accepts any `--json` report (`{"summary": {"total_tokens": N}}`) or a minimal `{"total_tokens": N}` file. `--max-delta` exits with code `1` only when tokens **increase** beyond the given percentage; a decrease, an unchanged total, or growth exactly at the budget passes.

---

## Health Status Indicators

| Indicator | Byte Threshold | Estimated Tokens | What it Means & Recommended Action |
|---|:---:|:---:|---|
| 🟢 `[OK]` | `< 4 KB` | `< 1,000 tokens` | **Optimal Context:** Fits cleanly within prompt cache. Preserves fast, cheap inference. |
| 🟡 `[WARN]` | `4 KB – 16 KB` | `1,000 – 4,000 tokens` | **Moderate Size:** Acceptable, but review for duplicate instructions or unneeded verbosity. |
| 🔴 `[ALERT]` | `> 16 KB` | `> 4,000 tokens` | **Severe Prompt Bloat:** Monolithic rules degrade reasoning and waste context budget every turn. Decouple into On-Demand Skills. |
| 🚨 `[HOOK ALERT]` | — | — | **Hidden Token Bleed:** The active `pre-commit` hook sends code to cloud LLMs on every `git commit`. Replace with static linters. |

---

## What To Do When a Leak is Detected (Action Paths)

When `token-diet` flags an irregularity, follow the corresponding action path (detailed runbooks available in the [Remediation Playbook](docs/remediation-playbook.md)):

### Path A: When an Irregular Git Hook is Flagged (`🚨 IRREGULAR HOOK DETECTED`)
* **Problem:** A local hook inside `.git/hooks/pre-commit` secretly sends git diffs to cloud AI tools on every single commit.
* **Action Path:**
  1. Inspect the script: `cat .git/hooks/pre-commit`
  2. Remove or comment out the automated AI review invocation.
  3. Replace it with fast, local deterministic linters (`ruff`, `eslint`, `biome`).
  4. Relocate semantic AI reviews to **Pull Request gates in CI** (see [CI/CD Integration](docs/ci-integration.md)), where reviews are metered and visible.

### Path B: When an Agent Directive is Bloated (`🔴 [ALERT]`)
* **Problem:** A root instruction file (`AGENTS.md`, `CLAUDE.md`, `.cursorrules`) exceeds 16 KB (> 4,000 tokens), exhausting your token budget on every turn and destroying cache discounts.
* **Action Path (The Look-Up Table / LUT Router Pattern):**
  1. **Extract Specialized Protocols:** Move detailed procedure manuals, test specifications, and review arbitrations into standalone on-demand skills (e.g., `skills/<skill-name>/SKILL.md`).
  2. **Compress Root Directive:** Keep only core behavioral invariants and a routing index in the root file (< 3 KB / < 750 tokens):
     ```markdown
     ## On-Demand Skills
     - When running database migrations, read `skills/db-migrate/SKILL.md`.
     - When preparing a release, read `skills/oss-publisher/SKILL.md`.
     ```
  3. Re-run `token-diet` to confirm the status turns 🟢 `[OK]`.

### Path C: When CI Fails on Token Budget Exceeded (`exit code 1`)
* **Problem:** A Pull Request introduced unbudgeted prompt expansion that violated `--max-tokens <N>` or `--strict`.
* **Action Path:**
  1. Run `token-diet --json` to get the machine-readable breakdown of offending files.
  2. Prune duplicate rules or extract text into skills.
  3. Re-run local check before pushing: `token-diet --strict`.

### Agentic Remediation: Prompt for AI Assistants
To have your AI coding assistant (Claude Code, Gemini CLI, Cursor, Windsurf, OpenCode) remediate prompt bloat autonomously without manual edits, copy and paste this prompt:

```text
Run `token-diet` in this repository. If you detect any files marked [ALERT] or [WARN], follow `docs/remediation-playbook.md` to decouple operational procedures into on-demand skills under `skills/<skill-name>/SKILL.md`. Compress the root instruction file to a concise Look-Up Table (LUT Router) under 3 KB (< 750 tokens). If an irregular hook is flagged, run `token-diet --fix-hooks`. Verify that all tests pass and `token-diet` reports [OK].
```

---

## Running Tests

Token Diet includes an automated test suite with deterministic coverage:

```bash
python3 -m unittest discover -s tests -p 'test_*.py'
```

---

## Repository Structure

```
token-diet/
├── .github/
│   └── workflows/
│       └── ci.yml             # Automated multi-version CI test runner
├── docs/
│   ├── remediation-playbook.md # Step-by-step token containment and LUT routing
│   └── ci-integration.md       # CI workflows & pre-commit gatekeeper examples
├── src/
│   └── token_diet/            # Core Python package (Zero external dependencies)
│       ├── __init__.py
│       ├── __main__.py
│       ├── baseline.py        # Baseline comparison & regression budget gate
│       ├── cli.py             # CLI implementation and footprint engine
│       ├── git_hooks.py       # Worktree, gitdir and core.hooksPath resolution
│       ├── mcp_audit.py       # MCP config discovery & payload accounting
│       ├── metrics.py         # Shared byte thresholds and token estimation
│       └── sections.py        # Per-heading Markdown footprint breakdown
├── tests/
│   ├── test_baseline.py       # Baseline budget unit tests
│   ├── test_git_hooks.py      # Hook resolution unit tests
│   ├── test_mcp_audit.py      # MCP payload unit tests
│   ├── test_sections.py       # Section breakdown unit tests
│   └── test_token_audit.py    # Core engine and CLI integration tests
├── .gitignore
├── AGENTS.md                  # Agent-friendly repository directives
├── CONTRIBUTING.md            # Contribution guidelines & engineering principles
├── LICENSE                    # MIT Open Source License
├── pyproject.toml             # PEP 621 packaging and CLI entrypoint
└── README.md                  # Project overview and documentation
```

---

## Core Methodology: The 6 Vectors of Token Leakage

Token Diet is built on empirical analysis of how developer environments silently deplete API quotas:

1. **Global Prompt Bloat:** Monolithic instruction files (`CLAUDE.md`, `AGENTS.md`) exceeding 20 KB inject tens of thousands of static tokens on every turn. *Solution: Keep root directives under 3 KB (<750 tokens) and route to on-demand skills.*
2. **Invisible Git Hooks:** Pre-commit hooks that intercept local commits to invoke cloud LLM reviews drain quotas without developer consent. *Solution: Keep local commits deterministic; reserve AI reviews for PR gates.*
3. **Reasoning Effort Inflation:** Running frontier models at `high` reasoning effort for simple syntax or diagnostic queries generates thousands of hidden chain-of-thought tokens. *Solution: Default to `medium` or `low`.*
4. **Static MCP Schemas:** Registering dozens of MCP tools serializes massive JSON parameter schemas into system prompts. *Solution: Prune unused tool servers and enforce lazy-loading.*
5. **Prompt Cache Invalidation:** Injecting dynamic timestamps or changing session IDs at the start of instructions destroys provider prompt caching discounts (up to 90%). *Solution: Maintain immutable instruction prefixes.*
6. **In-Session Runtime Inefficiencies:** Agents loading entire 1,500-line files to check one function, overwriting whole files instead of focused diffs, or running uncompacted 100k+ token threads. *Solution: Enforce active runtime hygiene (grep-first, line-range reads, targeted diffs, and context compaction thresholds; see [Remediation Playbook Phase 4](docs/remediation-playbook.md#phase-4-in-session-runtime-hygiene-active-execution-guards)).*

---

## Platform Compatibility

Token Diet is OS-agnostic and cross-platform:
- **CLI Diagnostic Tool:** Pure Python standard library with zero third-party dependencies. Runs seamlessly on **macOS**, **Linux**, and **Windows** (Python >= 3.8).
- **Universal Methodology:** Core principles apply across all assistants (Claude Code, Gemini CLI, OpenAI Codex, OpenCode, Cursor, Windsurf).

---

## Documentation & Guides

For in-depth operational procedures and automated pipeline setups:
- **[Remediation Playbook](docs/remediation-playbook.md):** Step-by-step runbook to prune prompt files, configure reasoning effort, and contain background services.
- **[CI/CD Integration](docs/ci-integration.md):** GitHub Actions workflows and pre-commit configurations for automated token budget enforcement.
- **[Agent Directives (AGENTS.md)](AGENTS.md):** Operational guidelines for AI coding assistants working on this codebase.
- **[Contributing Guidelines](CONTRIBUTING.md):** How to run tests, submit pull requests, and coding standards.

---

## License

This project is licensed under the terms of the [MIT License](LICENSE).
