Metadata-Version: 2.4
Name: tokenjuice-py
Version: 0.7.4
Summary: Python port of tokenjuice — lean output compaction for terminal-heavy AI agent workflows.
Author: tokenjuice-py contributors
License: MIT
Project-URL: Homepage, https://github.com/vincentkoc/tokenjuice
Project-URL: Bug Tracker, https://github.com/vincentkoc/tokenjuice/issues
Keywords: ai,agent,compaction,cli,llm,token,output-reduction
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Utilities
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# tokenjuice-py

**Lean output compaction for terminal-heavy AI agent workflows.**

A Python port of [tokenjuice](https://github.com/vincentkoc/tokenjuice) by [Vincent Koc](https://github.com/vincentkoc).

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![Original: tokenjuice](https://img.shields.io/badge/original-tokenjuice-orange)](https://github.com/vincentkoc/tokenjuice)

---

## What is tokenjuice-py?

`tokenjuice-py` is a deterministic output compactor for terminal-heavy agent workflows. When AI agents run noisy commands like `git status`, `docker build`, `pytest`, or `kubectl get pods`, tokenjuice keeps the command semantics untouched, observes the output, and returns a smaller payload driven by JSON rule-based reducers — instead of dumping entire walls of terminal text into context.

**Key benefits:**
- Less transcript waste
- Fewer useless re-runs caused by context overflow
- Cleaner handoff between tools

Raw output stays available via `--raw`. Rules stay inspectable JSON rather than LLM heuristics.

---

## Features

- **130 built-in JSON rules** covering git, docker, kubectl, pytest, npm, terraform, and more
- **Deterministic compaction** — never changes command semantics, only removes low-signal noise
- **Zero production dependencies** — pure Python standard library (3.10+)
- **CLI tools** — `reduce`, `wrap`, `reduce-json` subcommands
- **Extensible** — user-level (`~/.config/tokenjuice/rules`) and project-level (`.tokenjuice/rules`) custom rules

---

## Installation

```bash
pip install tokenjuice-py
```

> **Import name:** `tokenjuice`  
> **CLI command:** `tokenjuice`

---

## Quick Start

### As a library

```python
from tokenjuice import reduce_execution, ToolExecutionInput

# tokenjuice receives already-captured command output (stdout).
# It does NOT execute commands itself.
raw_stdout = (
    "On branch main\n"
    "Your branch is up to date with 'origin/main'.\n"
    "\n"
    "Changes not staged for commit:\n"
    '  (use "git add <file>..." to update what will be committed)\n'
    "        modified:   src/foo.py\n"
    "        modified:   src/bar.py\n"
    "\n"
    "Untracked files:\n"
    '  (use "git add <file>..." to include in what will be committed)\n'
    "        tests/test_new.py\n"
    "\n"
    "no changes added to commit (use \"git add\" and/or \"git commit -a\")\n"
)

result = reduce_execution(
    ToolExecutionInput(
        toolName="exec",
        command="git status",
        argv=["git", "status"],
        stdout=raw_stdout,
    )
)
print(result.inlineText)
# => "Changes not staged:\nM: src/foo.py\nM: src/bar.py\n\nUntracked files:\n?? tests/test_new.py"
print(result.stats.ratio)   # reduced chars / original chars
# => 0.21

# Clean working tree
result_clean = reduce_execution(
    ToolExecutionInput(
        toolName="exec",
        command="git status",
        argv=["git", "status"],
        stdout="On branch main\nnothing to commit, working tree clean\n",
    )
)
print(result_clean.inlineText)
# => "working tree clean"
print(result_clean.stats.ratio)
# => 0.34
```

### CLI usage

> **Note:** `tokenjuice reduce` reads from **stdin or a file** and uses `--command` only to select the
> matching reduction rule — it does **not** execute the command. Pipe the real output in:

```bash
# Pipe existing output through the reducer
git status | tokenjuice reduce --command "git status"

# One-step: wrap executes the command and compacts output automatically
tokenjuice wrap -- git status
tokenjuice wrap -- docker ps
tokenjuice wrap -- pytest tests/

# Read from a file
tokenjuice reduce build.log --command "make build"

# Output as JSON (includes classification and stats)
git status | tokenjuice reduce --command "git status" --format json

# Specify exit code for failure-aware rules
tokenjuice reduce --command "make build" --exit-code 1 < build.log

# reduce-json protocol (for AI client hooks)
echo '{"input": {"toolName": "exec", "argv": ["git", "status"], "stdout": "..."}}' \
  | tokenjuice reduce-json

# List all built-in rules
tokenjuice discover

# Verify all rules
tokenjuice verify
```

---

## Rule System

### Rule fields

| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Unique identifier (e.g. `git/status`) |
| `family` | string | Rule family (e.g. `git-status`) |
| `match` | object | Match conditions (argv0, gitSubcommands, etc.) |
| `filters` | object | skipPatterns / keepPatterns |
| `transforms` | object | stripAnsi, dedupeAdjacent, etc. |
| `summarize` | object | head / tail line counts |
| `counters` | array | Count facts (e.g. number of modified files) |
| `failure` | object | Special handling on non-zero exit code |

### Custom rules

- User-level: `~/.config/tokenjuice/rules/`
- Project-level: `.tokenjuice/rules/` (project root)

Priority: `project > user > builtin` (later layers override by rule id).

### Built-in rules (130 rules)

| Category | Rule IDs |
|----------|----------|
| **git** | `git/status` `git/diff` `git/diff-stat` `git/diff-name-only` `git/log-oneline` `git/branch` `git/show` `git/stash-list` `git/remote-v` `git/worktree-list` |
| **build** | `build/cmake` `build/dotnet` `build/esbuild` `build/go-build` `build/gradle` `build/maven` `build/msbuild` `build/pnpm-build` `build/swift-build` `build/tsc` `build/tsdown` `build/vite` `build/webpack` `build/xcodebuild` |
| **devops** | `devops/docker-build` `devops/docker-compose` `devops/docker-images` `devops/docker-logs` `devops/docker-ps` `devops/helm` `devops/kubectl-describe` `devops/kubectl-get` `devops/kubectl-logs` `devops/pulumi` `devops/terraform` `devops/terragrunt` |
| **tests** | `tests/jest` `tests/vitest` `tests/pytest` `tests/cargo-test` `tests/go-test` `tests/playwright` `tests/mocha` `tests/rspec` `tests/bun-test` `tests/swift-test` `tests/npm-test` `tests/pnpm-test` `tests/yarn-test` |
| **cloud** | `cloud/aws` `cloud/az` `cloud/gcloud` `cloud/gh` `cloud/flyctl` `cloud/vercel` |
| **network** | `network/curl` `network/wget` `network/ping` `network/ssh` `network/dig` `network/nslookup` `network/traceroute` |
| **install** | `install/npm-install` `install/npm-ci` `install/pnpm-install` `install/yarn-install` `install/bun-install` |
| **package** | `package/apt-install` `package/apt-upgrade` `package/brew-install` `package/brew-upgrade` `package/yum-install` `package/dnf-install` `package/composer` `package/fnm` `package/npm-ls` |
| **database** | `database/psql` `database/mysql` `database/sqlite3` `database/redis-cli` `database/mongosh` |
| **filesystem** | `filesystem/ls` `filesystem/find` `filesystem/fd` `filesystem/rg-files` `filesystem/git-ls-files` |
| **service** | `service/systemctl-status` `service/journalctl` `service/pm2` `service/lsof` `service/netstat` `service/ss` `service/service` `service/launchctl` |
| **observability** | `observability/top` `observability/htop` `observability/free` `observability/iostat` `observability/vmstat` |
| **search** | `search/rg` `search/grep` `search/git-grep` |
| **lint** | `lint/eslint` `lint/biome` `lint/oxlint` `lint/prettier-check` |
| **task** | `task/make` `task/just` `task/uv` `task/node` `task/python` `task/ruby` `task/php` `task/mise` `task/env` |
| **system** | `system/df` `system/du` `system/ps` `system/file` |
| **transfer** | `transfer/rsync` `transfer/scp` |
| **archive** | `archive/tar` `archive/zip` `archive/unzip` |
| **media** | `media/ffmpeg` `media/mediainfo` |
| **text** | `text/wc` |
| **generic** | `generic/fallback` (catch-all) `generic/help` |

> Run `tokenjuice discover` to list all rules loaded at runtime.

---

## Acknowledgements

This project is a Python port of **[tokenjuice](https://github.com/vincentkoc/tokenjuice)**,
the original TypeScript implementation by [Vincent Koc](https://github.com/vincentkoc).

The built-in JSON rule set, core compaction concepts, and CLI design are derived from the
original project. All credit for the original design goes to Vincent Koc and the tokenjuice
contributors.

If you use Node.js / TypeScript or need host integrations (Cursor, Claude Code, Codex, etc.),
please use the original project: https://github.com/vincentkoc/tokenjuice

---

## License

MIT — see [LICENSE](LICENSE) for details.

This project includes work derived from [tokenjuice](https://github.com/vincentkoc/tokenjuice),
Copyright (c) 2026 Vincent Koc, also under the MIT License.
