Metadata-Version: 2.4
Name: tokki
Version: 0.1.6
Summary: Local AI coding-agent wrapper for token-aware context, log, repository, and privacy checks.
Author: Jean-Pierre Morard
Maintainer: Jean-Pierre Morard
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://github.com/jpmorard/tokki-public
Project-URL: Repository, https://github.com/jpmorard/tokki-public
Project-URL: Issues, https://github.com/jpmorard/tokki-public/issues
Project-URL: Documentation, https://github.com/jpmorard/tokki-public#readme
Project-URL: Funding, https://github.com/sponsors/jpmorard
Keywords: agent-wrapper,ai-coding,claude,cli,codex,context-engineering,developer-tools,local-first,log-digest,mcp,privacy-audit,repository-map,token-savings,tokki
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Rust
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Utilities
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Tokki

<p align="center">
  <img
    src="https://raw.githubusercontent.com/jpmorard/tokki-public/main/docs/assets/tokki-logo.jpg"
    alt="Tokki logo"
    width="720"
  >
</p>

[![PyPI](https://img.shields.io/pypi/v/tokki.svg)](https://pypi.org/project/tokki/)
[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-red.svg)](LICENSE)
[![Sponsor](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-ea4aaa.svg)](https://github.com/sponsors/jpmorard)

Tokki is the local context, trail, and output economy layer for Codex, Claude,
and terminal coding agents. It reduces avoidable context usage before agents
start broad repository reads, long-log ingestion, repeated environment probing,
or stale-repo diagnosis.

<p align="center">
  <img
    src="https://raw.githubusercontent.com/jpmorard/tokki-public/main/docs/assets/tokki-savings-preview.svg"
    alt="Tokki savings report preview"
    width="820"
  >
</p>

**Proof figures from the maintainer checkout (2026-06-05, local metadata only):**

| Signal | Figure | Evidence boundary |
|---|---:|---|
| Current-session demo | 99.0% avoidable context removed | `tokki demo --html`, grade A local ledger |
| Current-session net savings | 4,233,059 tokens kept local | compact context, maps, queries, command digests, wrappers |
| Repeatable benchmark pack | 80,485 net tokens saved | exact `gpt-4o` / `gpt-4.5-preview` tokenizer counts |
| Dirty-worktree context | 80,263 baseline -> 136 Tokki tokens, 590x | source-hidden repo context benchmark |
| Cost projection | $8.218 projected avoided | default demo prices; not billing evidence |
| Privacy guard | 0 strict findings | benchmark bundles passed `tokki privacy-audit --strict` |

```sh
tokki
tokki demo --html
tokki gain report --html
tokki trail view --html
tokki live --html
tokki live --watch
tokki cache-audit --provider openai
```

Run bare `tokki` for the curated start-here view. Use `tokki --help` for the
grouped full reference.

The PyPI package installs the public opt-in wrapper CLI. The `tokki` runtime is
distributed as a native compiled binary and does not publish implementation
source.

## Current Release

Current public package: `tokki 0.1.6`.

On Ubuntu or Debian, install Tokki with `pipx`:

```sh
sudo apt update
sudo apt install -y pipx
python3 -m pipx ensurepath
python3 -m pipx install --force tokki
export PATH="$HOME/.local/bin:$PATH"
tokki --version
```

Do not combine `apt` and `pip` in one command. `apt` installs Debian packages;
Tokki is a PyPI application. If `python3 -m pip install tokki` reports
`externally-managed-environment`, use the `pipx` commands above.

The `0.1.6` release publishes platform wheels for macOS arm64, Linux x86_64,
and Windows x86_64, keeps `tokki gain` summary-only for end users, and keeps
the detailed savings internals private. Earlier `0.0.x`, `0.1.0`, `0.1.1`,
`0.1.2`, `0.1.3`, `0.1.4`, and `0.1.5` uploads are superseded by `0.1.6`.

## What It Does

Tokki provides protected local tools for:

- compact session-start summaries before broad context gathering
- metadata-only opening briefs with `tokki context`
- optional repository updates at agent launch for configured roots
- wrapper-generated runtime config for agent sessions
- local savings evidence with `tokki gain`
- summary-only HTML proof, private by default, with `tokki gain report --html`
- a hidden-output conversation trail with `tokki trail view --html`
- a live agent-session cockpit with `tokki live`, `tokki live --html`, and `tokki live --watch`
- A/B replay proof with `tokki replay-savings`
- post-session habit detection with `tokki habits`
- privacy-safe Codex history learning with `tokki learn codex-history`
- wrapper routing checks for supported agent commands
- focused privacy scans before publishing or sharing repositories
- compact repository maps and symbol-focused code slices
- local read handles that avoid resending full file bodies
- output filter packs for known build/test noise
- prompt-cache boundary diagnostics for provider-side cache reuse
- provider prompt-cache readiness audits with optional local usage reconciliation
- log digests that replace long raw command output
- test-command routing for known local repository shapes

Tokki does not cryptographically stop a user from launching another binary.
Its enforcement model is practical: route supported agent commands through
wrappers, generate launch evidence, and fail `tokki doctor --strict` when
an installed agent command bypasses the wrapper.

## PyPI Install

Install the public CLI on Ubuntu or Debian:

```sh
sudo apt update
sudo apt install -y pipx
python3 -m pipx ensurepath
python3 -m pipx install --force tokki
export PATH="$HOME/.local/bin:$PATH"
tokki doctor
```

Use `pipx` for Linux distributions that mark the system Python as externally
managed. Do not run commands like `sudo apt install python3 -m pip install
tokki`; that asks `apt` to install packages literally named `install` and
`tokki`.

On macOS, a user-site pip install is also supported:

```sh
python3 -m pip config set global.user true
python3 -m pip install --upgrade --force-reinstall tokki
mkdir -p "$HOME/.local/bin"
ln -sf "$(python3 -m site --user-base)/bin/tokki" "$HOME/.local/bin/tokki"
```

PyPI releases are platform wheels that contain the native `tokki` executable.
Tokki does not publish a source distribution or a universal `py3-none-any`
wheel, so installer selection is platform-specific. Pip installs `tokki` or
`tokki.exe` into the environment script directory that normal Python console
tools use.

The wheel does not silently replace any agent command. To make future launches
of an agent command go through Tokki, enable an opt-in PATH shim:

```sh
tokki agent install-wrapper <agent-command>
tokki agent status <agent-command>
```

The installer preserves the original command as `<agent-command>-real`, writes
a Tokki shim as `<agent-command>`, and reports whether the target bin directory
is already before the original agent directory in `PATH`. On Windows the shim
and preserved command use `.cmd` files in Tokki's local app-data bin directory.
Wrapped sessions
expose `TOKKI_ACTIVE=1`, `TOKKI_AGENT`, `TOKKI_SESSION_ID`, and
`TOKKI_RUNTIME_CONFIG`.

Common coding-agent commands work with the generic wrapper, including `aider`,
`opencode`, and `vibe` for Mistral Vibe. If an agent uses a different local
executable name, pass that command name to `tokki agent install-wrapper`.

Codex has a convenience shortcut:

```sh
tokki codex install-wrapper
tokki codex status
```

Remove the shim with:

```sh
tokki agent uninstall-wrapper <agent-command>
```

## Use Tokki on Any Repository

After the one-time wrapper setup, start a fresh terminal or login shell. Tokki
then works wherever the wrapped agent command is launched:

```sh
cd /path/to/repo
<agent-command>
```

Inside the new agent session, verify the wrapper with:

```sh
env | grep TOKKI
```

Expected markers include `TOKKI_ACTIVE=1`, `TOKKI_AGENT`, `TOKKI_SESSION_ID`, and
`TOKKI_RUNTIME_CONFIG`.

The public wheel intentionally excludes `.github`, source-managed `bin`, `docs`,
`tests`, `tools`, protected Rust source, and Python implementation source.

## Private Source Checkout

Maintainers with private repository access can build the local runtime and link
agent wrappers from the source checkout:

```sh
git clone https://github.com/jpmorard/tokki.git
cd tokki
bin/build-protected-rust
bin/install-agent-wrappers
```

Run a compact session summary:

```sh
tokki session-start
tokki-gate test-router --all-changed --dry-run
tokki context
```

Inspect the install:

```sh
tokki doctor
tokki doctor --strict
tokki gain
```

## Automatic Agent Wrappers

Tokki is automatic when an agent command resolves to a wrapper.

The installer:

- installs `tokki`
- keeps `agent-gate` as a backwards-compatible launcher alias
- preserves real agents as `<agent>-real`
- points `codex`, `claude`, `aider`, `opencode`, and `vibe` at Tokki wrappers
  when those agents exist
- refuses recursive wrapper links
- runs `tokki doctor --strict` for installed wrappers
- prints the launch commands

Use a custom install directory when needed:

```sh
bin/install-agent-wrappers --bin-dir "$HOME/bin"
```

Before launching the real agent, wrappers can run:

- optional `brew upgrade` only when `TOKKI_BREW_UPGRADE` is explicitly enabled
- `tokki update-repos --quiet`
- runtime config generation
- non-secret wrapper launch evidence in `wrapper-sessions.jsonl`

`TOKKI_BREW_UPGRADE` defaults to `off`. Set it to `always`, `1`, `true`, `yes`,
or `on` only when you intentionally want a broad Homebrew upgrade before agent
launch.

If update roots are configured, starting a wrapped agent performs a network and
filesystem operation before the real agent starts: Tokki discovers repositories
under those roots and runs `git pull --ff-only --autostash` in each repo that has
an upstream branch. `--ff-only` refuses history rewrites, but an autostash pop
can still leave local conflicts that need manual repair.

## Runtime Config

Wrappers write JSON runtime metadata before delegating to the real agent. This
is automatic when `codex`, `claude`, or another installed agent command resolves
to a Tokki wrapper. Manual commands such as `tokki context`, `tokki gain`, and
`tokki log-digest` remain optional tools inside the session.

Each launched agent gets its own active runtime file. The active file path is
exported as `TOKKI_RUNTIME_CONFIG`:

```text
.../tokki/sessions/<agent>-<tokki-session-id>.json
```

Tokki also updates the platform `runtime.json` as a latest-session snapshot for
commands launched outside an agent session. Runtime JSON includes
`runtime_config`, `latest_runtime_config`, `tokki_session_id`, `agent`, and the
shared `session_log` path.

Inspect it with:

```sh
printf '%s\n' "$TOKKI_RUNTIME_CONFIG"
python3 -m json.tool "$TOKKI_RUNTIME_CONFIG"
```

Parallel launches are isolated. A Codex session and a Claude session can run at
the same time; each keeps its own `TOKKI_RUNTIME_CONFIG`, while both append
metadata-only events to `wrapper-sessions.jsonl` for savings reports.

If `TOKKI_RUNTIME_CONFIG` is set manually before launching a wrapper, Tokki uses
that exact file for compatibility with tests and debugging. If the variable is
only inherited from an already active Tokki session (`TOKKI_ACTIVE=1`), a child
agent launch gets a fresh runtime file instead of reusing the parent session's
file.

Runtime JSON is metadata-only. Do not write prompts, chat content, API keys,
tokens, credentials, or command output into it.

If launch-time repository updates fail, wrappers still launch the real agent and
record `repo_update_status`, `repo_update_exit_code`, and `repo_update_log` so
the failed fast-forward check can be inspected without replaying the launch.
The same metadata is appended to `wrapper-sessions.jsonl` for local savings
reports.

## Opening Context Brief

Use `context` when an agent needs a read-free orientation before deciding what
to inspect:

```sh
tokki context
tokki context --json
tokki context --delta
```

The brief reports repository metadata, dirty-file counts, upstream divergence,
latest wrapper repo-update status, compact scoped `AGENTS.md` policy rules, and
local `gain` buckets. `--task` only marks that a task was supplied; Tokki does
not echo or store the task text.
Each context brief also records a grade `B` savings event comparing measured
local metadata bytes with the compact brief size.

`--delta` is optional and is designed for repeated calls inside one agent
session. The first context call emits the normal full brief. Later calls with
`tokki context --delta` compare the current repository, runtime, policy, and
changed-file limit against the latest same-session context fingerprint. If the
fingerprint is unchanged, Tokki emits a small known-state frame instead of
resending already-known fields such as root, branch, head, dirty counts,
runtime paths, latest wrapper status, policy source fingerprints, warnings, and
suggested commands. This is the same token-saving idea as header compression:
send the stable reference once, then only send the compact proof that it is
still valid. The delta output uses a stable `suppressed_field_group` dictionary
ID and field hash instead of repeating the dictionary on every call. Delta
events record `context_delta_suppressed_tokens` so savings from avoided repeat
parameters can be audited locally.

Tokki treats `AGENTS.md` like repo policy, not prompt filler. `context` walks
the active repo from root to the current directory, fingerprints scoped
`AGENTS.md` files, and emits a small rule digest for high-value instructions
such as push preflights, owner-protected paths, local validation, and
stdout/stderr artifact handling. The full policy text stays on disk as the
artifact; agent-facing output gets path, hash, counts, and compact rule
summaries.

## Configuration

`tokki update-repos` reads platform config from:

- Linux: `$XDG_CONFIG_HOME/tokki/config.toml` or `~/.config/tokki/config.toml`
- macOS: `~/Library/Application Support/tokki/config.toml`
- Windows: `%APPDATA%\tokki\config.toml`

Runtime session metadata follows the same platform conventions: Linux uses
XDG config/state paths, macOS uses `Library/Application Support` and
`Library/Logs`, and Windows uses `%APPDATA%` for runtime config plus
`%LOCALAPPDATA%` for logs and wrapper shims.

Example:

```toml
[update_repos]
roots = ["$HOME/src"]
max_depth = 5
quiet = true
log_dir = "$HOME/.local/state/tokki/logs"
skip_paths = [
    "$HOME/src/large-archived-repo",
]
```

## Start Here

```sh
tokki context
tokki map
tokki query <symbol>
tokki run -- <test-or-build-command>
tokki compact --current --resume-prompt --token-budget 1200
tokki gain report --html
tokki live
tokki live --watch
```

Specialized commands remain available when needed:

```sh
tokki context --delta
tokki demo --html
tokki trail view --html
tokki live --html
tokki replay-savings --task "fix failing test"
tokki habits
tokki habits --codex-history
tokki habits --transcript local-transcript.txt
tokki learn codex-history
tokki doctor --strict
tokki gain
tokki gain report --html
tokki session-start
tokki-gate test-router --all-changed
tokki run -- <test-or-build-command>
tokki filters validate
tokki map
tokki map --delta
tokki query <symbol>
tokki query --handles <symbol>
tokki read <file> --mode signatures
tokki retrieve <handle>
tokki prompt-cache
tokki cache-audit --provider openai
tokki mcp
tokki acp
tokki privacy-audit --strict
tokki log-digest <log-file>
tokki update-repos --dry-run
```

`tokki-gate test-router --all-changed` inspects changed files and routes the
narrow checks that match them. It always runs `git diff --check`; Rust changes
route to `cargo test --manifest-path protected-rust/Cargo.toml`, Python/CLI
changes route to `python3 -m unittest discover -s tests`, and changed
`.codex/skills/*` folders route to the skill-creator validator when available.
Each routed check is executed through `tokki run`, so the agent sees a compact
digest and the full log stays local.

## Command Output Gate

Use `run` for tests, builds, checks, and diffs that can produce large output:

```sh
tokki run -- pytest
tokki run -- cargo test
tokki run -- git diff
tokki run -- npm test
```

Tokki executes the command locally, stores the full stdout/stderr in a private
local log, and emits only an output digest: exit code, result ID, signal lines,
the full-log path, and before/after token estimates. Repeating the same command
against the same repository state emits an `unchanged` proof frame when the
result hash matches, instead of resending the failure or build output. If the
result changes, Tokki emits the new signal lines and records a fresh result ID.

This is the highest-impact token-saving path. Raw test/build output can be
thousands of tokens; `tokki run` keeps the raw artifact on disk and gives the
agent the small decision payload it needs.

Local audit measurements on this repository for commit `adac64a`, using exact
`gpt-4o` token counts:

| Output shape | Raw tokens | Digest tokens | Saved |
| --- | ---: | ---: | ---: |
| 2000 benign lines | 33,055 | 124 | 99.6% |
| Failing run with 1600 noise lines plus errors | 10,386 | 249 | 97.6% |

`tokki run` is the most defensible Tokki saving because the baseline is the
real command output the agent would otherwise read. The digest intentionally
prioritizes signal lines such as `FAILED`, `ERROR`, and command exit metadata.
Adjacent traceback/context lines may be omitted from the digest; use the
reported full-log path when the agent needs more context.

Privacy boundary: `tokki run` reads stdout/stderr and writes the full raw output
to a local artifact. By default, command logs live outside the repository:

- macOS: `~/Library/Logs/tokki/command-runs/`
- Linux: `$XDG_STATE_HOME/tokki/logs/command-runs/` or `~/.local/state/tokki/logs/command-runs/`
- Windows: `%LOCALAPPDATA%\tokki\logs\command-runs\`

Those logs can contain secrets printed by tests or build tools. They are local
and not transmitted by Tokki, but if `TOKKI_LOG_DIR` points inside a repository,
add that path to `.gitignore`. `privacy-audit` scans the path you pass it; scan
command-run logs explicitly before sharing them.

## Conversation Trail

Use `trail` when an agent session has become long and the user needs the
conversation's Ariadne thread without replaying the hidden chat transcript:

```sh
tokki trail
tokki trail --current
tokki trail view --html
tokki trail --json
```

The trail reads Tokki's local `wrapper-sessions.jsonl` metadata ledger and shows
the compact operational spine: wrapped session starts, context briefs, command
runs, log digests, repo maps, repo queries, actual usage reconciliation, token
savings, result IDs, and hashes. It intentionally hides prompts, LLM output,
raw stdout/stderr, raw logs, and file content. If a ledger event contains keys
such as `prompt`, `assistant_output`, `stdout`, or `stderr`, the trail reports
that those payloads were hidden but does not echo their values.

`trail` is the human-facing audit/history surface. The older
`tokki trail --resume-prompt` form remains a compatibility path, but the
canonical agent-facing continuation command is `tokki compact`.

## Compact Handoff

Use `compact` when an agent needs the equivalent of a `/compact` companion: a
bounded resume frame built from Tokki's local metadata ledger.

```sh
tokki compact
tokki compact --current
tokki compact --resume-prompt
tokki compact --current --resume-prompt --token-budget 1200
```

Deterministic compact is the default. It reads Tokki metadata only and can
summarize git state, dirty-file counts, latest validation or command results,
local handles/log IDs, the next recommended Tokki command, and what the next
agent should not reread.

It cannot deterministically know the current objective, open decisions, private
prompt intent, or assistant reasoning because Tokki deliberately does not store
prompts or chat content. Those intent fields are marked unavailable unless the
agent/user supplies them separately.

Optional local-LLM enrichment is explicit and local:

```sh
tokki compact --transcript path/to/transcript.txt --local-llm ollama:llama3
```

The transcript stays local, but the LLM summary is nondeterministic and graded
lower than the deterministic ledger frame. The deterministic compact frame
remains the canonical artifact.

## Live Cockpit

Use `live` when the current agent session needs an operator view instead of a
single command report:

```sh
tokki live
tokki live --html
tokki live --html --watch
tokki live --watch
tokki live --current
tokki live --agent claude
tokki live --agents codex,claude
tokki live --all-agents
tokki live --json
```

The live HUD shows the current trail, a real event breadcrumb, net tokens saved,
prompt-cache readiness, last command digest link, hidden payload policy, the next
cheapest Tokki action, and the MCP Context Cart. By default it also scans the
per-session runtime files and renders one latest pane per agent, so parallel
Codex and Claude sessions appear in the same cockpit. Use `--current` for the
focused single-session view, `--agent claude` or `--agents codex,claude` to
filter panes, and `--all-agents` to show every known session runtime instead of
only the latest runtime per agent.

The breadcrumb is generated from recent metadata events such as wrapper launch,
context, command digests, map/query/read handles, and usage records. It reads the
same local metadata ledger as `trail` and `compact`; prompts, assistant output,
raw command output, raw logs, and file content remain hidden unless the user
explicitly selects a log or read-handle resource.

`--watch` refreshes the terminal cockpit every 5 seconds by default. Use
`--interval SECONDS` to change the cadence. With `--html PATH --watch`, Tokki
rewrites the same HTML file on every refresh; pass `--open` to open it once,
then refresh or reload the browser tab as needed.

## Replay Savings

Use `replay-savings` when you need a local, privacy-checked proof for one
agentic task:

```sh
tokki replay-savings --task "fix failing test"
tokki replay-savings --task "fix failing test" --html
tokki replay-savings --task "fix failing test" --json
tokki replay-savings --task "fix failing test" --provider openai --usage response.json
```

The report compares baseline context versus Tokki-assisted context using the
existing local evidence ledger. It includes:

- token delta: baseline tokens, Tokki output tokens, net saved, savings percent,
  grade, and reconciliation status
- cache delta: cache readiness, stable-prefix size, optional provider usage
  counters, and cache-hit rate
- command-output delta: raw command/log tokens versus digest tokens
- hidden-payload compliance: whether reports hide prompt, assistant, raw log,
  stdout/stderr, and file-content payloads

This is an A/B replay proof over local evidence, not a second autonomous agent
run. Billing-grade claims require provider usage counters imported through
`tokki gain reconcile` or passed with `--usage`. The `--task` value is a
user-supplied proof label, so use a short non-sensitive label for public
reports.

## Agent Habits

Use `habits` after a session to find expensive agent behavior:

```sh
tokki habits
tokki habits --html
tokki habits --json
tokki habits --codex-history
tokki habits --transcript local-transcript.txt
```

Ledger-visible checks are deterministic:

- repeated `tokki read` of the same file, default threshold 5
- repeated context/map/query/log recomputation from the same fingerprint
- broad test commands when Tokki's router suggests a cheaper command
- full file reads or broad non-handle queries where signatures or handles would
  be cheaper
- prompt-cache boundary volatility from current local cache diagnostics

Raw agent behavior is not visible to Tokki unless you provide a local transcript.
Without `--transcript`, checks like raw agent file reads and pasted full logs are
marked `unavailable_without_transcript`, not guessed. With `--transcript`, Tokki
scans the local file for repeated read patterns and long log-like blocks, but it
does not emit transcript text or raw log lines in the report.

`--codex-history` adds a second opt-in signal source: structured local Codex
rollout events from `~/.codex/sessions/**/*.jsonl`. This catches unwrapped
patterns such as large raw command outputs, repeated shell probes, and repeated
`sed`/`nl`/`cat` source peeks. It emits hashes, command families, counts, and
size estimates only; prompts, assistant output, raw commands, raw tool output,
local account names, repo names, and session file names stay hidden behind
aliases and hashes.

## Codex History Learning

Use `learn codex-history` to mine local Codex session history for new Tokki
token-saving rules:

```sh
tokki learn codex-history
tokki learn codex-history --json
tokki learn codex-history --recent 100 --limit 30
tokki learn codex-history --write-rules
```

By default, Tokki scans structured rollout tool events only. It does not read
`~/.codex/history.jsonl` conversation text, and it never emits raw prompts, raw
assistant output, raw commands, raw tool output, local account names, repo names,
or session file names. The report grades the result as B evidence because
savings are estimated from local structured history using Tokki's
chars-per-token estimator, not provider billing counters.

The learner reports:

- missed `tokki run` opportunities for large `exec_command` / `shell_command`
  outputs
- repeated command hashes where prior results should be reused through deltas,
  handles, or digests
- source-peek families such as `sed`, `nl`, and `cat` that should route through
  `tokki read --mode signatures`, `tokki query --handles`, or `tokki retrieve`
- learned local rules that can be written to Tokki's state/log area with
  `--write-rules`

`--include-history-text` is intentionally explicit. When supplied, Tokki reads
local `history.jsonl` text only to count characters/tokens; the raw text is still
not emitted.

## Repository Map

Use `map` when an agent needs to understand a codebase before deciding what to
read. It replaces broad source reads with a compact, ranked symbol map:

```sh
tokki map
tokki map --json
tokki map --token-budget 2000
tokki map --token-model gpt-4o
tokki map --changed
tokki map --changed origin/main
tokki map --delta
```

Use `--changed [REF]` to map only the working set: files that differ from `REF`
(default `HEAD`) plus untracked, non-ignored files. This keeps an incremental
session focused on the files in play instead of the whole repository.

Use `--delta` after a previous map in the same local ledger. Tokki compares the
current file-symbol fingerprints with the latest map snapshot. If the map is
unchanged, it emits only an unchanged proof. If it changed, it emits changed and
removed file blocks instead of the whole map.

Tokki caches parsed repo-map metadata locally so repeated maps avoid reparsing
unchanged source files. In a Git checkout the default cache is
`.git/tokki/repo-map-cache-v1`; outside Git it falls back to the OS user cache
directory. Set `TOKKI_MAP_CACHE=0` to disable this cache, `TOKKI_MAP_CACHE_DIR`
to force the repo-map cache location, or `TOKKI_CACHE_DIR` to move Tokki's
general cache root.

Local audit measurement on this repository for commit `adac64a`, using exact
`gpt-4o` counts: a warm unchanged map went from about 1,414 tokens for the full
map to 118 tokens for the delta proof, about 92% saved. The incidence is lower
than `tokki run`: it only helps when the agent repeats a map after the mapped
source set has not changed.

`map` lists tracked source files (via `git ls-files`, honoring `.gitignore`) and
their top-level symbols (functions, classes, structs, types), ranked so the most
widely-referenced files appear first. The map is budgeted to a token ceiling so
it stays small regardless of repository size.

Each run records a `repo_map` savings event comparing the estimated tokens of
reading the included files in full with the compact map. Without `--token-model`
the savings are grade `B` (measured local bytes); with `--token-model` they are
grade `A` (exact local tokenizer counts). The map is built locally and is never
transmitted; `map` reads source on disk but emits only paths, symbol names, and
signatures.
`tokki gain` reports these events under the `repo_map` token bucket and dedupes
repeat maps with the same baseline fingerprint by default.

Ranking runs PageRank over a cross-file reference graph: a file that mentions a
symbol points to every file that defines it, and each reference is weighted by
the inverse number of definition sites, so names defined in many files (for
example `run`) carry little signal while uniquely-defined symbols carry full
weight. Python, Rust, JavaScript, TypeScript, Go, and Ruby symbol extraction
plus reference tokens use tree-sitter ASTs, which avoids comment/string false
references and handles nested definitions, methods, traits, structs, enums,
interfaces, and type aliases. Languages without a bundled grammar fall back to
Tokki's lightweight heuristic extractor. References are still local lexical
mentions rather than full type-resolved bindings.

## Symbol Query

Use `query` after `map` when the agent knows the symbol it needs and should not
read whole files just to find definitions or nearby references:

```sh
tokki query build_repo_map
tokki query Widget --json
tokki query Widget --handles
tokki query Widget --token-budget 1600 --context-lines 3
tokki query Widget --token-model gpt-4o
```

`query` scans tracked source files locally, ranks definition matches ahead of
references, and emits small line-numbered snippets around the matching lines.
It records a `repo_query` savings event comparing the estimated tokens of the
matched files with the returned snippets; `tokki gain` reports this under the
`repo_query` bucket and dedupes repeat queries by default.

Use `--handles` when the agent needs a compact routing list before choosing
which slices to inspect. Handles include path, line, symbol kind, and snippet
hash, but omit source snippets. Run the same query without `--handles` only for
the handle that needs code context.

Local audit measurement on this repository for commit `adac64a`, using exact
`gpt-4o` counts: a full query response was 628 tokens and the handle response
was 430 tokens, about 31.5% saved. The net win depends on behavior: it helps
when the agent would otherwise inspect snippets for several matches but only
needs to drill into a few.

Privacy boundary: unlike `map`, `query` emits code slices by design. It still
runs locally and does not write prompts, chat content, or command output to the
ledger, but the command output itself can contain source snippets.

## High-Gain Context Optimizers

Tokki also applies the same general ideas as prompt-cache, LeanCTX, and token
filtering tools: keep stable context stable, replace large source reads with
handles, and strip known low-signal output before it reaches the agent. Tokki's
implementation is local, agent-agnostic, and measured through its own ledger; it
is not a clone of any one project.

These commands are optional when used by a human in a shell. In an agent session
they are meant to be triggered by the agent through Tokki's wrapper guidance,
MCP tools, or ACP prompt commands before broad reads.

### Output filter packs

`tokki run` uses built-in filter packs for common build/test tools such as
Cargo, pytest, npm, and Git. Filters never delete the full local log; they only
shape the compact digest the agent sees. Signal lines such as failures, panics,
tracebacks, and fatal errors are kept ahead of known repetitive progress lines.

Custom filters live in `.tokki/filters/*.toml`:

```toml
name = "local-cargo"
command = "cargo"
keep_contains = ["error", "failed", "panic", "thread '"]
drop_contains = ["Compiling ", "Finished ", "Running "]
max_lines = 80
```

Validate or scaffold filters:

```sh
tokki filters validate
tokki filters init
```

### Read and retrieve handles

Use `read` when the agent needs a file orientation but should not ingest the
whole file body:

```sh
tokki read src/app.py --mode signatures
tokki read src/app.py --mode outline
tokki retrieve <handle>
```

`read` stores the full file content in a local handle artifact under Tokki's log
directory and emits only path, content hash, handle, token counts, and symbols.
`--mode outline` omits source signatures and emits signature hashes only.
`--mode signatures` emits compact symbol signatures but not function bodies.
`--mode full` is explicit and sends the full content.

Each read records a `repo_read` savings event. `tokki gain` compares the full
file token count with the compact read output and dedupes repeat reads by path,
mode, and content hash.

### Prompt-cache diagnostics

Use `prompt-cache` when the agent or wrapper needs a provider-neutral cache
boundary plan:

```sh
tokki prompt-cache
tokki prompt-cache --provider openai --json
tokki prompt-cache --provider anthropic --runtime-config "$TOKKI_RUNTIME_CONFIG"
```

The command does not call provider APIs. It emits a stable Tokki policy ID,
suggested cache key, cacheable-prefix guidance, volatile-context guidance, and
warnings such as missing runtime config or dirty worktree state. Put stable
system/tool/policy text before the provider cache boundary and put Tokki
context, command digests, read handles, and changed-file data after it.

Use `cache-audit` when provider usage counters are available locally and the
agent needs to know whether the cache boundary is actually paying off:

```sh
tokki cache-audit --provider openai --usage response.json --json
tokki cache-audit --provider anthropic --model claude-3-5-haiku-latest --usage response.jsonl
```

The audit still does not call provider APIs. It reads stable local metadata
files and an explicitly supplied usage JSON/JSONL file, normalizes OpenAI and
Anthropic cache counters, reports hit rate and minimum cacheable-token
thresholds, and never echoes prompt or assistant payload text from the usage
file.

## MCP Server

`tokki mcp` runs a stdio Model Context Protocol server. Configure an MCP-capable
agent to launch this command from the repository root, then prefer Tokki tools
before broad reads:

```sh
tokki mcp
```

Exposed tools:

- `tokki_context`: metadata-only opening brief for git state and policy rules
- `tokki_compact`: canonical agent-facing handoff/resume frame with optional
  local-LLM transcript enrichment
- `tokki_live`: current-session cockpit with trail, savings, cache readiness,
  last command digest, hidden payload policy, and next cheapest action
- `tokki_replay_savings`: shareable A/B replay proof with token, cache,
  command-output, and hidden-payload deltas
- `tokki_habits`: post-session mistake detector with optional local transcript
  checks and optional Codex-history learning signal
- `tokki_learn_codex_history`: privacy-safe learned-rule report from structured
  local Codex rollout history
- `tokki_map`: ranked repository symbol map with optional `root`,
  `token_budget`, and `max_files`
- `tokki_query`: definition and reference snippets with optional `root`,
  `limit`, and `context_lines`
- `tokki_read`: local file handle plus signatures or outline instead of a full
  file body
- `tokki_retrieve`: explicit full-content retrieval for a prior local read
  handle
- `tokki_run`: command-output gate with a `command` argv array, optional `cwd`,
  `limit`, and `token_model`
- `tokki_log_digest`: compact digest for an existing local log file
- `tokki_filters`: list or validate output filter packs
- `tokki_prompt_cache`: stable-prefix and volatile-context cache guidance
- `tokki_cache_audit`: provider cache-readiness audit with optional local
  usage JSON/JSONL reconciliation
- `tokki_gain`: summary-only local savings evidence

The MCP transport uses standard `Content-Length` framed JSON-RPC over stdio.
Use MCP when Tokki is a tool server for an agent. `tokki_compact` is the
agent-facing resume frame; `tokki_map` keeps emitting metadata; `tokki_query`
emits local code slices by design; `tokki_read` keeps full file content behind a
local handle unless `full` mode or `tokki_retrieve` is explicitly requested.
`tokki_run` executes the requested local command and returns the same compact
digest as the CLI; full stdout/stderr remains in Tokki's private local
command-run log. MCP responses include `_meta.tokki` with the tool name, Tokki
argv, success flag, exit code, and protocol run policy when applicable.

MCP also exposes a Context Cart through `resources/list` and `resources/read`.
Cart resources use `tokki://cart/...` URIs:

- `tokki://cart/live/current`: current live HUD JSON
- `tokki://cart/compact/current`: current compact handoff JSON
- `tokki://cart/trail/current`: metadata-only current trail
- `tokki://cart/map/current`: current repository symbol map
- `tokki://cart/prompt-cache/current`: cache-readiness report
- `tokki://cart/cache-audit/current`: provider cache audit report
- `tokki://cart/replay-savings/current`: current replay-savings proof
- `tokki://cart/habits/current`: current agent habit detector report
- `tokki://cart/learn/codex-history/current`: current Codex-history learning
  report
- `tokki://cart/trail-entry/<n>`: one selected trail event
- `tokki://cart/log/<id>`: compact digest for a known local log
- `tokki://cart/handle/<id>`: explicit local read-handle content

Tool calls return MCP `resource_link` items for relevant cart resources, so an
agent can select only the exact map, handle, digest, or trail entry it needs
instead of rereading broad source or raw terminal output.

## ACP Terminal Bridge

`tokki acp` runs a newline-delimited JSON-RPC Agent Client Protocol server for
terminal-oriented coding clients. It is not a terminal UI and does not replace a
full coding agent. It is a Tokki bridge that lets an ACP client create a session
and ask Tokki for compact, local-first terminal/context operations:

```sh
tokki acp
```

Supported prompt commands:

```text
tokki context
tokki context --delta
tokki trail
tokki trail --current
tokki compact
tokki compact --current
tokki compact --resume-prompt
tokki live
tokki live --html
tokki live --watch
tokki replay-savings [--task "short task label"]
tokki habits
tokki habits --html
tokki learn codex-history
tokki gain
tokki log-digest <log-file>
tokki filters validate
tokki prompt-cache
tokki cache-audit [--provider openai|anthropic|generic]
tokki read <file> [--mode signatures|outline|full]
tokki retrieve <handle>
tokki run -- <command> [args...]
```

The bridge responds with ACP `session/update` message chunks and tool-call
status updates. `tokki run` keeps full stdout/stderr in Tokki's private local
command-run log and returns only the compact digest to the ACP client. The
bridge advertises `sessionCapabilities.close`, so ACP clients can release
in-memory Tokki bridge sessions without terminating the process.

ACP `tokki run` uses shell-like parsing for quotes and backslash escapes, but it
does not run through a shell. The parsed argv is passed directly to Tokki's
command-output gate.

Protocol-triggered command execution is guarded by
`TOKKI_PROTOCOL_RUN_POLICY`:

- `safe` (default): allow common inspect/test/build commands such as
  `git status`, `git diff`, `cargo test`, `cargo check`, `pytest`,
  `python3 -m pytest`, `uv run pytest`, `npm test`, and selected package
  `run` scripts (`test`, `build`, `lint`, `check`, `typecheck`)
- `deny`: reject all protocol-triggered `tokki_run` calls
- `allow`: allow any command from trusted MCP/ACP clients

The safe policy rejects shell entry points, destructive commands, Git config
injection (`-c` / `--config-env`), Git pager escape options, and Git external
diff/textconv execution options. Git commands run with a hardened environment
that disables inherited fsmonitor, pager, and external diff hooks. Direct CLI
usage of `tokki run -- <command>` is unchanged.

Safe mode is a guardrail for a trusted local repository and trusted ACP/MCP
client prompts, not a sandbox. Allowed build and test commands still execute
repo-defined code such as Cargo build scripts/tests, Python pytest/conftest
hooks, and package-manager scripts. Use `deny` for untrusted protocol clients,
and use an OS/container sandbox for untrusted repositories.

Use ACP when Tokki is a terminal-agent bridge for ACP clients. Use MCP when
Tokki is an agent-callable tool server. Both surfaces call the same Tokki
primitives: context, compact handoff, gain, log digest, filter packs, read
handles, prompt-cache diagnostics, and command-output gating.

ACP uses newline-delimited JSON-RPC over stdio, so `tokki acp` is separate from
`tokki mcp`, which uses MCP `Content-Length` frames.

`privacy-audit` is a focused pre-publish guard, not a general DLP scanner. It
detects secret-shaped GitHub, OpenAI, Anthropic, AWS, PyPI, and PEM/private-key
material plus local user paths. It does not claim generic entropy scanning or
coverage for every vendor token family.
The `--limit` value must be greater than zero; symlinks are skipped so scans stay
inside the requested tree and avoid symlink cycles.

## HTML Proof Reports

Tokki now has three visible local report surfaces:

```sh
tokki demo --html
tokki gain report --html
tokki trail view --html
```

Default HTML reports are private local artifacts. Without an explicit path,
Tokki writes them outside the active repository under the OS user state area
such as `~/Library/Application Support/tokki/reports`,
`%LOCALAPPDATA%\tokki\reports`, or `$XDG_STATE_HOME/tokki/reports`
(`~/.local/state/tokki/reports` when `XDG_STATE_HOME` is unset). Set
`TOKKI_REPORT_DIR` to choose a different private default. Use `--html PATH` or
`--output PATH` only for intentional exports, then run `tokki privacy-audit
--strict PATH` before publishing or sharing the artifact.

`tokki demo --html` creates a first-run product demo with before/after token
math, a baseline-vs-Tokki inline SVG, an explicitly labeled cost projection,
and a short "what this measured" note. `tokki gain report --html` turns the
local scorecard into a summary-only proof with headline savings, compression,
grade, optional pricing, and a clear privacy boundary. It intentionally omits
per-event scorecards, bucket breakdowns, source paths, runtime paths,
session-log paths, log handles, and command details.
`tokki trail view --html` shows the conversation thread while keeping prompts,
LLM output, raw command output, raw logs, and file content hidden.

## Protected Rust Binary

For source protection, Tokki also has a private Rust implementation path. The
Rust binary is meant for distribution as a compiled artifact while the source
stays in this private repository.

Build it when Rust is installed:

```sh
bin/build-protected-rust
tokki session-start
tokki log-digest <log-file>
tokki privacy-audit --strict --no-github
```

`tokki` requires a compiled protected binary. It searches `dist-protected/tokki`,
`protected-rust/target/release/tokki`, `dist-protected/tokki-protected`,
`protected-rust/target/release/tokki-protected`, or `tokki-protected` on `PATH`.
Set `TOKKI_PROTECTED_BIN` to override the binary path:

```sh
TOKKI_PROTECTED_BIN=/path/to/tokki-protected tokki session-start
```

The protected binary does not make reverse engineering impossible, but it avoids
distributing readable Python source for the protected runtime.

### Redistribution Deterrence

Tokki's practical protection model is private source plus published signed
native builds, not hidden binaries. Watermarking can still add value by carrying
an optional signed customer or license identifier into local logs, benchmark
bundles, and evidence reports. That discourages redistribution because leaked
artifacts remain attributable, while normal local operation stays offline-friendly
and supportable.

### Signed License Watermarking

Commercial/protected builds can enable attribution with an Ed25519-signed
license. The verifier key is public and can be compiled into the protected
binary with `TOKKI_LICENSE_PUBLIC_KEY_HEX`, or supplied at runtime through the
same environment variable for a pilot. No signing secret is stored in Tokki.

```sh
tokki license status
tokki license status --json
tokki license activate path/to/license.json
tokki license activate path/to/license.json --license-dir ~/.config/tokki
```

Tokki reads the license from `TOKKI_LICENSE_FILE`, or
`TOKKI_LICENSE_DIR/license.json`, or the platform Tokki config directory:

- macOS: `~/Library/Application Support/tokki/license.json`
- Linux: `${XDG_CONFIG_HOME:-~/.config}/tokki/license.json`
- Windows: `%APPDATA%\tokki\license.json`

License payloads are signed over the stable fields, including `expires_at`. For
the current pilot window, use end of June:

```json
{
  "schema_version": 1,
  "signature_algorithm": "ed25519",
  "license_id": "lic_2026_001",
  "customer_id": "cust_acme",
  "customer_label": "ACME",
  "issued_at": "2026-06-05T00:00:00Z",
  "expires_at": "2026-06-30T23:59:59Z",
  "features": ["watermark", "protected-runtime"],
  "signature": "ed25519:<hex signature>"
}
```

`tokki license status` reports `active`, `expired`, `missing`, or `invalid`.
Watermarks are emitted only for active, signature-valid licenses. Expired
licenses stay visible as `license_status: expired`, but Tokki does not emit an
active watermark. Evidence surfaces include only non-secret fields:
`license_id`, `customer_label`, `customer_hash`, `license_status`, `expires_at`,
and the active `watermark`. They do not include customer emails, personal names,
or the raw `customer_id`.

## Measuring Savings

Tokki reports local savings evidence:

```sh
tokki gain
tokki gain --json
tokki gain --dedupe-window session
tokki gain --price-input-per-1m 2.00 --price-output-per-1m 8.00
tokki gain reconcile --provider openai --model gpt-5-codex --input-tokens 120000 --output-tokens 18000
tokki gain benchmark
tokki gain
```

`gain` reads Tokki's local metadata ledger and emits a summary-only proof:
net tokens saved, compression ratio, evidence grade, reconciliation status, and
optional cost projection. Public `gain` output intentionally omits per-event
scorecards, bucket breakdowns, source paths, runtime paths, session-log paths,
log handles, and command details. It does not read prompts. Command output is
only read when the user explicitly runs a command through `tokki run`, and the
full output is stored as a private local artifact rather than copied into the
agent-facing digest. Treat the number as a local trend signal, not billing
evidence.

Cost projection is opt-in and uses user-supplied prices only:

```sh
tokki gain --price-input-per-1m 2.00 --price-output-per-1m 8.00 --json
```

Tokki does not hardcode model pricing.

Provider-priced benchmark reports are benchmark-purpose projections unless they
are reconciled with provider usage counters. It is fine to apply an external
price table to Tokki's exact local tokenizer counts to estimate economics, but
publish that as benchmark-grade evidence, not billing-grade evidence. For
example:

```text
98.69% benchmark token reduction, priced against Claude Opus 4.8 as a cost projection.
```

Do not claim exact Claude/Anthropic token savings from a local `gpt-4o` or
`gpt-4.5-preview` tokenizer run. Exact provider billing evidence requires the
provider's own token counters or a reconciled usage report.

By default, `gain` dedupes repeated `context_brief`, `log_digest`,
`command_run`, `repo_map`, `repo_query`, and `repo_read` events inside the current session
using local hashes:

```sh
tokki gain --dedupe-window session
tokki gain --dedupe-window day
tokki gain --dedupe-window none
```

New context events carry `baseline_hash`, `brief_hash`,
`context_fingerprint_hash`, `context_delta_suppressed_tokens` when applicable,
and `dedupe_key`; log-digest, map, query, and read events carry local
baseline/output fingerprints and token counts. `repo_read` events use path,
mode, and content hash, not file content. `command_run` events carry a command
key, result hash, result ID, exit code, raw/digest token counts, and the private
full-log path. Context dedupe uses the stable fingerprint of repo state, runtime
metadata, changed-file limit, and policy hashes, so rerunning the same opening
brief does not double-count savings just because the ledger counters changed.
The hashes are derived from local metadata, fingerprints, and token counts, not
prompt text.

Actual usage reconciliation imports provider usage counters into the same
metadata ledger:

```sh
tokki gain reconcile \
  --provider openai \
  --model gpt-5-codex \
  --input-tokens 120000 \
  --output-tokens 18000 \
  --cached-input-tokens 40000 \
  --cost-usd 0.82 \
  --baseline-input-tokens 160000 \
  --baseline-cost-usd 1.10
```

`gain` then reports summary reconciliation status. Baseline usage and baseline
cost are optional; when they are absent, Tokki still records actual usage
counters locally but leaves estimation error out of the public summary.

Evidence grades keep the score honest:

- `A`: exact before/after token counts
- `B`: measured local bytes or lines with a deterministic estimator
- `C`: metadata events
- `D`: heuristic wrapper-launch estimate

`tokki log-digest <log-file>` records a grade `B` metadata event with raw
line/character counts, digest line/character counts, estimated raw tokens,
estimated digest tokens, and estimated tokens saved. The event does not store
the log content.

`tokki run -- <command>` records a grade `B` `command_run` event with raw
stdout/stderr token estimates, digest token estimates, result hashes, and the
private full-log path. Repeated unchanged command results are collapsed to a
small proof frame.

For exact local tokenizer counts, pass a tokenizer model when generating new
events:

```sh
tokki context --token-model gpt-4o
tokki log-digest path/to/failure.log --token-model gpt-4o
tokki run --token-model gpt-4o -- pytest
tokki read src/module.py --mode signatures --token-model gpt-4o
```

Exact events are recorded as grade `A` with exact before/after token counts.
Without `--token-model`, Tokki keeps the deterministic local estimator.

Context-brief savings include avoided raw policy payload when scoped
`AGENTS.md` files exist. Tokki estimates the full policy text as baseline input
and the compact policy digest as the agent-facing brief.

`tokki gain benchmark` reports a counterfactual benchmark pack for repeatable
scenarios: scoped `AGENTS.md` policy, dirty-worktree opening context, the latest
repo-update failure log when present, and an explicitly supplied cluster/install
failure log:

```sh
tokki gain benchmark --json
tokki gain benchmark --log-file path/to/cluster-install.log --json
tokki gain benchmark --token-model gpt-4o --json
```

Unavailable scenarios are marked as unavailable instead of inventing savings.

### Shareable benchmark bundle

To produce a reproducible, privacy-checked result that is safe to publish, use:

```sh
bin/tokki-benchmark-report                 # exact gpt-4o counts (grade A)
bin/tokki-benchmark-report --estimated     # byte estimate (grade B)
bin/tokki-benchmark-report --out "$HOME/.local/state/tokki/reports/benchmark-2026-06-04" --log-file build.log
```

It writes `result.json` (machine-readable), `provenance.txt` (tool version,
commit, dirty-source marker, platform, command — so anyone can reproduce), and `SUMMARY.md` (the
scenario table). Local user paths are scrubbed and the bundle must pass
`tokki privacy-audit --strict` before it is reported as ready to publish. Share
bundles through the public `tokki-public` repository, not the private source
repository.

When a bundle includes provider pricing, label it explicitly:

```text
Benchmark-grade: yes.
Billing-grade: no.
Provider price: projection only unless reconciled with provider counters.
```

For external A/B comparisons, measure Tokki with repeatable tasks.

```text
token_savings_% = ((baseline_tokens - tokki_tokens) / baseline_tokens) * 100
```

Track input tokens first. Tokki mainly saves context by avoiding broad file
reads, repeated shell probes, and full-log ingestion. If exact token counts are
unavailable, estimate with:

```text
estimated_tokens ~= characters / 4
```

Example:

```text
baseline_tokens = 80000
tokki_tokens = 52000
token_savings_% = ((80000 - 52000) / 80000) * 100 = 35%
```

## Sponsoring

The repository includes `.github/FUNDING.yml` for GitHub Sponsors. GitHub shows
the native Sponsor button only after the configured account has GitHub Sponsors
enabled.

## License

Tokki source tooling is proprietary and all rights are reserved. See `LICENSE`.

Public-facing package and project information lives at
`https://github.com/jpmorard/tokki-public`; the implementation source remains
private.

End-user feedback belongs in that public repository: stars, issues,
installation questions, wrapper behavior reports, and supported-agent requests.
PyPI provides downloads and package metadata, but not a project rating system.
The current review of agent-skill guidance that should become executable Tokki
checks is published at
`https://github.com/jpmorard/tokki-public/blob/main/docs/skill-migration-candidates.md`.
