Metadata-Version: 2.4
Name: optitokenize
Version: 0.1.1
Summary: Local-only CLI that prices Claude Code token usage and attributes cost to your repo's modules.
Project-URL: Homepage, https://github.com/aviad-buskila/tokenomics
Project-URL: Repository, https://github.com/aviad-buskila/tokenomics
Author: optitokenize contributors
License: MIT
License-File: LICENSE
License-File: NOTICE
Keywords: claude,claude-code,cli,cost,tokens
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.0
Description-Content-Type: text/markdown

# optitokenize

**Local-only economics for your Claude Code sessions.** `optitokenize` reads the
transcripts Claude Code already writes to disk, prices the recorded token usage,
and attributes that cost to the modules of your current git repository — so you
can see *where* your agent spend went and *what survived* into commits.

- **Local-only.** Zero network calls at runtime. No telemetry, no update checks.
  Everything runs against files already on your machine.
- **Read-only.** Your transcripts and your git repo are never modified.
- **Honest.** Costs are recomputed from recorded token usage against vendored
  list prices — never fabricated, never silently substituted.

> optitokenize is a measurement tool, not a scoreboard. Attribution is a
> heuristic. A module that costs a lot per accepted change is usually a *hard*
> module, not a badly written one — read the numbers as a map of where the work
> was, not a verdict on code quality.

**New here?** [PRODUCT.md](PRODUCT.md) is a non-technical overview of what
optitokenize does and who it's for. This README covers installation, commands,
and exact behavior.

---

## Install

Requires Python ≥ 3.11 and [`uv`](https://docs.astral.sh/uv/).

```console
$ uv sync
$ uv run optitokenize doctor
```

Or build and install the wheel:

```console
$ uv build
$ uv tool install dist/optitokenize-0.1.0-py3-none-any.whl
```

`optitokenize` has exactly two runtime dependencies: `click` and `rich`.

## Quickstart

Run it from inside any git repository you've worked on with Claude Code:

```console
$ optitokenize report --since 30d
```

optitokenize maps the current repo to its Claude Code transcript directory under
`~/.claude/projects/`, parses the sessions, and prints a per-module breakdown.

## What it measures

For each **module** (by default the first two path components under the repo
root), optitokenize reports:

| Column      | Meaning                                                                                     |
| ----------- | ------------------------------------------------------------------------------------------- |
| `SPEND`     | Total attributed cost of turns that touched files in the module.                            |
| `ACCEPTED`  | Number of agent edit events in the module whose content survived into a commit.             |
| `$/DIFF`    | `SPEND ÷ ACCEPTED` — dollars per accepted change. Shown as `—` when nothing was accepted.   |
| `COMP. TAX` | Comprehension tax: how much the agent had to *read* relative to what it *wrote* (see below).|
| `REWORK`    | Fraction of edits in the module that were re-edited in the same file, same session.         |

Below the module table, three **buckets** account for spend that isn't code:
`(conversation)` (turns with no file tool calls), `(bash)` (shell turns), and
`(outside repo)` (paths outside the repo root). Module spend plus bucket spend
always sums to the reported total.

### A note on "Comp. tax"

Conceptually, comprehension tax is the ratio of input tokens the agent consumed
to the output tokens it produced — a module you have to re-read a lot to change
is expensive to comprehend. But *billed* input includes the entire conversation
context, so a module touched late in a long session looks "unreadable"
regardless of its code. optitokenize therefore defaults to a
**module-local, bytes-based** ratio — bytes the agent *read* from the module's
files divided by bytes it *wrote* to them — and offers the literal billed ratio
via `--comp-tax=billed`. See [docs/METRICS.md](docs/METRICS.md) for exact
definitions and caveats.

### A note on cost

The number is a **list-price reconstruction** of the token usage recorded in
your transcripts (cache reads priced at 10% of input, cache writes at their
1.25× / 2× premiums). For API-key users it closely matches the billed amount;
for Pro/Max subscription users it is *not* an invoice — it's what the same usage
would have cost at API list prices. Prices are vendored in
`src/optitokenize/pricing_data.json` with their retrieval date; override with
`--pricing FILE`.

## Example

```text
optitokenize report — /home/you/acme-api
window: 2026-06-24 → 2026-07-23 · 34 sessions · 1204 turns
total spend: $128.40 · cache hit rate: 71%

MODULE            SPEND  ACCEPTED  $/DIFF  COMP. TAX  REWORK
src/api          $52.10        21   $2.48       3.1x     19%
src/db           $22.60         4   $5.65       7.4x     50%
src/parser       $18.90        11   $1.72       2.2x      9%
tests/api        $11.30         8   $1.41       1.4x      0%
(root)            $3.20         2   $1.60       1.1x      0%

(conversation)   $12.10
(bash)            $6.80
(outside repo)    $1.40

warnings:
  src/db costs 3.3x the repo median per accepted change
  src/db comprehension tax is 3.4x the repo median

Attribution is heuristic — compare --attribution=uniform. Cost reflects task difficulty as well as code quality.
```

When a module has zero accepted edits, `$/DIFF` shows `—` — and the module is
still listed. Those are often the interesting ones: spend that never landed.

## Commands

```text
optitokenize report  [--since 30d|YYYY-MM-DD] [--logs PATH] [--attribution bytes|uniform]
                    [--depth N] [--comp-tax readwrite|billed] [--match-threshold F]
                    [--pricing FILE] [--no-git] [--lenient] [--json] [--plain] [--no-prompts]
optitokenize cache | trend | explain <path> | doctor        # inherit the relevant flags
```

- `report` — the per-module economics table above.
- `cache` — modules ranked by full-rate (uncached) input spend, plus sessions
  with a cache-hit rate below 0.3 flagged as offenders.
- `trend` — ISO-week buckets of total spend, accepted diffs, and $/diff.
- `explain <path>` — one line per session that touched the module: date, short
  session id, spend, files touched, accepted/rejected/pending counts, and a
  truncated preview of the first prompt (disable with `--no-prompts`).
- `doctor` — what optitokenize found: transcript directories, session/turn
  counts, Claude Code versions seen versus those it's tested against, parse
  failures, models missing from the pricing table, git availability, and repo
  detection.

### Useful flags

- `--since 30d` or `--since 2026-06-01` — bound the reporting window.
- `--attribution bytes` (default) splits a turn's cost by bytes moved per file;
  `--attribution uniform` splits evenly. Compare the two — if a module's rank
  swings, its attribution is soft.
- `--depth N` — module granularity (path components under the repo root).
- `--no-git` — skip acceptance inference; `ACCEPTED` and `$/DIFF` render as `—`.
- `--lenient` — skip unparseable transcript lines and report a skip count
  instead of failing. Strict is the default.
- `--json` — stable, versioned machine-readable output (`"schema": 1`).
- `--plain` — deterministic plain-text tables (no colour, no box drawing).
- `--no-prompts` — never print any prompt text.

### Exit codes

| Code | Meaning                              |
| ---- | ------------------------------------ |
| `0`  | Success.                             |
| `1`  | Usage error (bad flag value).        |
| `2`  | No transcripts found for this repo.  |
| `3`  | Parse or pricing failure (strict).   |

## Privacy

Transcripts contain your prompts and your source code. optitokenize **never
writes transcript content anywhere.** The only place prompt text is displayed is
`explain`'s first-message preview, truncated to 80 characters and suppressed
entirely by `--no-prompts`. There is no cache, no log file, no upload.

## How it works

1. **Discover** — map the current git root to its transcript directory under
   `~/.claude/projects/` (override with `--logs PATH`).
2. **Parse** — stream each session's JSONL into typed turns and edit events,
   de-duplicating records that resumed sessions rewrite.
3. **Price** — recompute each turn's cost from its recorded token usage against
   vendored list prices. Any `costUSD` in the transcript is ignored.
4. **Attribute** — attach each turn's cost to the file paths its tool calls
   referenced, roll paths up into modules, and set aside conversation/bash/
   outside-repo spend into buckets.
5. **Match** — infer which edit events survived into commits using content-based
   matching (normalized to survive formatter mutations), so squash-merges and
   rebases don't break attribution.
6. **Report** — compute the metrics and render.

Format drift is the main risk: Claude Code's transcript format is not a stable
public contract. optitokenize is strict where it matters — a **corrupt line** or
a malformed **cost-bearing** record (an `assistant` turn missing its `usage`)
fails loudly in strict mode, naming the file, line, and Claude Code version.
Unknown **auxiliary** record types that newer Claude Code versions add (session
titles, housekeeping) are non-cost, so they're skipped and surfaced in a report
note and in `doctor` rather than aborting the run. Dated model ids (e.g.
`claude-haiku-4-5-20251001`) resolve to their base price automatically. The
versions the parser is tested against are listed in
[SUPPORTED.md](SUPPORTED.md). See [docs/FORMAT.md](docs/FORMAT.md) for the
observed format and [docs/DECISIONS.md](docs/DECISIONS.md) for dated deviations.

## Acknowledgements

optitokenize is built with two excellent libraries:

- [**click**](https://github.com/pallets/click) (BSD-3-Clause) — the command-line interface.
- [**rich**](https://github.com/Textualize/rich) (MIT) — the terminal tables and formatting.

Developed with [ruff](https://github.com/astral-sh/ruff),
[mypy](https://github.com/python/mypy), [pytest](https://github.com/pytest-dev/pytest),
and [uv](https://github.com/astral-sh/uv). Every dependency is permissively
licensed; see [NOTICE](NOTICE) for the full list.

## License

MIT. See [LICENSE](LICENSE).

## Trademarks and disclaimer

optitokenize is an independent, community project. It is **not affiliated with,
sponsored by, or endorsed by Anthropic.** "Claude", "Claude Code", and
"Anthropic" are trademarks of Anthropic, used here only descriptively to
identify the tool whose local session transcripts optitokenize reads. Pricing
figures are reconstructed from Anthropic's public list prices (see
[`pricing_data.json`](src/optitokenize/pricing_data.json) for the source and
retrieval date) and are informational, not an invoice.
