Metadata-Version: 2.4
Name: tokenjam
Version: 0.5.3
Summary: TokenJam — local-first OTel-native observability for Autonomous AI agents
Project-URL: Homepage, https://opencla.watch
Project-URL: Repository, https://github.com/Metabuilder-Labs/openclawwatch
Project-URL: Issues, https://github.com/Metabuilder-Labs/openclawwatch/issues
Author-email: Anil Murty <anil@metabldr.com>
License: MIT
License-File: LICENSE
Keywords: agents,ai,llm,observability,opentelemetry
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: apscheduler>=3.10
Requires-Dist: click>=8.1
Requires-Dist: duckdb>=0.10
Requires-Dist: fastapi>=0.110
Requires-Dist: fastmcp>=0.2
Requires-Dist: genson>=1.2
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.0
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.25
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.25
Requires-Dist: opentelemetry-exporter-prometheus>=0.46b0
Requires-Dist: opentelemetry-sdk>=1.25
Requires-Dist: pytz>=2024.1
Requires-Dist: rich>=13.0
Requires-Dist: tomli-w>=1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: uvicorn>=0.27
Requires-Dist: websockets>=12.0
Provides-Extra: autogen
Requires-Dist: pyautogen>=0.2; extra == 'autogen'
Provides-Extra: bloat
Requires-Dist: llmlingua>=0.2; extra == 'bloat'
Provides-Extra: crewai
Requires-Dist: crewai>=0.28; extra == 'crewai'
Provides-Extra: dev
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: langchain
Requires-Dist: langchain>=0.2; extra == 'langchain'
Provides-Extra: litellm
Requires-Dist: litellm>=1.40; extra == 'litellm'
Provides-Extra: mcp
Description-Content-Type: text/markdown

<div align="center">

<img src="docs/brand/tokenjam-banner.png" alt="TokenJam" width="440">

### Token Efficiency For AI Agents

TokenJam reads your agent's telemetry and tells you when to downsize, when to trim prompts, what to cache, what to script, and what plans you've already paid to figure out — then shows it all in a local browser dashboard. Runs entirely on your machine.

[![CI](https://github.com/Metabuilder-Labs/tokenjam/actions/workflows/ci.yml/badge.svg)](https://github.com/Metabuilder-Labs/tokenjam/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/tokenjam?color=3d8eff&labelColor=0d1117)](https://pypi.org/project/tokenjam/)
[![Downloads](https://img.shields.io/pypi/dm/tokenjam?color=3d8eff&labelColor=0d1117&label=downloads)](https://pypi.org/project/tokenjam/)
[![Python](https://img.shields.io/badge/python-3.10%2B-3d8eff?labelColor=0d1117)](https://pypi.org/project/tokenjam/)
[![npm](https://img.shields.io/npm/v/@tokenjam/sdk?color=3d8eff&labelColor=0d1117)](https://www.npmjs.com/package/@tokenjam/sdk)
[![License: MIT](https://img.shields.io/badge/license-MIT-3d8eff?labelColor=0d1117)](LICENSE)
[![OTel](https://img.shields.io/badge/OTel-GenAI%20SemConv-3d8eff?labelColor=0d1117)](https://opentelemetry.io/docs/specs/semconv/gen-ai/)

```bash
pipx install tokenjam && tj onboard
```

<sub>The full local tool: captures your agent telemetry, runs the five analyzers, and serves the Lens dashboard + MCP server for Claude Code. Runs entirely on your machine. (<code>pipx</code> recommended — sidesteps PEP 668; <code>pip install tokenjam</code> works in a clean venv.) Just want a 15-second peek with no install? <code>uvx --from tokenjam tj</code> or <code>npx tokenjam</code> reads your existing Claude&nbsp;Code sessions and shows where your quota goes.</sub>

**No cloud · No signup · No vendor lock-in**

<sub>⭐ If TokenJam saves you tokens, **star it** · 👁 **Watch for releases** — we ship often</sub>

</div>

---

## 15-second peek — no install

Not ready to install? Get the headline in one command:

```bash
uvx --from tokenjam tj               # or:  npx tokenjam
```

TokenJam ingests your existing Claude Code sessions from
`~/.claude/projects/*.jsonl` into a throwaway in-memory database and prints:

- **Quota composition** — what share of your tokens went to *re-reading context*
  (history, CLAUDE.md, accumulated tool output) versus *net-new work*.
- **A session timeline** — your most recent sessions, token spend, and re-read share.

Nothing is written to disk, no daemon runs, no config is created. For live capture, the Lens dashboard, and the MCP server, use the full install above (`pipx install tokenjam && tj onboard`).

<sub>`uvx --from tokenjam tj` and `npx tokenjam` launch the Python CLI via `uvx`/`pipx` — see [docs/installation.md](docs/installation.md) for the runner requirements and the full install matrix.</sub>

---

## Five Analyzers + Lens. One Install.

TokenJam reads telemetry from every major agent runtime, framework, provider, and observability tool and surfaces savings across five areas — then brings them together in a local browser dashboard.

<table>
<tr>
<td width="50%" valign="top">

### 🪶 Downsize

Flags sessions where a cheaper model in the same family is worth a look. Never claims quality equivalence — surfaces examples so you can spot-check.

<pre><code>tj optimize downsize</code></pre>

[Details →](docs/optimize/downsize.md)

</td>
<td width="50%" valign="top">

### 💾 Cache

Shows your current caching ratio per (provider, model) and suggests Anthropic prompt-cache breakpoints from stable prefixes in your real usage.

<pre><code>tj optimize cache</code></pre>

[Details →](docs/optimize/cache.md)

</td>
</tr>
<tr>
<td width="50%" valign="top">

### 📜 Script

Finds clusters of deterministic `(tool_name, arg_shape)` sequences that match the shape of work a plain script could replace.

<pre><code>tj optimize script</code></pre>

[Details →](docs/optimize/script.md)

</td>
<td width="50%" valign="top">

### ✂️ Trim

Predicts which regions of your prompts the model gives little weight to. Surfaces what's safe to cut.

<pre><code>tj optimize trim</code></pre>

[Details →](docs/optimize/trim.md)

</td>
</tr>
<tr>
<td width="50%" valign="top">

### 🔁 Reuse

Detects clusters of sessions where your agent re-plans the same work and exports reviewable skeleton templates you can drop into a slash command or script.

<pre><code>tj optimize reuse</code></pre>

[Details →](docs/optimize/reuse.md)

</td>
<td width="50%" valign="top">

### 🔭 Lens

A local browser dashboard that brings every analyzer's findings, your real spend, and your alerts together in one place. No cloud, no signup, fully offline.

<pre><code>tj serve</code></pre>

[Details →](https://tokenjam.dev/products/lens)

</td>
</tr>
</table>

Run all five analyzers with `tj optimize`. Run several with `tj optimize downsize cache reuse`.

---

## 30-second quickstart

For **Claude Code** users — zero code, auto-backfills your last 30 days:

```bash
pipx install tokenjam
tj onboard --claude-code
tj optimize          # cost-saving candidates from your actual usage
tj serve             # open the dashboard at http://127.0.0.1:7391/
```

Onboarding also wires a **zero-token statusline** into Claude Code — `tj statusline` runs out-of-band each turn (no model quota) and shows this session's re-read share with a `/compact` nudge: `◆ Opus 4.8  2.4M tok  🕳️ re-read 95%  → /compact to reclaim quota`. It does **not** add an in-loop MCP server (that's an SDK / API surface — an MCP would tax every turn).

That's it. Run `tj` any time and it points you to the next best action:

```text
 _____    _              _
|_   _|__| |_____ _ _   | |__ _ _ __
  | |/ _ \ / / -_) ' \  | / _` | '  \
  |_|\___/_\_\___|_||_|_/ \__,_|_|_|_|
                     |__/
  TokenJam · cost-optimization for AI agents · local-first, OTel-native · no signup

You're set up. Next best actions:
  tj status      agent overview — what's running, recent cost
  tj tokenmaxx   your shareable spend tier
  tj optimize    cost-saving candidates from your usage
  tj serve       open Lens (web UI) at http://127.0.0.1:7391/
```

To upgrade later: `pipx upgrade tokenjam` (then `tj stop && tj serve &` to reload the daemon, and `tj --version` to verify). See [docs/installation.md](docs/installation.md#upgrading).

For any Python agent:

```python
from tokenjam.sdk import watch
from tokenjam.sdk.integrations.anthropic import patch_anthropic

patch_anthropic()

@watch(agent_id="my-agent")
def run(task: str) -> str:
    ...
```

→ [Python SDK](docs/python-sdk.md) · [TypeScript SDK](docs/typescript-sdk.md) · [Codex](docs/claude-code-integration.md#codex) · [OTel-compatible agents](docs/framework-support.md)

---

## Lens — the local dashboard

`tj serve` runs Lens at `http://127.0.0.1:7391/`: a **Dashboard** that lands you on recoverable waste and health at a glance, with an embedded explorer to slice your usage any way (metric × dimension × chart); plus Status, Traces, Cost, Analytics, Alerts, Drift, Optimize, and Budget screens. Plan-tier-aware, fully offline, no signup.

<table>
<tr>
<td width="50%"><img src="docs/screenshots/tj-dashboard.png" alt="Dashboard — recoverable waste, health at a glance, and the embedded pivot explorer" /></td>
<td width="50%"><img src="docs/screenshots/tj-cost.png" alt="Cost — spend over time + cache savings" /></td>
</tr>
<tr>
<td width="50%"><img src="docs/screenshots/tj-traces.png" alt="Trace waterfall — session-level spans with cost annotations" /></td>
<td width="50%"><img src="docs/screenshots/tj-status.png" alt="Status — per-agent cards" /></td>
</tr>
<tr>
<td width="50%"><img src="docs/screenshots/tj-dashboard-tools.png" alt="Analytics explorer — tool-usage leaderboard" /></td>
<td width="50%"><img src="docs/screenshots/tj-dashboard-leaderboard.png" alt="Analytics explorer — cost-by-model leaderboard" /></td>
</tr>
</table>

→ [tokenjam.dev/products/lens](https://tokenjam.dev/products/lens) for the visual walkthrough.

---

## Beyond optimization

TokenJam is also a full observability stack. The five analyzers and Lens ride on top.

- **Real-time cost tracking** — every LLM call priced as it happens
- **Safety alerts** — 13 alert types, 6 channels (ntfy, Discord, Telegram, webhook, file, stdout)
- **Behavioral drift detection** — Z-score baselines, no LLM required
- **Schema validation** — declare or infer JSON Schema for tool outputs
- **OTel-native** — point any OTLP exporter at `tj serve` and you're done
- **Statusline** — a zero-token Claude Code status line (`tj statusline`, wired by `tj onboard --claude-code`) showing this session's re-read share + a `/compact` nudge
- **MCP server** — in-request-path tools for **SDK / API** users (not Claude Code / Codex subscription users — an in-loop MCP is a per-turn quota burden there; they get the out-of-band statusline instead)

---

## Prove a swap holds — TokenJam Bench

`tj optimize downsize` flags *candidates*: cheaper models worth a look. It never claims the cheaper model would have produced the same answer. **[TokenJam Bench](https://github.com/Metabuilder-Labs/tokenjam-bench)** is the companion that checks. It runs your original and candidate models against real task suites and reports the pass-rate difference with statistics (Wilson CI + McNemar), so you get a hedged verdict ("holds" or "regressed") instead of a guess.

```bash
pip install tokenjam-bench
tjb run --original anthropic:claude-opus-4-7 --candidate anthropic:claude-haiku-4-5
```

Bench reports measured pass-rate on a suite, never "certified" or "quality preserved." Open source and local, like TokenJam. [Learn more →](https://github.com/Metabuilder-Labs/tokenjam-bench)

---

## CLI

```bash
tj optimize            # all five cost-optimization analyzers
tj optimize downsize   # one analyzer (positional args)
tj tokenmaxx           # shareable spend-tier callout
tj status              # current cost, tokens, active alerts
tj cost --since 7d     # spend by agent / model / day / tool
tj alerts              # everything that fired while you were away
tj drift               # behavioral drift Z-scores
tj report --reuse      # HTML + Markdown skeleton export for the Reuse analyzer
tj backfill claude-code # ingest historical ~/.claude/projects/ sessions
tj serve               # start Lens + REST API
```

[Full CLI reference →](docs/cli-reference.md)

---

## Documentation

| Topic | Where |
|---|---|
| 🪶 Downsize / Cache / Script / Trim deep-dives | [docs/optimize/](docs/optimize/) |
| 🔁 Reuse analyzer deep-dive | [docs/optimize/reuse.md](docs/optimize/reuse.md) |
| 🧪 Prove a downsize candidate holds (TokenJam Bench) | [tokenjam-bench](https://github.com/Metabuilder-Labs/tokenjam-bench) |
| Claude Code & Codex integration | [docs/claude-code-integration.md](docs/claude-code-integration.md) |
| Harness run grouping (governors / fan-out launchers) | [docs/harness-integration.md](docs/harness-integration.md) |
| Python SDK reference | [docs/python-sdk.md](docs/python-sdk.md) |
| TypeScript SDK reference | [docs/typescript-sdk.md](docs/typescript-sdk.md) |
| Framework support (LangChain / CrewAI / etc.) | [docs/framework-support.md](docs/framework-support.md) |
| Alert channels & rule reference | [docs/alerts.md](docs/alerts.md) |
| Backfill from Langfuse / Helicone / OTLP | [docs/backfill/](docs/backfill/) |
| Configuration | [docs/configuration.md](docs/configuration.md) |
| Architecture deep-dive | [docs/architecture.md](docs/architecture.md) |
| Installation extras (Trim, framework patches) | [docs/installation.md](docs/installation.md) |
| Export to Grafana / Datadog / NDJSON | [docs/export.md](docs/export.md) |
| NemoClaw sandbox observer | [docs/nemoclaw-integration.md](docs/nemoclaw-integration.md) |
| Release notes | [GitHub Releases](https://github.com/Metabuilder-Labs/tokenjam/releases) |

---

## Roadmap

**Shipped in 0.3.x:** Downsize · Cache · Script · Trim · Claude Code + Codex onboarding · MCP server · Web UI · Backfill adapters (Langfuse, Helicone, OTLP) · Period comparison · Routing-config export · Read-only policy preview

**Shipped in 0.4.x:**
- [x] **[TokenJam Lens](https://github.com/Metabuilder-Labs/tokenjam/milestone/1)** — local dashboard rebrand: Overview triage front-door, Optimize detail tab, real spend-over-time charts, cross-screen drill-through
- [x] **[Reuse analyzer](https://github.com/Metabuilder-Labs/tokenjam/milestone/2)** — fifth analyzer: detects clusters of sessions with repeated planning, exports reviewable skeleton templates you can convert into slash commands or scripts
- [x] **Daemon DB concurrency** — per-thread DuckDB cursors so the Overview's fan-out doesn't block on a single shared connection (v0.4.1)
- [x] **Cache cost transparency** — `cache_read` + `cache_write` token columns surfaced in CLI + UI + API (the previously-hidden ~91% cost driver on cache-heavy workloads)

**Shipped in 0.5.x:**
- [x] **Lens Visualizations** — an Analytics pivot explorer (metric × dimension × chart, presets, CSV), stacked cost-by-model, a cache-savings chart, KPI sparklines, a cost-annotated trace waterfall, and consistent series coloring
- [x] **Merged Dashboard** — the explorer and the triage front-door unified into one default screen, with in-place drill-through from recoverable-waste tiles
- [x] **First-run polish** — backfill fidelity (session-level traces, cache read/write split, honest session counts), plan-tier-aware framing throughout (subscription users see token-share, never raw spend), an onboarding welcome banner + next-steps guidance, and a contribution funnel

**Up next** (roughly):
- [ ] Continued Lens polish + per-product visual branding
- [ ] `tj policy add | edit | apply` — unified rule surface
- [ ] `tj replay` — replay captured sessions against new model versions
- [ ] TypeScript framework patches (LangChain JS, OpenAI Agents SDK)
- [ ] Vercel AI SDK & Mastra integrations
- [ ] Docker image
- [ ] GitHub Actions for CI drift/cost checks

---

## Contributing

TokenJam is MIT, and contributions are welcome — from a one-line pricing fix to a whole new framework integration. A few easy on-ramps:

- 🟢 **[Good first issues →](https://github.com/Metabuilder-Labs/tokenjam/labels/good%20first%20issue)** — scoped, newcomer-friendly tasks, ready to pick up.
- 💸 **Model pricing** — `tokenjam/pricing/models.toml` is community-maintained. Fix a rate or add a model in a single PR — no issue needed.
- 🔌 **Framework integrations** — provider/framework patches follow one clear pattern (`tokenjam/sdk/integrations/anthropic.py` is the reference). Open an issue first to align on approach.
- 🤖 **Built with coding agents** — TokenJam is built by AI coding agents, and contributing with one is first-class. **Claude Code:** read [CLAUDE.md](CLAUDE.md) and run `/init` to bring your agent up to speed. **Codex / other agents:** [AGENTS.md](AGENTS.md) has the critical rules.

Setup and the full dev workflow are in **[CONTRIBUTING.md](CONTRIBUTING.md)**.

If TokenJam saves you tokens, **⭐ star it** and **👁 watch for releases** — we ship often.

---

<div align="center">

**[tokenjam.dev](https://tokenjam.dev)** · [PyPI](https://pypi.org/project/tokenjam/) · [npm](https://www.npmjs.com/package/@tokenjam/sdk) · [TokenJam Bench](https://github.com/Metabuilder-Labs/tokenjam-bench) · [Issues](https://github.com/Metabuilder-Labs/tokenjam/issues)

MIT License · Built by [Metabuilder Labs](https://github.com/Metabuilder-Labs)

TokenJam was created by [Anil Murty](https://github.com/anilmurty) — reach him at [anil@metabldr.com](mailto:anil@metabldr.com).

</div>
