Metadata-Version: 2.4
Name: tokenome-ai
Version: 0.5.4
Summary: Code provenance for AI-assisted work: capture your AI conversations, index them on your machine, link every line of code to the conversation that produced it
Author: Naive Empiricist
Author-email: Naive Empiricist <yggdrasil@gmail.com>
License-Expression: BUSL-1.1
License-File: LICENSE
Requires-Dist: fastapi>=0.136.3
Requires-Dist: fastembed>=0.8.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: mcp>=1.28.1,<2
Requires-Dist: pydantic>=2.13.4
Requires-Dist: tokenome-core
Requires-Dist: typesense>=2.0.0
Requires-Dist: uvicorn>=0.49.0
Requires-Python: >=3.12
Project-URL: Homepage, https://tokenome.ai
Project-URL: Releases, https://github.com/tokenome/releases
Project-URL: Issues, https://github.com/tokenome/releases/issues
Description-Content-Type: text/markdown

# tokenome

Code provenance for AI-assisted work. Your code says what changed; tokenome
says why. It captures your conversations with AI coding tools (Claude Code,
Claude Desktop, ChatGPT, Gemini), indexes them on your own machine, and links
any line of code to the conversation that produced it.

Search is hybrid (BM25 keyword plus vector meaning) over a local Typesense.
You reach it from the desktop app, the command line, or a web page; an AI
agent reaches it over MCP. Nothing leaves the machine unless you join a team
server and opt a project in.

## Install

**Desktop app (macOS Apple silicon, Linux).** Download the installer from
[github.com/tokenome/releases](https://github.com/tokenome/releases). The app
runs the daemon, bundles its own Python and Typesense, and installs a
`tokenome` command at `~/.local/bin/tokenome` that uses the app's runtime.

**Command line only (Linux, servers, anywhere uv runs).** The distribution on
PyPI is `tokenome-ai`; the command and the import name are `tokenome`:

```bash
uv tool install tokenome-ai
uv tool upgrade tokenome-ai
```

Needs [uv](https://docs.astral.sh/uv/getting-started/installation/) and
Python 3.12 or later. Typesense is downloaded on first run; no Docker. If the desktop
app is installed on the same machine, `uv tool install` replaces the app's
`~/.local/bin/tokenome` with the wheel's own: two runtimes, one data
directory. Keep one or the other.

**From a checkout (contributors):** `uv sync`, then prefix every command
below with `uv run`. The examples use that form; a wheel or app install drops
the prefix.

## Quick Start

```bash
# 1. Start Typesense (auto-downloads binary on first run)
uv run tokenome start

# 2. Initialize the collection
uv run tokenome setup

# 3. Ingest your Claude Code transcripts
uv run tokenome ingest ~/.claude/projects/ --format claude-code

# 4. Search your conversation history
uv run tokenome search "Python type hints"

# 5. Check what's in the index
uv run tokenome status
```

## Ingesting Conversations

### Claude Code (CLI transcripts)

Claude Code stores full JSONL transcripts in `~/.claude/projects/<project>/<uuid>.jsonl`.

Alongside the conversation text, the tool actions Claude Code took are captured
as compact, searchable lines - `[Edit: src/config.py]`, `[Bash: uv run pytest]`,
`[Read: …]`, `[Grep: …]` - so you can find *what was done*, not just what was
said. Full file contents and diffs are omitted to keep the index lean.

```bash
# Single transcript
uv run tokenome ingest ~/.claude/projects/my-project/session.jsonl

# Entire directory of transcripts
uv run tokenome ingest ~/.claude/projects/my-project/ --format claude-code

# Preview without indexing
uv run tokenome ingest ~/.claude/projects/ --format claude-code --dry-run
```

### Claude Desktop / Cowork (API)

Claude Desktop conversations are fetched via the claude.ai API using your
session cookie.

```bash
# 1. Get your sessionKey from Claude Desktop:
#    - Open Claude Desktop
#    - Press Cmd+Option+I (DevTools)
#    - Go to Application > Cookies > claude.ai
#    - Copy the "sessionKey" cookie value

# 2. Set it as an environment variable
export TOKENOME_CLAUDE_SESSION_KEY=sk-ant-sid01-...

# 3. Fetch and index your conversations
uv run tokenome fetch-claude --max-conversations 50

# Preview first
uv run tokenome fetch-claude --max-conversations 10 --dry-run
```

### Claude Cowork Output Files

Cowork writes deliverables (reports, code, spreadsheets) to your filesystem.
Index them as AI-generated documents. For code files inside a git repository,
git blame metadata (committer and timestamp per line) is automatically harvested.

```bash
uv run tokenome ingest ~/projects/cowork-output/ --format cowork-output
```

### ChatGPT Exports

Export your ChatGPT data (Settings > Data Controls > Export Data), unzip,
and ingest the `conversations.json`:

```bash
uv run tokenome ingest conversations.json --format chatgpt-export
```

### Other Formats (Generic JSON)

For flat JSON arrays of `{role, content}` messages from third-party exporters:

```bash
uv run tokenome ingest chat-export.json --format generic-json
```

## Automatic Ingestion (Recommended)

The simplest way to keep your memory current: just run the app and pick your AIs
 -  no paths to configure.

```bash
uv run tokenome app
```

`app` auto-discovers conversation sources on your machine, **auto-enables the
ones it detects**, and polls them in the background while the app is open. Open
the **Sources** panel in the web UI to see what was detected, toggle sources on
or off, or click **Ingest now**:

| Source | How it's found | Notes |
|--------|----------------|-------|
| **Claude Code** | `~/.claude/projects/` (auto) | Detected and polled automatically |
| **Claude Desktop / Cowork** | claude.ai API | Available once `TOKENOME_CLAUDE_SESSION_KEY` is set |
| **ChatGPT** | drop folder `~/.tokenome/import/chatgpt/` | Drop an exported `conversations.json` there |
| **Gemini** | drop folder `~/.tokenome/import/gemini/` | Drop a flat-JSON export there |

Directory and drop-folder sources are polled every minute (only new/changed
files are re-indexed); the Claude Desktop API is polled every 15 minutes. The
drop folders are created automatically - the Sources panel shows their exact
paths. Within a changed file, only the turns that are new or changed are
re-embedded (see [Incremental ingestion](#incremental-ingestion)), so a live,
growing session costs roughly one turn's work per poll, not the whole thread.

For headless/CLI-driven ingestion, the watch mode and Claude Code hook below
remain available.

## Auto-Ingestion (Watch Mode)

Watch a directory for new transcripts and ingest them automatically:

```bash
uv run tokenome watch ~/.claude/projects/ --format claude-code --interval 5
```

Press Ctrl+C to stop.

## Connect Claude Code (Recommended)

One command gives Claude Code memory - the MCP server, a skill that teaches
Claude when and how to use it, the retrospective-prompt nudge, automatic
session ingestion, and `/tokenome:why` / `/tokenome:search`:

```bash
uv run tokenome app            # once: starts the daemon and mints your API token
uv run tokenome claude install # registers the MCP server + installs the plugin
```

The daemon serves MCP at `http://127.0.0.1:8741/mcp`; the plugin's hooks and
Claude's tool calls all go through it, so team routing (when enrolled) and
local auth apply everywhere. Keep the daemon running (the tokenome desktop app,
or `tokenome app`) whenever you want memory available. Details:
`plugins/claude/README.md` and `docs/design/DESIGN_8_CLAUDE_SKILL.md`.

The sections below are the manual, piece-by-piece alternative.

## Auto-Ingestion (Claude Code Hook)

Index every Claude Code session automatically when it ends - no manual step.
Register `tokenome ingest-hook` as a `SessionEnd` hook. It reads the hook
payload from stdin, finds the session's `transcript_path`, and ingests it.

Add this to your Claude Code `settings.json` (`~/.claude/settings.json` for all
projects, or `.claude/settings.json` for one project). Use the **absolute path**
to tokenome's installed entry point, since the hook fires in any directory:

```json
{
  "hooks": {
    "SessionEnd": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "/absolute/path/to/tokenome/.venv/bin/tokenome ingest-hook",
            "timeout": 120,
            "async": true
          }
        ]
      }
    ]
  }
}
```

`"async": true` lets Claude Code background the hook so it never delays session
exit. The hook is intentionally silent and always succeeds - if Typesense isn't
running, it skips quietly. Diagnostics are appended to `~/.tokenome/hook.log`.

Typesense must be running for the hook to index (`tokenome start`). The hook
re-reads the full transcript each time it fires, but ingestion is
**incremental**: only new or content-changed turns are re-embedded and
upserted, so re-runs are cheap and idempotent (see
[Incremental ingestion](#incremental-ingestion)).

## Team sharing (experimental)

Share opted-in projects with your organization's self-hosted tokenome server
(see `docs/remote-search-spec.md` and `packages/tokenome-server/`). Nothing is
shared by default; secrets are scrubbed client-side before anything leaves the
laptop, and pushes queue offline-safe with retry.

```bash
# One-time: enroll this laptop (token comes from your org admin, out-of-band)
uv run tokenome remote enroll --server https://tk.example.org --token <token>

# Opt a project in - from then on its new/changed segments mirror to the
# server automatically as the poller indexes them (secret-redacted, with
# locally computed vectors, so the server never re-embeds)
uv run tokenome remote share tokenome

# Inspect / manage
uv run tokenome remote status      # enrollment, server health, queue depth
uv run tokenome remote projects    # what you're sharing
uv run tokenome remote push        # flush the offline queue now
uv run tokenome remote unshare tokenome --purge   # stop + retract from server
```

Once enrolled, `tokenome search` (and the web Search page) go **team-first**:
queries hit the org server, and if it can't answer within ~1.5s you get your
local results with a visible warning instead. Control it with `--scope`
(web: the Scope selector): `team` (server), `mine` (local only - includes
your private, unshared context), `all` (both, as two sections, deduplicated
so a document you shared never appears twice), or `auto` (the default:
team when enrolled, else mine). Team results show who shared each hit.

Authentication is device-key based (an Ed25519 key generated on your laptop;
only the public key is sent). No passwords, no third-party identity provider.

The complete guide - starting the server, admin workflows, connecting
laptops, scopes, troubleshooting - is
[`docs/team-server-guide.md`](docs/team-server-guide.md); the turnkey docker
compose bundle (Typesense + server + automatic TLS, with backup/restore
runbook) is in [`deploy/`](deploy/README.md).

## Server Management

```bash
# Start Typesense (auto-downloads ~40MB binary on first run)
uv run tokenome start

# Stop Typesense
uv run tokenome stop

# Reset everything - drop and re-create the collection
uv run tokenome clean           # prompts for confirmation
uv run tokenome clean --force   # skips confirmation
```

The same reset is available in the app under **Settings → Danger zone**
("Clear all indexed data"), which also resets background-poller tracking so
detected sources re-ingest from scratch on the next poll.

The binary is cached in `~/.tokenome/bin/`, data in `~/.tokenome/data/`.

## Incremental ingestion

Ingestion is incremental at the segment level. Every ingest path - the CLI,
the web poller, the Claude Code hook, and the Claude Desktop API - reconciles a
conversation against what's already indexed rather than rebuilding it:

- Each segment carries a `content_hash` of its text, stored in the index.
- On (re-)ingest, only segments that are **new** or whose **content changed**
  (e.g. a live session's final turn that grew) are re-embedded and upserted.
  Unchanged turns are skipped - no wasted embedding work.
- Upserts are per-document, so a growing conversation is never deleted and
  re-added; there's no window where a search misses it.
- Segments that segmentation no longer produces are pruned as orphans.

**Clear all is lossless.** Because the hashes live in the index itself,
"Clear all indexed data" (`tokenome clean` / Settings → Danger zone) drops
them along with everything else, so the next ingest re-embeds and re-indexes
the full corpus from scratch under the current code - verified by an
integration test that asserts the rebuilt index is identical to the original.

## Searching

### CLI

```bash
# Basic search
uv run tokenome search "deployment strategies"

# Filter by platform, model, or project
uv run tokenome search "error handling" --platform claude-code
uv run tokenome search "error handling" --model claude-sonnet-4
uv run tokenome search "refactor" --project tokenome

# Filter by the derived AI label (kept for compatibility)
uv run tokenome search "error handling" --ai-label claude-sonnet-4

# Search within a specific conversation
uv run tokenome search "*" --conversation My-Project-Name

# Time-range search
uv run tokenome search "refactor" --after 2025-03-01 --before 2025-03-31
uv run tokenome search "*" --around 2025-03-15T14:30 --window 30

# JSON output (for scripting)
uv run tokenome search "Python" --json | jq .
```

### Cross-referencing code with conversations

Find the conversations that led to a change: tokenome blames a line to learn
when it was last edited, then surfaces the conversations from around that time.

```bash
# When was this line last edited, and what did I discuss just before?
uv run tokenome context src/config.py 42

# Widen to conversations on both sides of the edit (±30 min by default)
uv run tokenome context src/config.py 42 --symmetric --window 45
```

By default it shows the conversations **immediately before** the edit - the
discussions most likely to have driven it. Blame is resolved **live** via
`git blame` (so it reflects the current state and needs no prior ingestion, as
long as the file is in a local git repo); if the file isn't in git, it falls
back to blame captured at ingest time from a `--format cowork-output` import.

The same lookup is available in the app's **Code context** pane.

### Web UI

```bash
# One command - starts Typesense and the daemon, and opens the browser
uv run tokenome app
```

The page the daemon serves at `http://127.0.0.1:8741/` **is the desktop
app's UI** (DESIGN_7 §10): Dashboard, Search, Code context, Conversations,
Journal, FAQ, Team, Sources, and the Settings that make sense in a
browser. Shell-only controls (launch at login, updates) disable
themselves outside the app. It is the UI for Linux and headless daemons,
where there is no desktop app yet.

The page itself carries no credentials. `tokenome app` opens the browser
with a single-use launch link that signs the tab in (an HttpOnly session
cookie). To open a signed-in tab on a daemon that is already running (the
desktop app's, or `tokenome daemon`), run:

```bash
uv run tokenome open            # --port if the daemon is not on 8741
```

Typing `http://127.0.0.1:8741/` by hand loads the page but not your data;
the daemon answers only requests addressed to a loopback host name
(`127.0.0.1`, `localhost`, `[::1]`), which blocks DNS-rebinding pages.

From a checkout the UI must be built once (needs Node):

```bash
apps/desktop/scripts/install-spa.sh   # stages the build for the daemon
```

Until it is staged, the page at `/` says so; the API works regardless.

### MCP Server (for LLMs)

The daemon serves MCP over Streamable HTTP at `http://127.0.0.1:8741/mcp`,
guarded by the local API token (`~/.tokenome/api.token`). Claude Code,
Claude Desktop, and other MCP clients query your conversation memory through
it - one implementation, with team routing when you are enrolled.

**Claude Code:** `uv run tokenome claude install` (see
[Connect Claude Code](#connect-claude-code-recommended)). It runs
`claude mcp add --transport http --scope user --header "X-Tokenome-Token: …"
tokenome http://127.0.0.1:8741/mcp` for you.

**Claude Desktop** (stdio only): keep the `tokenome-mcp` entry in
`~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "tokenome": {
      "command": "uv",
      "args": ["run", "tokenome-mcp"]
    }
  }
}
```

`tokenome-mcp` is a thin proxy to the daemon's `/mcp`, so the daemon must be
running. Without a checkout - the desktop app's *Install command-line tool*
puts `tokenome` on your PATH - use the equivalent subcommand instead:

```json
{
  "mcpServers": {
    "tokenome": { "command": "tokenome", "args": ["mcp"] }
  }
}
```

`TOKENOME_MCP_DIRECT=1` restores the legacy direct-engine mode until it is
removed.

Tools available to the model:

| Tool | Description |
|------|-------------|
| `search_memory(query, top_k, ai_label, platform, model, project, scope)` | Hybrid search, **routed**: `scope=auto` (team when enrolled, else local), `team`, `mine`, `all`. Team hits carry `owner`; a `warning` says when the scope could not be honored |
| `why_was(decision, top_k)` | Distilled rationale from journal why-records, with the evidence span |
| `find_related(doc_id, top_k)` | Nearest-neighbor segments to a result - walk the memory graph |
| `get_document(doc_id)` | Full text of a single segment (expand a search/related excerpt) |
| `get_conversation(conversation_id, full)` | A whole thread, compact per-turn excerpts (or verbatim with `full=True`) |
| `find_conversations_near(file, line, window, symmetric)` | Git-blame a line, then the conversations just before that edit |
| `ask_faq(question)` | Curated/pinned answers |
| `get_journal(date, days)` / `get_recent(limit)` | The daily journal; the most recently indexed segments |
| `browse_by_label(ai_label, limit)` | Browse conversations with a specific AI (compact) |

Search-style tools return **compact** results to keep an agent's context lean:
each result is a highlighted excerpt plus `doc_id`/`conversation_id`/`title`.
The agent expands what it needs with `get_document` (one segment) or
`get_conversation` (the whole thread), and explores laterally with
`find_related`.

## Configuration

All settings via environment variables with sensible defaults:

| Variable | Default | Description |
|----------|---------|-------------|
| `TOKENOME_TYPESENSE_HOST` | `localhost` | Typesense server host |
| `TOKENOME_TYPESENSE_PORT` | `8108` | Typesense server port |
| `TOKENOME_TYPESENSE_API_KEY` | `xyz` | Typesense API key |
| `TOKENOME_TYPESENSE_PROTOCOL` | `http` | `http` or `https` |
| `TOKENOME_EMBEDDING_MODEL` | `BAAI/bge-small-en-v1.5` | fastembed model name |
| `TOKENOME_EMBEDDING_BATCH_SIZE` | `32` | Embeddings per batch |
| `TOKENOME_INGESTION_BATCH_SIZE` | `50` | Documents per Typesense import batch |
| `TOKENOME_CLAUDE_SESSION_KEY` | (none) | sessionKey cookie for `fetch-claude` |
| `TOKENOME_CLAUDE_ORG_ID` | (auto) | Organization UUID for `fetch-claude` |

## Architecture

```
Transcripts / API → Segmenter → Embedder → Typesense (BM25 + vector)
                                              │
                    ┌─────────────────────────┼──────────────────────┐
                    ▼                         ▼                      ▼
              CLI (tokenome)          MCP Server (tokenome-mcp)   Web UI (tokenome app)
              ingest/search/status    AI-accessible tools         http://127.0.0.1:8741
```

- **Segmentation**: Pluggable adapters parse each AI's transcript format into
  user-query/ai-response pairs, tagged with `platform`, `model`, and `project`
  for filtering (with a derived `ai_label` kept for compatibility)
- **Embedding**: fastembed (ONNX, local, no GPU) generates 384-dim vectors
  from BAAI/bge-small-en-v1.5. Long turns are split into overlapping chunks,
  embedded, and mean-pooled so content beyond the model's ~512-token limit is
  not lost
- **Indexing**: Typesense stores documents with hybrid search - BM-25 text
  matching fused with vector similarity at alpha=0.5
- **Access**: Same search backend serves CLI, web UI, and MCP server

## Development

```bash
uv sync                             # Install dependencies
uv run pytest -m "not integration"  # Unit tests (fast, no services needed)
uv run pytest -m integration        # Integration tests: starts an isolated,
                                    #   throwaway Typesense (never touches your
                                    #   real index) and uses the real embedding
                                    #   model (downloaded on first run)
uv run pytest -m e2e                # Browser E2E tests over the app's UI (needs Node)
                                    #   and the desktop SPA (needs Node:
                                    #    npm ci in apps/desktop, once)
                                    #   (one-time setup:
                                    #    uv run playwright install chromium)
uv run pytest                       # Every suite
uv run ruff check .                 # Lint
uv run ruff format .                # Format

(cd apps/desktop && npm ci && npm test)   # Desktop app unit tests (Vitest)

# End-to-end smoke test (starts Typesense, ingests, searches, cleans up)
uv run python scripts/smoke_test.py
```

CI enforces a branch-aware coverage floor on the unit suite
(`--cov-fail-under`, ratchet-up-only - see `.github/workflows/ci.yml` for the
current value); the integration suite reports coverage informationally but is
gated only on passing. To see coverage locally, add `--cov` to either command.

## License

The tokenome client, meaning the `tokenome-ai` and `tokenome-core` packages
and the Claude Code plugin, is licensed under the Business Source License
1.1 (see [LICENSE](LICENSE)). You may use, modify and redistribute it,
including in production, except to offer a competing product. Four years
after each version is published, that version converts to the Apache License
2.0. The desktop app is distributed under its own end-user terms from
[tokenome/releases](https://github.com/tokenome/releases). The team server
is proprietary and licensed separately.

