Metadata-Version: 2.4
Name: costtrace
Version: 0.1.0
Summary: A deterministic cost-attribution profiler for LLM traces — an X-ray for where your token spend goes.
Author: Sanat Bhalla
License: MIT
Project-URL: Homepage, https://github.com/sanat77/CostTrace
Project-URL: Repository, https://github.com/sanat77/CostTrace
Project-URL: Issues, https://github.com/sanat77/CostTrace/issues
Keywords: llm,cost,finops,tokens,profiler,observability
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0
Requires-Dist: tiktoken>=0.7
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pyflakes>=3.0; extra == "dev"
Provides-Extra: examples
Requires-Dist: openai-agents>=0.18; extra == "examples"
Dynamic: license-file

# CostTrace

**An X-ray for your LLM token spend.**

CostTrace is a deterministic cost-attribution profiler for LLM applications. You
add **one line** after each agent run to capture the durable identity of your
billed calls. Later, CostTrace re-fetches the exact usage from the provider,
prices it, and tells you three things most dashboards can't:

1. **Where** your money goes — split across system prompt, conversation history,
   retrieved context, the user's turn, and the model's output.
2. **What changed** over time — cost-per-turn jumps, model switches, history
   bloat, longer outputs, cache regressions, or system-prompt edits.
3. **Why** — an LLM-as-judge explains the most likely driver in plain English,
   grounded strictly in your real numbers.

> **Integrations:** today CostTrace supports the **OpenAI Agents SDK (Python)**
> only. The design is provider-tagged, and **support for more SDKs and providers
> is planned** (see [Roadmap](#roadmap)).

---

## Table of contents

- [Why CostTrace](#why-costtrace)
- [How it's different](#how-its-different)
- [Install](#install)
- [Quickstart](#quickstart)
- [Core concepts](#core-concepts)
- [Providers](#providers)
- [CLI reference](#cli-reference)
- [Programmatic API](#programmatic-api)
- [How it works](#how-it-works)
- [The cost model](#the-cost-model)
- [Data, privacy & retention](#data-privacy--retention)
- [Configuration](#configuration)
- [Development](#development)
- [Roadmap](#roadmap)
- [FAQ](#faq)
- [License](#license)

---

## Why CostTrace

LLM bills are opaque. A provider hands you a token count per call, not dollars,
and never tells you *which part* of the prompt cost the money. In a multi-turn
agent, the conversation history is re-sent on every turn, a verbose system
prompt is billed on every call, and a single model swap can 100× your cost — but
the invoice just says "tokens."

CostTrace turns that opaque number into an **itemized, timestamped, queryable**
picture of your spend, so you can answer questions like:

- "What fraction of my bill is just re-sending history?"
- "Our costs doubled last week — what changed?"
- "How much cheaper is `nano` than `pro` for the *same* workload?"
- "Did that system-prompt edit break prompt caching?"

## How it's different

- **Capture is tiny and cheap.** We store only `{provider, conversation_id,
  response_ids}` — no usage numbers, no message content. Everything else is
  re-fetched on demand (provider GET reads are **not** billed).
- **Accounting is deterministic.** Costs are computed by us as `rate × tokens`
  using live [LiteLLM](https://github.com/BerriAI/litellm) pricing (with a
  bundled offline fallback) — reproducible, auditable, no guessing.
- **Detection is general, not single-cause.** Rather than blaming one thing,
  CostTrace compares a recent window against a prior one across *every*
  dimension and surfaces each signal that moved.
- **The LLM only explains, never computes.** The judge receives a compact,
  number-grounded digest and is instructed to reason over it — it never invents
  figures.
- **It's a profiler, not a proxy.** No gateway to route traffic through, no
  latency added to your requests. You capture ids; analysis happens offline,
  on demand.

## Install

```bash
pip install costtrace
```

This installs the importable **`CostTrace`** package and the **`costtrace`** CLI.

Set your OpenAI key via the environment or a `.env` in your project root:

```bash
export OPENAI_API_KEY=sk-...
# or, in ./.env
# OPENAI_API_KEY=sk-...
```

Requires Python 3.9+.

## Quickstart

### 1. Capture — one line after each agent run

```python
from agents import Agent, OpenAIConversationsSession, Runner
from CostTrace import capture

session = OpenAIConversationsSession()
result = await Runner.run(agent, user_input, session=session)

capture(result, conversation_id, provider="openai_agents")   # -> ./.costtrace/captures.json
```

`capture()` extracts the response ids from the run, validates them, de-dupes,
timestamps them, and appends them under the conversation id. It's idempotent and
prunes anything older than the retention window on every write.

`provider` is **required** — it records which backend the ids came from so they
can be re-fetched later. Passing an unknown/missing provider raises an error that
lists the [available providers](#providers).

You can also pass a plain list of ids instead of a `RunResult`:

```python
capture(["resp_abc", "resp_def"], conversation_id, provider="openai_agents")
```

### 2. Analyze — from the CLI

```bash
costtrace analyze                              # everything captured
costtrace analyze --limit 5                    # 5 most recent conversations
costtrace analyze --date 2026/07/20            # a single day
costtrace analyze --starting-date 2026/07/01   # from that day until today
costtrace analyze --conversationId conv_abc    # one conversation
costtrace analyze --no-judge                   # skip the LLM explanation
```

Example output:

```
======================================================================
WHERE THE MONEY GOES
======================================================================
sessions: 3   total: $6.143908   avg cache-hit: 53%
  output      85.9%   $5.275134
  history     14.0%   $0.857593
  system       0.1%   $0.009336
  user         0.0%   $0.001846
  retrieved    0.0%   $0.000000

======================================================================
PER-MODEL COMPARISON
======================================================================
  model               cost/turn   cache  out tok/turn  turns
  gpt-5.4-pro        $ 1.118778     0%          5376      5
  gpt-5.4            $ 0.043458    80%          2305     12
  gpt-5.4-nano       $ 0.002377    78%          1472     12

  gpt-5.4-pro costs 470.6x more per turn than gpt-5.4-nano

======================================================================
WHAT CHANGED (deterministic signals)
======================================================================
  [1] (medium) dominant_bucket: 'output' is 86% of total spend ($5.275134)

======================================================================
JUDGE (LLM explanation)
======================================================================
Output dominates spend; the cost gap tracks the switch to gpt-5.4-pro
(higher rate + 3.6x longer outputs + 0% cache vs ~80%). Consider routing
routine turns to gpt-5.4-nano and capping output length.
```

## Core concepts

| Term | Meaning |
|------|---------|
| **capture** | The dev-side one-liner. Persists `{provider, conversation_id, response_ids}` — the durable *identity* of billed calls, nothing else. |
| **conversation** | One session, grouped by the provider's conversation id. |
| **call** | One billed model request == one `response_id`. |
| **message kind** | Each item in a call is classified as `system`, `history`, `retrieved`, `user`, or `output`. |
| **bucket** | Cost aggregated by message kind — the "where the money goes" split. |
| **session metric** | One durable, timestamped row per conversation (cost, buckets, cache rate, cost/turn, model, system-prompt fingerprint). |
| **provider tag** | Routes fetching to the right backend. Currently only `openai_agents`. |

## Providers

`provider` is a **required** argument to `capture()` — it records which backend
the response ids belong to, so CostTrace knows where to re-fetch usage from.

| Provider tag | Backend | Status |
|--------------|---------|--------|
| `openai_agents` | OpenAI Agents SDK (Python) | ✅ Available |
| _(more)_ | OpenAI Python SDK, Anthropic, Langfuse/Helicone exports, … | 🚧 Planned ([roadmap](#roadmap)) |

Passing an unknown or missing provider fails fast with an error that lists the
currently supported tags:

```python
>>> capture(result, conversation_id)                     # missing
CaptureSchemaError: provider is required — pass one of the supported providers:
['openai_agents'] (e.g. capture(result, conversation_id, provider='openai_agents'))

>>> capture(result, conversation_id, provider="anthropic")   # unknown
CaptureSchemaError: unknown provider 'anthropic'; supported providers: ['openai_agents']
```

The available tags are also exposed programmatically:

```python
from CostTrace import Provider
Provider.values()        # ['openai_agents']
```

## CLI reference

```
costtrace analyze [options]
```

| Option | Description |
|--------|-------------|
| `--limit N` | Analyze the N most recent conversations. |
| `--conversationId conv_xxx` | Analyze a single conversation (alias: `--conversation-id`). |
| `--date yyyy/mm/dd` | Only conversations from that day. |
| `--starting-date yyyy/mm/dd` | Conversations from that day until today. |
| `--no-judge` | Skip the LLM explanation (fully offline). |

`--date` and `--starting-date` are mutually exclusive. Run `costtrace --help` or
`costtrace analyze --help` for the full, self-documenting listing.

## Programmatic API

```python
from CostTrace import load_captures, fetch_captures, Pricer

captures = load_captures(".costtrace/captures.json")   # validated ids
report = fetch_captures(captures)                       # validated traces

report.conversations   # list[ConversationTrace] — ready to analyze
report.rejections      # ids dropped (expired/invalid), each with a reason
report.fetched_calls   # count of successfully fetched calls
```

Build your own analysis with the `analysis` module:

```python
from CostTrace.analysis import (
    compute_session_metric, upsert_metrics, load_metrics,
    summarize_overall, summarize_by_model, detect_all, judge,
)

pricer = Pricer.default()  # live LiteLLM pricing, cached, yaml fallback
metrics = [compute_session_metric(c, pricer) for c in report.conversations]
upsert_metrics(metrics)                       # persist to .costtrace/metrics.jsonl

rows = load_metrics()
overall = summarize_overall(rows)
findings = detect_all(overall, rows)
verdict = judge(overall, findings, by_model=summarize_by_model(rows))
print(verdict.verdict)
```

## How it works

```
                your app
                   │  capture(result, conversation_id)
                   ▼
   .costtrace/captures.json   ── just ids, 15-day retention, auto-pruned
                   │
   costtrace analyze:
                   ▼
   ingest ──► fetch ──► metrics ──► store ──► aggregate ──► detect ──► judge
   (validate  (re-fetch  (bucket    (durable   (overall +   (what      (LLM
    ids)       usage &    split +    JSONL)     per-model +  changed)    explains
               content)   cost)                 per-prompt)             why)
```

1. **Ingest** — strict schema + id-format validation; clear errors; duplicate
   conversations merged.
2. **Fetch** — for each id, pull the response (model + usage) and its input
   items from the provider, then classify each message (system/history/
   retrieved/user/output). **Resilient:** expired or invalid ids are recorded as
   rejections and skipped; only auth failures are fatal.
3. **Metrics** — per conversation, compute total cost and split the input cost
   across buckets by token share (estimated with a tokenizer, then scaled so the
   parts sum to the provider's real `input_tokens`). Output cost is exact.
4. **Store** — append/upsert one row per conversation to
   `.costtrace/metrics.jsonl`; refresh rows that gained turns; prune expired
   rows. This history survives the id-retention window.
5. **Aggregate** — overall spend split, per-model comparison, per-prompt-version
   grouping.
6. **Detect** — compare a recent window vs a prior one and flag every dimension
   that moved: `cost_increase`, `model_shift`, `bucket_shift`, `cache_drop`,
   `output_growth`, `prompt_change`, plus a static `dominant_bucket`.
7. **Judge** — assemble a compact, grounded digest and ask an LLM to name the
   likely cause and a fix. It reasons over the signals; it does not compute
   numbers.

## The cost model

Providers return **tokens, not dollars** — and never a per-call price. CostTrace
computes cost itself:

```
cost = uncached_input_tokens × input_rate
     + cached_input_tokens   × cache_read_rate
     + output_tokens         × output_rate
```

Rates come from LiteLLM's community-maintained pricing table (≈3,000 models),
fetched and cached locally, with a small bundled `pricing.yaml` as an offline
fallback. Because accounting is pure arithmetic on real token counts, results
are reproducible and auditable.

The **bucket split** for the input side is an approximation: the provider bills
one `input_tokens` number per call, so CostTrace estimates each message's share
with a tokenizer and scales those shares to sum exactly to the real input cost.
The **output** bucket is exact.

## Data, privacy & retention

- **What's stored on disk:** only response/conversation **ids** in
  `.costtrace/captures.json`, plus computed **metrics** (numbers, model names,
  and a hash + copy of the system prompt) in `.costtrace/metrics.jsonl`, plus a
  cached pricing table. No user/assistant message content is persisted.
- **Where:** a project-local `.costtrace/` directory, discovered by walking up
  to the nearest `.git` / `pyproject.toml` (like git finds its root), so
  `capture()` and the CLI always agree regardless of your working directory.
  Override with `COSTTRACE_HOME`.
- **Retention:** ids are only re-fetchable from the provider for a limited
  window, so captures older than **15 days** are pruned automatically on write —
  you lose nothing that could still be fetched.
- **The judge digest** contains aggregate numbers and (only when a prompt change
  is detected) the system-prompt text — never raw conversation content.
- **Secrets:** your API key is read from the environment or a `.env`; it is
  never written to the data directory.

## Configuration

| Variable | Purpose |
|----------|---------|
| `OPENAI_API_KEY` | Required for fetching usage and for the LLM judge. |
| `COSTTRACE_HOME` | Override the `.costtrace/` data directory location. |

## Development

```bash
git clone https://github.com/sanat77/CostTrace.git
cd CostTrace
python -m venv .venv && source .venv/bin/activate
pip install -e .[dev]

python -m pytest -q                       # run the test suite
python -m pyflakes src tests conftest.py  # lint
```

The package source lives in `src/` and is mapped to the import name `CostTrace`
via `package-dir` in `pyproject.toml`, so the built wheel ships `CostTrace/` and
`pip install` / `import CostTrace` are unaffected by the on-disk layout. A
`conftest.py` replicates that mapping so tests import from the source tree
without installing.

## Roadmap

CostTrace v1 targets the **OpenAI Agents SDK (Python)** (`provider =
"openai_agents"`). Planned next:

- **More Python LLM SDKs / providers** (e.g. the OpenAI Python SDK directly,
  Anthropic, and others) behind the same provider-tagged architecture.
- **Ingesting exports** from existing observability tools (e.g. Langfuse /
  Helicone) as additional provider tags.
- **Longer retention & trend history** as an opt-in tier.
- **Richer detectors** and per-tenant / per-feature attribution.

Contributions and provider requests are welcome — open an issue.

## FAQ

**Does this add latency to my app?** No. You only save ids at runtime; all
fetching and analysis happens later, offline.

**Do the re-fetches cost money?** No — provider GET reads of past responses are
not billed. Only your original generations were.

**Why can't I analyze conversations from last month?** Ids expire at the
provider (~retention window), so they can't be re-fetched. The computed metrics
you already stored remain, but new deep analysis needs live ids.

**Distribution name vs import name?** You `pip install costtrace` (lowercase)
and `import CostTrace` (capitalized) — standard PyPI convention.

## License

MIT — see [LICENSE](LICENSE).
