Metadata-Version: 2.4
Name: tokenfair-digest
Version: 0.1.0
Summary: Client-side digest tool: tokenizes LLM logs locally on the shared ruler and emits metrics-only JSON. No text ever leaves the machine.
Author-email: Garzillion-labs <tadeosun004@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/tflux2011/tokenfair
Project-URL: Repository, https://github.com/tflux2011/tokenfair
Project-URL: Issues, https://github.com/tflux2011/tokenfair/issues
Keywords: llm,tokens,openai,anthropic,gemini,audit,tiktoken
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# tokenfair-digest

Client-side digest tool for LLM API billing audits. Captures SDK responses, recounts tokens locally on a shared `cl100k_base` ruler, and emits metrics-only JSON. **No prompt or completion text ever leaves your machine** — only token counts are written.

## Install

```bash
pip install tokenfair-digest
```

## Capture (drop-in, one line)

Add a single line where you create your client, and every response is logged to a local JSONL file.

### OpenAI

```python
from openai import OpenAI
from tokenfair_digest.capture import capture_openai

client = capture_openai(OpenAI(), "openai_logs.jsonl")
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}],
)
# openai_logs.jsonl now has one record per response
```

### Anthropic

```python
from anthropic import Anthropic
from tokenfair_digest.capture import capture_anthropic

client = capture_anthropic(Anthropic(), "anthropic_logs.jsonl")
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}],
)
```

## Generate a digest

Once you have a JSONL log file, convert it to a metrics-only digest:

```bash
tokenfair-digest openai_logs.jsonl -f openai -o digest.json
```

Supported formats: `openai`, `anthropic`, `gemini`, `atap`

The digest JSON can be opened in the [TokenFair web app](https://github.com/tflux2011/tokenfair) or VS Code extension to audit where your tokens went.

## Programmatic usage

```python
from tokenfair_digest import build_digest, count_tokens, CaptureWriter

# Count tokens on the shared ruler
n = count_tokens("Hello, world!")

# Build a digest from parsed records
from tokenfair_digest.parsers import parse_openai_export
with open("openai_logs.jsonl") as fh:
    digest = build_digest(parse_openai_export(fh))
```

## Guarantees

- **No text on disk.** Only metric fields are written (`record_id`, `provider`, `model`, token counts). Prompt and completion text stay in memory only.
- **Never breaks your call.** The capture tee swallows any logging error — your API call is unaffected.
- **Thread-safe.** Writes are serialized with a lock.
- **Streaming is skipped, not corrupted.** Streamed responses have no `usage` to record.
- **Matches the JS side.** Uses the same `cl100k_base` tokenizer as `@garzillion-labs/capture`, so counts line up.

## License

MIT
