Metadata-Version: 2.5
Name: ai-tally
Version: 0.1.0
Summary: ai-tally Python SDK: cost-and-value observability for AI products (OTel gen_ai.* native)
Project-URL: Homepage, https://ai-tally.com
Project-URL: Documentation, https://ai-tally.com/docs/connect/python-sdk
Project-URL: Changelog, https://ai-tally.com/docs/changelog
Author: Tallynomics LLC
License-Expression: MIT
License-File: LICENSE
Keywords: ai,cost,finops,llm,observability,opentelemetry
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: celery>=5.4; extra == 'dev'
Requires-Dist: django>=5.1; extra == 'dev'
Requires-Dist: fakeredis>=2.26; extra == 'dev'
Requires-Dist: fastapi>=0.115; extra == 'dev'
Requires-Dist: flask>=3.0; extra == 'dev'
Requires-Dist: gevent>=24.11; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: jsonschema>=4.26.0; extra == 'dev'
Requires-Dist: pytest>=9.1.1; extra == 'dev'
Requires-Dist: pyyaml>=6.0.3; extra == 'dev'
Requires-Dist: rq>=2.0; extra == 'dev'
Requires-Dist: ruff<0.18,>=0.17.2; extra == 'dev'
Provides-Extra: litellm
Requires-Dist: litellm==1.105.1; extra == 'litellm'
Provides-Extra: mcp
Requires-Dist: mcp>=1.2; extra == 'mcp'
Requires-Dist: pyyaml>=6.0.3; extra == 'mcp'
Description-Content-Type: text/markdown

# ai-tally (Python SDK)

The Python SDK for [AI Tally](https://ai-tally.com). It records what each AI call in your app
costs, which feature made the call, and which of your customers it was for. By default it sends
only counts, costs and hashed ids. Your prompts and answers stay in your app.

```bash
pip install ai-tally
```

The SDK has no dependencies. Your calls always go ahead as normal: if something inside the SDK
goes wrong, it logs a warning and carries on.

Full documentation: https://ai-tally.com/docs/connect/python-sdk

## Quick start

There are two ways to connect. Pick one.

### With no code change: `tally run` (preview)

Install the SDK in the same environment as your app, set your ingest key from the dashboard's
Setup page as `TALLY_KEY`, and put `tally run --` in front of the command that starts your app:

```bash
pip install ai-tally
export TALLY_KEY=tally_sk_live_...
tally run -- uvicorn main:app
```

Your app runs as before, with its model calls recorded. Its own output is unchanged, and each
process that loads an AI library writes a startup line to stderr saying what it records.
`tally run` is a preview, so its options and output may change. "Running your app with
`tally run`" below says where the command goes and what it records.

In a uv, Poetry or Pipenv project, run it from the project's environment, for example
`uv run tally run -- uvicorn main:app` or `poetry run tally run -- uvicorn main:app`.

### In your code: `aitally.init()`

Call `aitally.init()` once when your app starts. Pass the ingest key from the dashboard's Setup
page, or set it as `TALLY_KEY`.

```python
import aitally

aitally.init("tally_sk_live_...")

# Your existing calls are now recorded. You don't change them:
client.chat.completions.create(model="gpt-4o-mini", messages=[...])     # openai
client.messages.create(model="claude-sonnet-4-5", messages=[...])       # anthropic
client.models.generate_content(model="gemini-2.5-flash", contents=...)  # google-genai
bedrock.converse(modelId="us.amazon.nova-pro-v1:0", messages=[...])     # boto3
```

`init()` is safe to call more than once, and it doesn't wait on the network. If the key is missing
or the address is refused, it logs one warning and records nothing. Calls run normally if AI Tally
is unreachable or the key is wrong, and the SDK logs a warning.

## What gets recorded on its own

Both ways record the same calls.

| Library | Calls | Async | Streaming |
| --- | --- | --- | --- |
| `openai` | `chat.completions.create`, `responses.create`, `embeddings.create` | yes | yes |
| `anthropic` | `messages.create`, `messages.stream` | yes | yes |
| `google-genai` | `models.generate_content`, `models.generate_content_stream` | yes (`client.aio`) | yes |
| `boto3` (Bedrock) | `converse`, `converse_stream` | no | yes |

The SDK only touches a library you already have installed. It doesn't install any of them.

Anything else needs one line after the call, `aitally.record_llm_call(...)`. That includes
Bedrock `invoke_model`, the old `google-generativeai` package, and any other client. For LiteLLM,
see the LiteLLM section below.

Good to know:

- **Choose which libraries to record.** `aitally.init(key, instrument={"openai", "anthropic"})`
  records only those two. The names are `"openai"`, `"anthropic"`, `"google"` and `"bedrock"`.
  `instrument=False` turns all of them off.
- **Recording a library only once it is imported.** Recording a library means importing it, so
  the default `init()` imports every one you have installed. To import nothing up front, call
  `aitally.init(key, instrument=False)`, then `aitally.patch("openai")` (or any name above) once
  your code has imported that library. `patch()` imports no other provider, is safe to call more
  than once, and never raises; it returns `False` when there was nothing to record yet. After
  `init(key, instrument={"openai"})`, `patch()` records only the libraries in that set.
- **Other providers through the `openai` library.** Calls to Groq, Fireworks, Together, DeepSeek,
  xAI, OpenRouter, Mistral, Azure or Gemini through the `openai` library are labeled with the real
  provider, never as OpenAI. A provider AI Tally has no price for shows a blank cost, not a guess.
  A host the SDK doesn't recognize is labeled `openai-compatible` and its cost is blank.
- **Calls the table doesn't cover.** Once the SDK records `openai`, `anthropic` or `google`, a
  model call to `api.openai.com` or `api.anthropic.com` that none of the methods above made is
  recorded too: your own `httpx` call, or a method such as `chat.completions.parse` or
  `beta.messages.create`. That covers Chat Completions, Responses, Embeddings and Messages,
  streamed or not, tested with `httpx` 0.27.2 and 0.28.1. Each call is counted once, also when you already
  record it with `record_llm_call`. Only the model and the token counts are read from the
  response, and nothing from your request, so such a call shows no requested model. Calls sent
  with `requests`, `aiohttp` or another HTTP library aren't seen. With the LiteLLM logger loaded
  this is off, because the logger records those calls itself. `TALLY_HTTPX_GAP=0` (or `false`, `no`, `off`) turns it off
  and keeps the methods in the table recorded.
- **Streaming with OpenAI.** OpenAI only reports token counts on a stream if you ask for them with
  `stream_options={"include_usage": True}`. Without that, the cost shows as blank. Pass
  `aitally.init(..., instrument_stream_usage=True)` and the SDK asks for you. It is off by default
  because it changes your request: every `chat.completions.create(stream=True)` that sets no
  `stream_options` is sent with `include_usage`. OpenAI then ends the stream with one extra chunk
  whose `choices` list is empty, so code that reads `chunk.choices[0]` on every chunk fails on it. Some OpenAI-compatible hosts
  reached through the `openai` library reject the parameter and fail the request. Turn it on only
  if your stream handling skips empty chunks and every host you call accepts it.
- **Bedrock streams** are recorded once your code has read the stream to the end.
- **Recorded streams are a stand-in.** A streamed call returns a stand-in that passes on the
  provider's own chunks and works with `for`, `next()`, `async for`, `anext()`, `with`, `close()`
  and the stream's attributes (`next()` works even on a Bedrock stream, which has no `next()` of
  its own). It is not the provider's class, so `isinstance(stream, openai.Stream)` (or
  `anthropic.Stream`) is False. The call is recorded once, from the chunks your code read, when
  the stream ends, is closed, raises, or its `with` block exits. A stream your code drops
  without doing any of those is not recorded.
- **Bedrock inference profiles and provisioned models.** If the model can't be read from the ARN
  (an application inference profile or a provisioned model), the call is recorded with no model
  and a blank cost.
- **Other kinds of cost** are one line each: `aitally.record_vector_call` for vector search,
  `aitally.record_tool_call` for your own tool calls, and `aitally.record_embedding_call` for
  embeddings not made through a recorded client. They do nothing until `init()` has run.
- **Before your process exits**, the SDK sends what it has buffered. Call `aitally.flush()` to send
  it sooner. `aitally.uninstrument()` undoes everything, which helps in tests.
- **Forking servers.** Workers forked after `init()` (gunicorn `--preload`, Celery prefork,
  `multiprocessing` with the fork start method) each get their own connection and send only their
  own calls; nothing the parent had buffered is sent twice. You don't need to call `init()` again
  after the fork.
- **Calls through the AI Tally proxy** are recorded by the proxy, so the SDK skips them and nothing
  is counted twice.

### Remove manual lines you no longer need

If you added `aitally.record_llm_call(...)` after a Gemini or Bedrock `converse` call, delete it.
`init()` records those calls now. Until you do, the SDK notices most duplicates (same call, same
thread or async task, a few seconds apart), drops the copy and warns you once. Don't rely on that:
it can't catch a manual line that runs in a different thread or task, for example under
`asyncio.gather`.

### The connection must be secure

The SDK sends your key with every request, so it only connects over `https://`. Plain `http://`
works only for your own machine (such as `localhost` or `127.0.0.1`), so local testing still works. To use
`http://` for anything else, such as another machine on your network, you have to say so:
`aitally.init(key, endpoint=..., allow_insecure_http=True)`, or `TALLY_ALLOW_INSECURE_HTTP=1`.
Under `tally run`, set a different address with `TALLY_ENDPOINT`.

If the address is refused, the SDK logs one warning and records nothing. It also doesn't follow
redirects, so give it the final address.

## Running your app with `tally run` (preview)

`tally run -- <your start command>` starts your command, and each Python process it runs loads
the SDK when it first imports an AI library. Your code stays as it is. It reads the key from
`TALLY_KEY`, so set it in the environment your app runs in, such as your container settings, your
platform's config vars or your secret manager.

Once your app imports an AI library (for Bedrock, at its first call), its process writes a line
like this to stderr:

```
tally run[4211]: metering openai 2.8.1 (from first use) | account from header x-tenant-id, features from routes | eval capture off | frameworks seen: starlette, fastapi
```

A library or framework your app imports after that line gets a line of its own.

`tally doctor -- <your start command>` shows each call's fields as well (see "What leaves your
app" below).

### Where the start command goes

Put `tally run --` in front of the command you already use to start your app. Change it only in
the one place that matches how you start your app today. Docker is optional.

From a shell, a script or a systemd unit:

```bash
tally run -- uvicorn main:app --host 0.0.0.0
```

In a Dockerfile:

```dockerfile
CMD exec tally run -- uvicorn main:app --host 0.0.0.0
```

`exec` makes your server the container's main process, so it gets the stop signal directly and
sends its last calls before it exits. The exec form,
`CMD ["tally", "run", "--", "uvicorn", "main:app", "--host", "0.0.0.0"]`, works the same way.

In a Procfile:

```
web: tally run -- gunicorn app:app
worker: tally run -- celery -A tasks worker
```

Any command that starts Python from the environment `ai-tally` is installed in works, such as
`python main.py`, `gunicorn`, `uvicorn`, `celery` or `python manage.py runserver`. Python processes
that the command starts, such as gunicorn, uvicorn and Celery workers, are recorded too.

To set a variable or change directory first, write `env VAR=1 tally run -- uvicorn main:app` or
`cd app && exec tally run -- uvicorn main:app`.

### What it records on its own

- **Model calls.** The libraries and calls in "What gets recorded on its own" above: `openai`,
  `anthropic`, `google-genai` and Bedrock through `boto3`. Each library is recorded from the
  moment your app first uses it. Your requests are sent as your code wrote them. For an OpenAI
  stream's cost, pass `stream_options={"include_usage": True}` in your call.
- **Features, from your routes.** In FastAPI, Starlette, Flask and Django apps, each model call
  made while serving a request is named after the request's matched route. The route
  `/api/v1/tickets/{id}/summary` becomes the feature `api-tickets-id-summary`: a leading version
  such as `v1` is dropped, and each path parameter shows by its name. A request that matches no
  route, such as a 404 or a static file, has no feature name. If two routes would share a name,
  the method is added (`-get`, `-post`). Streamed responses, and work the request hands to a
  `ThreadPoolExecutor` or a new `threading.Thread`, count toward the request while it is open.
- **Background jobs.** Each Celery task and RQ job run is recorded on its own, named after the
  task (`tasks.summarize` becomes `tasks-summarize`). Its customer comes from
  `[jobs] account_from`.
- **Your customer.** Tell `tally run` where your app keeps the customer id, in the settings
  below, and each call is recorded for that customer. The id is hashed in your process, as with
  `aitally.with_account()`. Where your code calls `with_account()` or sets a `feature_tag`
  itself, that value is used.

With no settings, features come from your routes and calls are recorded without a customer.

### Settings

Settings go in `tally.toml` in your app's working directory, or in your `pyproject.toml` there
under `[tool.tally.account]`, `[tool.tally.features]` and so on. If both files are there,
`tally.toml` is used. To use another file, set `TALLY_RUN_CONFIG` to its path. Each setting also
has an environment variable, which wins over the file.

```toml
[account]
from = "header"         # header, jwt_claim, request_state or none
name = "x-tenant-id"    # the header, claim or attribute that holds your customer id

[features]
ignore = ["health"]                                      # no feature name for /health
rename = { api-tickets-id-summary = "ticket-summary" }   # a shorter name for one route

[jobs]
account_from = "arg0"   # the task's first argument, or the name of a keyword argument
```

| Setting | Variable | Value |
| --- | --- | --- |
| `[account] from` | `TALLY_ACCOUNT_FROM` | `header`, `jwt_claim`, `request_state` or `none` (the default) |
| `[account] name` | `TALLY_ACCOUNT_NAME` | the header, claim or attribute to read |
| `[account] label` | `TALLY_ACCOUNT_LABEL` | optional: where a readable customer name is, read the same way |
| `[features] from` | `TALLY_FEATURES_FROM` | `route` (the default) or `none` |
| `[features] ignore` | `TALLY_FEATURES_IGNORE` | feature names to leave blank, such as a health check |
| `[features] rename` | `TALLY_FEATURES_RENAME` | new names for features (`old=new` pairs in the variable) |
| `[jobs] account_from` | `TALLY_JOBS_ACCOUNT_FROM` | `arg0` or a keyword argument's name |
| `[eval] capture`, `dir`, `sample_rate`, `retention_days` | `TALLY_EVAL_CAPTURE`, `TALLY_EVAL_DIR`, `TALLY_EVAL_SAMPLE_RATE`, `TALLY_EVAL_RETENTION_DAYS` | local replay samples, as in "Keeping replay samples on your own server" below |

In the variables, lists and pairs are comma-separated.

Where the customer id comes from:

- **`header`**: a request header, such as `x-tenant-id`.
- **`jwt_claim`**: a claim in the request's `Authorization: Bearer` token, such as `org_id`, or
  `org.id` for a nested claim (a dot always means nesting). `tally run` reads the claim to name
  the customer and leaves checking the token to your app's own auth.
- **`request_state`**: a plain attribute your app sets before the model call: on `request.state`
  in FastAPI and Starlette, on `flask.g` in Flask, or on the `request` in Django. A dotted path
  such as `user.org_id` follows attributes. Values your app has already set are read, so
  properties and lazy values your app hasn't loaded yet are left as they are.

A label is sent as readable text, so use a company or account name for it. Label sources that
name a person (`name`, `email`, `given_name`, `family_name`) are left out.

A setting the SDK can't use is named in a warning on stderr, and your app starts as normal. The
section it belongs to is then off: for example, a problem in `[account]` means calls are recorded
without a customer. A file that isn't valid TOML leaves metering on and turns customers and route
features off. On Python 3.10, reading the file needs `tomli` installed. The environment variables
work on every version.

### Turning it off

Set `TALLY_DISABLED=1` to turn `tally run` off with the same start command and image. Your app
then starts as it does without `tally run`. Your own `sitecustomize`, if you have one, runs either
way. An `aitally.init()` call in your own code keeps recording. Unset it, or set it to `0`, to
turn `tally run` back on.

## Cost per customer

Tell the SDK which of your customers a call is for, and AI Tally can show cost per customer. Under
`tally run`, the `[account]` settings above do this with no code. In your code, set the customer
once, for example in your request middleware, and every call inside picks it up:

```python
import aitally

aitally.init("tally_sk_live_...")

with aitally.start_trace(feature_tag="support-bot"), aitally.with_account("acct_northwind"):
    client.chat.completions.create(model="gpt-4o", messages=[...])   # recorded for acct_northwind
    aitally.record_tool_call(provider="openai", tool="web_search")
```

A batch job that works through many customers can pass the customer on each call instead:

```python
import aitally
from aitally.pricing import Usage

with aitally.start_trace():
    for account in accounts:
        aitally.record_llm_call(
            provider="openai", model="gpt-4o",
            usage=Usage(input_tokens=1200, output_tokens=300),
            account_id=account,
        )
```

`with_account(None)` clears the customer for a block, for background work that shouldn't inherit
it.

**Your customer ids stay in your app.** The SDK replaces each id with a hash (HMAC-SHA256) made
with a key that belongs to your organization only. AI Tally sees the hash, never the id, and the
same id gives a different hash for every organization. The SDK first fetches this hashing key
when `init()` runs. Calls made before it arrives are recorded without a customer. If that first
fetch fails, calls are still recorded but without a customer until the app restarts.

You can add a readable name so the dashboard shows "Northwind Traders" instead of a short hash:

```python
with aitally.with_account("acct_northwind", label="Northwind Traders"):
    ...
```

The name is kept apart from your usage data. Names are optional.

### Hashing an id yourself

If you use the AI Tally proxy instead of the SDK, send the customer as a hash in the
`X-Tally-Account-Id-Hash` header. Make the hash on your machine with the same key:

```python
from aitally import hash_account
h = hash_account("acct_northwind")
```

```bash
python -m aitally.hash_account acct_northwind   # reads TALLY_KEY
```

Unlike recording, `hash_account` raises an error if it can't get the hashing key.

## Command line

The SDK comes with a `tally` command. It reads `TALLY_KEY`, else the key `tally login` saved.

```bash
tally run -- uvicorn main:app     # starts your app with its model calls recorded (preview)
tally doctor -- uvicorn main:app  # runs your app and prints what its first calls would send
tally login                    # saves your key (not shown on screen) to ~/.config/tally/config.ini
tally logout                   # removes the saved key
tally health                   # what your key can do, and model calls in the last 24 hours
tally keys                     # your organization's keys, without the secrets (needs a write or admin key)
tally test-event               # sends one empty test event and shows what arrived
tally spend --month 2026-10    # that month's spend by feature and provider
tally eval --samples DIR ...   # compares models on your local replay samples
tally --version                # the installed SDK version
```

`TALLY_KEY` overrides the saved key, so CI needs no config file. `tally run` and `tally doctor` read
`TALLY_KEY`, and `aitally.init()` reads the key you pass or `TALLY_KEY`. Neither reads the saved
key. `health`, `keys`,
`test-event`, `spend` and `eval` take `--json`. The key stays hidden in all output. A number AI
Tally hasn't worked out yet shows as `unknown`. If some calls in a month have no price, the total
shows as "at least".

## Supported versions

The SDK supports the last two major versions of each library it records or hooks. Its
compatibility run checks every acceptance gate against the oldest and the newest release in that
window, on every supported Python, against a local stand-in for the providers, and fails if any
gate fails for any library and version.

<!-- compat-matrix:start -->
Tested on Python 3.10 to 3.13, at the oldest and the newest release of each:

| Library | Supported majors | Oldest tested | Newest tested |
| --- | --- | --- | --- |
| `openai` | 2.x, 3.x | 2.0.0 | the newest release |
| `anthropic` | 1.x, 2.x | 1.0.0 | the newest release |
| `google-genai` | 2.x, 3.x | 2.0.0 | the newest release |
| `boto3` | 1.x | 1.34.116 (Bedrock Converse first shipped in boto3 and botocore 1.34.116) | the newest release |
| `fastapi` | 0.x | 0.115.0 (FastAPI has no major release yet; 0.115.0 is the floor the SDK's own tests use) | the newest release |
| `starlette` | 0.x, 1.x | 0.37.2 (the oldest Starlette FastAPI 0.115.0 accepts) | the newest release |
| `flask` | 2.x, 3.x | 2.0.0 | the newest release |
| `django` | 5.x, 6.x | 5.0 | the newest release |
| `celery` | 5.x | 5.0.0 (Celery 4 does not run on Python 3.10) | the newest release |
| `rq` | 2.x, 3.x | 2.0.0 | the newest release |
<!-- compat-matrix:end -->

## Upgrading

Upgrade `ai-tally` like any other library your app depends on:

```bash
pip install -U ai-tally
```

Or bump it in your lockfile, or let Dependabot or Renovate open the pull request for you. To see
whether a newer version is out:

```bash
pip list --outdated
uv pip list --outdated
```

The changelog (https://ai-tally.com/docs/changelog, and `CHANGELOG.md` in each release's source
archive) lists what each release changes and the library versions it supports. The SDK makes no
version-check calls of its own.

## What leaves your app, and checking it with `tally doctor`

`tally doctor -- <your start command>` runs your app for real under `tally run`: it serves real
requests and makes real model calls, so run it where that's fine, such as staging or your laptop.
For the first five model calls of each process (`--calls N` for more), it prints on stderr every
field that call would send, where the customer's account came from (for example
`from header x-tenant-id`, never the id itself) and which route named the feature. Then it prints
a summary. Nothing is sent to AI Tally unless you add `--send`. The hashing key is still fetched,
so you see the account hash a real call would carry. A call made before the key arrives says
`account not attributed yet: key not fetched`, and doctor waits up to ten seconds for the key.

```bash
TALLY_KEY=tally_sk_live_... tally doctor -- uvicorn main:app
tally doctor --fields          # this list, from the installed SDK
```

These are the only fields that leave your app. No prompt or completion text is sent, and your
customers' ids leave only as an HMAC-SHA256 hash under your organization's key.

<!-- tally-doctor-fields:start -->
| Sent for each model call | What it is |
| --- | --- |
| `gen_ai.system` | provider |
| `gen_ai.operation.name` | operation (chat, embeddings) |
| `gen_ai.request.model` | model the app asked for |
| `gen_ai.response.model` | model that answered |
| `gen_ai.usage.input_tokens` | input tokens |
| `gen_ai.usage.output_tokens` | output tokens |
| `gen_ai.usage.cached_input_tokens` | cached input tokens |
| `gen_ai.cost.estimated_micro_usd` | estimated cost, in millionths of a dollar |
| `gen_ai.cost.currency` | currency of the cost |
| `gen_ai.cost.price_catalog_version` | price catalog version the cost came from |
| `gen_ai.account_id_hash` | your customer's id, HMAC-SHA256 under your tenant's key (never the id itself) |
| `gen_ai.account_id_hash_key_version` | version of the key that made the hash |
| `gen_ai.account_label` | your customer's display label, only when labels are on |
| `gen_ai.feature_tag` | feature name |
| `gen_ai.session_id` | session id, only when the app sets one |
| `trace_id` | trace id |
| `gen_ai.trace_id_synthetic` | true when the SDK made the trace id itself |
| `span_id` | span id |
| `timestamp_ns` | when the call was recorded, nanoseconds since 1970 |
| `gen_ai.integration.name` | what recorded the call (tally-sdk) |
| `gen_ai.integration.version` | SDK version |
| `tally.replay.client_sampled` | true when local eval capture made this call's sampling decision |

| Sent once per batch | What it is |
| --- | --- |
| `tenant_id` | always empty: the ingest key names the tenant |
| `sdk_version` | SDK version |
| `batch_id` | batch id |
| `client_send_ts_ns` | when the batch was sent, nanoseconds since 1970 |
| `sampling` | sampling settings |
| `client_health` | counts (dropped spans, internal errors, buffer high water, unattributed accounts), the process kind, and each hook's library, version and state |
| `resource_spans` | the calls, each with the fields above |
| `business_events` | empty: nothing under tally run sends one |
| `identity_links` | empty: nothing under tally run sends one |
<!-- tally-doctor-fields:end -->

## LiteLLM

If you call models through LiteLLM, install the extra and add the AI Tally logger:

```bash
pip install "ai-tally[litellm]"
```

```python
import litellm
import aitally
from aitally.integrations.litellm import TallyLiteLLMLogger

aitally.init("tally_sk_live_...", instrument=False)
litellm.callbacks = [TallyLiteLLMLogger()]

litellm.completion(
    model="gemini/gemini-2.5-flash",
    messages=[...],
    litellm_metadata={
        "tally_account": customer_id,          # hashed before it leaves your app
        "tally_feature": "title_search.main",  # shows as the feature
        "tally_session": session_id,           # optional
        "tally_account_label": "Northwind",    # optional, readable name
    },
)
```

- **Use `instrument=False`.** LiteLLM sometimes uses the `openai` library, so without it a call
  could be counted twice. With it, direct provider calls outside LiteLLM aren't recorded.
- **Use `litellm_metadata`, not `metadata`.** LiteLLM sends `metadata` on to the provider in some
  cases. It never sends `litellm_metadata`.
- **Every call needs `tally_account`.** Without it the cost counts toward your total but not toward
  any customer. `with_account` doesn't apply to LiteLLM calls.
- **Streams** need `stream_options={"include_usage": True}`, except for Anthropic and Gemini.
  Without it the cost is blank.
- **Shut down before you exit.** A normal script is fine. In an async script, await `ashutdown()`
  before `asyncio.run(main())` returns, or the last calls are lost:

  ```python
  from aitally.integrations.litellm import ashutdown, shutdown

  shutdown(timeout_s=5)         # batch jobs, after your event loop has finished
  await ashutdown(timeout_s=5)  # inside a running loop, such as a server's shutdown hook
  ```
- **Supported providers.** Calls that aren't streamed get token counts only for OpenAI,
  Anthropic, Gemini (the `gemini/` route, not Vertex) and Mistral. Non-streamed calls to other
  providers show a blank cost.
- The extra installs LiteLLM 1.105.1, the version the logger was tested with. On other versions it
  still runs, but token counts come out blank.

## Keeping replay samples on your own server

To try other models on your real traffic without sending your prompts to AI Tally, the SDK can save
a small sample of calls on your own server:

```python
aitally.init(replay_capture="local", replay_dir="/var/lib/tally/replay")
```

It saves only while an admin has turned on local capture for your organization. By default it
saves 1 in 100 calls that aren't streamed (`replay_sample_rate`) and keeps them for 7 days
(`replay_retention_days`). Nothing is sent to AI Tally. **The files hold real prompts and
answers**, so treat that folder like any other customer data.

The SDK only deletes its own sample files, in a folder it marked as its own, and only by
retention (7 days unless `replay_retention_days` says otherwise) or `tally eval purge`. If an admin
turns local capture off, the SDK stops saving new samples and logs a warning naming the folder, but
it never deletes files on your server because of that switch. To delete them yourself:

```bash
tally eval purge --samples /var/lib/tally/replay                      # everything
tally eval purge --samples /var/lib/tally/replay --before 2026-10-01  # older days only
```

Or from code: `aitally.purge_local_samples(replay_dir, before=...)`.

### Comparing models on those samples

```bash
pip install openai anthropic   # the providers you want to test
tally eval --samples /var/lib/tally/replay --models gpt-4o-mini,claude-haiku-5-5 --max-spend-usd 5
```

Each sample is run again on each model you list, using your own provider keys
(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`). A judge model then compares each answer with the one your
app actually gave. You get one row per model with its cost, the judge's cost, how often it won
(with a 95% range), how often it refused, and how many samples were scored, skipped or failed.
Skipped and failed samples never count as losses.

The models you compare, and the judge, must be OpenAI or Anthropic models. Only samples from your
OpenAI and Anthropic calls are used. Samples that used tools or images are skipped.

- `--max-spend-usd` is a hard limit on what the run can spend. The command shows the worst case
  and asks before it starts. `--yes` skips the question.
- `--judge` picks the judge model. Otherwise it uses your organization's judge.
- Answers are capped at 1024 tokens (`--max-output-tokens`, `--judge-max-output-tokens`). An
  answer cut off at the cap counts as failed.
- `--format json` also checks that each answer is valid JSON. `--json` prints the results as JSON.

`tally eval` only runs while local capture is on. If your organization has turned it off, it
refuses and leaves the samples where they are.
`aitally.run_local_eval(...)` does the same from code.

## Setup help for your coding agent (MCP)

`aitally.init()` records model calls by itself. Vector search, tool calls and cost per customer
need a few lines in your code. The onboarding MCP server gives your coding agent (Claude Code,
Cursor) the right lines for your app. It runs on your machine, your code never leaves it, and it
only suggests code. It never edits your files.

```bash
pip install "ai-tally[mcp]"
```

This installs the `tally-onboarding-mcp` command. Give it your ingest key as `TALLY_KEY`. Only the
`coverage_report` tool uses the key. The others work offline.

### Claude Code

```bash
claude mcp add ai-tally-onboarding --env TALLY_KEY=tally_sk_... -- tally-onboarding-mcp
```

To share it with your team, add it to `.mcp.json` in your project. `${TALLY_KEY}` is read from each
person's own environment, so the key stays out of the file:

```json
{
  "mcpServers": {
    "ai-tally-onboarding": {
      "command": "tally-onboarding-mcp",
      "args": [],
      "env": { "TALLY_KEY": "${TALLY_KEY}" }
    }
  }
}
```

### Cursor

Add this to `~/.cursor/mcp.json`. A project `.cursor/mcp.json` works too, but never commit a file
that holds your key:

```json
{
  "mcpServers": {
    "ai-tally-onboarding": {
      "command": "tally-onboarding-mcp",
      "args": [],
      "env": { "TALLY_KEY": "tally_sk_..." }
    }
  }
}
```

If your agent can't find `tally-onboarding-mcp`, give the full path instead, for example
`"command": "/path/to/.venv/bin/tally-onboarding-mcp"`.

### The tools

| Tool | What it does |
|---|---|
| `detect_stack` | takes the contents of your `requirements.txt`, `pyproject.toml` or `package.json` and lists the providers, frameworks and vector databases it finds, what is already recorded, and what it has no recipe for |
| `get_recipe` | returns one setup recipe, by name (for example `pinecone` or `fastapi`) |
| `generate_startup` | writes the `aitally.init()` line for your app |
| `generate_middleware` | writes the middleware that sets the customer and feature, using the header or lookup you tell it |
| `instrument_call_site` | writes the `record_*` line for one call in your code |
| `explain_layer` | explains which `record_*` call covers a kind of cost, and why |
| `coverage_report` | asks AI Tally which kinds of cost are arriving, and why any are missing |

The server never guesses. It won't pick a customer id for you: tell it how your app knows the
customer, or the customer part stays empty. Anything it can't fill in from your code is left as a
visible `<FILL:...>` for you to complete. It also won't suggest a `record_llm_call` for a call
`init()` already records, since that would count the cost twice.

#### `coverage_report` settings

`coverage_report` is the only tool that connects to anything. It asks AI Tally which kinds of cost
are arriving for your organization.

| Variable | What it is |
|---|---|
| `TALLY_KEY` | your ingest key. The report only ever covers your own organization |
| `TALLY_GATEWAY_URL` | optional. The address to ask. Defaults to the same address the SDK uses. Must be `https://` unless it's your own machine |
| `TALLY_MCP_TENANT_ID` | optional. Your organization id (shown on the dashboard's connector setup). If set, a key for a different organization is refused |

The tool reads the key when it needs it and never saves, shows or logs it. If AI Tally can't be
reached, every kind of cost shows as `unknown` with the reason. A kind of cost only shows as
covered when real data has arrived. Pass `wired` with the kinds you just set up and the report
says "waiting for the first event" instead of "not set up".

### Node and TypeScript apps

The SDK is Python, but the MCP server also helps with Node apps. Give `detect_stack` your
`package.json` and it suggests setups that need no SDK:

| If your app uses | It sets up |
|---|---|
| `openai`, `@ai-sdk/openai` | the AI Tally proxy: `baseURL: "<ingest>/openai/v1"` and an `X-Tenant-Key` header holding your ingest key |
| `@anthropic-ai/sdk`, `@ai-sdk/anthropic` | the proxy: `baseURL: "<ingest>/anthropic"` (`<ingest>/anthropic/v1` for `@ai-sdk/anthropic`) and the same header |
| `@opentelemetry/*`, `@traceloop/*` | an OpenTelemetry exporter sending to `<ingest>/v1/otlp/traces` |
| any of the above | cost per customer, with a hashed customer id in the `X-Tally-Account-Id-Hash` header |

`<ingest>` is `https://ingest.ai-tally.com`. The proxy passes your provider key on to the provider
and doesn't store it, but the key does pass through AI Tally. OpenTelemetry never sends it. An
admin has to turn the proxy on first, under Settings > API keys. Until then it answers `403`.

As in Python, the customer id is hashed in your app and only the hash is sent. Use one setup per
call: if a call goes through the proxy and OpenTelemetry, it's counted twice, and `detect_stack`
warns you. There's no Node recipe yet for Gemini, Bedrock, Mistral, Cohere, Groq or Node vector
databases. `detect_stack` lists them as gaps.

## License

MIT. See `LICENSE`.
