Metadata-Version: 2.4
Name: agentdevlabs
Version: 0.2.0
Summary: Capture and replay LLM calls from the OpenAI and Anthropic Python SDKs.
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: cryptography>=46.0.3
Requires-Dist: httpx2<3,>=2.16.0
Requires-Dist: rfc8785<0.2,>=0.1.4
Provides-Extra: anthropic
Requires-Dist: anthropic<3,>=2; extra == 'anthropic'
Provides-Extra: openai
Requires-Dist: openai<4,>=3; extra == 'openai'
Description-Content-Type: text/markdown

# agentdevlabs

Python SDK for Agent Dev Labs Capture & Replay. `adl(client)` records the calls your app makes through an OpenAI or
Anthropic client. Your provider key stays with your app.

```sh
pip install agentdevlabs
```

```py
import openai
from agentdevlabs import adl

client = adl(openai.OpenAI())
```

## Set up with a coding agent

1. Check that `AGENT_DEV_LABS_TOKEN` is set in `.env`. If it isn't, stop and ask the user for it. Never create a token or write one into code, and never commit `.env`. The SDK reads the process environment, not `.env` itself, so make sure the app loads it. In Python call `load_dotenv()` from python-dotenv before wrapping the client, or export the variable.
2. Install `agentdevlabs` and the provider SDK (`openai` or `anthropic`) with the project's own package manager.
3. Wrap the provider client where it is created: `adl(openai.OpenAI())` or `adl(anthropic.Anthropic())`. Leave `AGENT_DEV_LABS_MODE` unset unless the user asks, because it outranks the devkit toggle.
4. Run the app twice with `AGENT_DEV_LABS_VERBOSE=1`, letting each run exit on its own (a killed run loses its upload). Every call prints a line starting `[agentdevlabs]`: run 1 should print `captured` and run 2 `replayed`. Unset the setting after.
   - `passed-through` or `failed`: the reason follows the word; read it.
   - `captured` on run 2: read the SDK's warnings, and look for a prompt that changes between runs, such as a date or an id.
   - No line: the call wasn't hooked; read the SDK's warning.
   - Ask the user to look in the console's Traces and Calls only if that doesn't settle it, since you can't sign in.

Settings are environment variables only:

| Variable | Meaning |
| --- | --- |
| `AGENT_DEV_LABS_TOKEN` | Your workspace token. Without it the SDK warns once and records nothing: each call passes through, and its report says `passed-through`, `off` and `disabled`. |
| `AGENT_DEV_LABS_MODE` | `auto`, `capture`, `replay` or `off`, in any case. Overrides the server's mode, except a server `off` for production or the plan limit, which always wins. Any other value runs as `auto`, with one warning naming it. |
| `AGENT_DEV_LABS_DISABLED` | `1`, `true`, `yes` or `on` turns the SDK off; `0`, `false`, `no`, `off` or empty leaves it on. Any other value turns it off, with one warning naming it. Beats every other setting. |
| `AGENT_DEV_LABS_VERBOSE` | `1`, `true`, `yes` or `on` prints one line per call to standard error: `[agentdevlabs] captured`, `replayed`, `passed-through` or `failed`, the last two with their reason. `0`, `false`, `no`, `off` or empty leaves it off; any other value leaves it off, with one warning naming it. Sends nothing. |
| `AGENT_DEV_LABS_API_URL` | The API to talk to. |
| `AGENT_DEV_LABS_REPLAY_SPEED` | How fast a replayed stream plays: `realtime` (the default), `fast` (10x), `instant`, or a number to divide the recorded pace by. Non-streamed replies are instant unless this is set. |
| `AGENT_DEV_LABS_GROUP` | Names a trace group: calls are captured into it and replays use only its recordings. Letters, digits, `.`, `_`, `/` and `-`, up to 100. |
| `AGENT_DEV_LABS_CACHE_DIR` | Where replayed captures are cached, encrypted. Defaults to your OS cache directory. |

## Make one trace from a block

Calls inside `with trace(name):` (or a function decorated with `@trace(name)`) make one trace. The trace ends when its last call finishes, which can be after the block returns.

```py
from agentdevlabs import trace

with trace("checkout"):
    client.chat.completions.create(...)

@trace("summarise")
def summarise(text): ...
```

## See what each call did

`adl(client, on_call=...)` calls your listener once per provider call with the mode in effect, where it came from, and what happened. A request that raises gets no report: your app gets the error. It sends nothing anywhere.

```py
client = adl(
    openai.OpenAI(),
    on_call=lambda report: print(report.outcome, report.mode, report.mode_from),
)
```

| Field | Meaning |
| --- | --- |
| `outcome` | `replayed` (a stored capture answered), `captured` (the provider answered and the call is recorded), `passed-through` (the provider answered and nothing was recorded) or `failed` (strict replay had no capture, so the provider was not called). |
| `mode` | The mode this call ran in: `auto`, `capture`, `replay` or `off`. |
| `mode_from` | `env` (`AGENT_DEV_LABS_MODE`), `server` (the console or devkit toggle, the environment's default, or the token), `default` (the server could not be reached or rejected the token) or `disabled` (`AGENT_DEV_LABS_DISABLED` or no token). |
| `pin` | `pin.trace_id` and `pin.set_at` when the environment's pinned trace answered the call, else `None`. |
| `trace_id`, `step`, `cache_key` | Which capture the call matched or made. |
| `reason` | Why a call failed or went unrecorded. |

The listener runs on the calling thread, or the calling event loop for an async client. A listener that raises is ignored: the call goes on and the SDK warns once. A report holds ids and reasons, never a prompt, a response or a key. A provider retry is its own call, and an attempt about to be retried gets no report.

## Replay in CI

Record a group on your machine, then replay only that group in CI. A call with no recording in the group fails with an error naming the cache key and the group (or saying the group doesn't exist), so CI never reaches the provider.

The group name uses letters, digits, `.`, `_`, `/` and `-`, up to 100 characters; anything else fails at startup. `AGENT_DEV_LABS_TOKEN` in CI must be a `replay-only` service token, which an owner or admin creates in the console's Access tokens page.

```sh
AGENT_DEV_LABS_GROUP=checkout-flow AGENT_DEV_LABS_MODE=capture pytest   # once, locally
```

```yaml
# .github/workflows/test.yml
name: test
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv sync
      - run: uv run pytest
        env:
          AGENT_DEV_LABS_TOKEN: ${{ secrets.AGENT_DEV_LABS_TOKEN }}
          AGENT_DEV_LABS_GROUP: checkout-flow
          AGENT_DEV_LABS_MODE: replay
```

## Project rules: a worked example

A travel agent's system prompt carries today's date, a session UUID and a request id, so Monday's recording never matches on Tuesday. In the console's Project rules page, three masks make the calls match:

- `[0-9]{4}-[0-9]{2}-[0-9]{2}` masks the date.
- `[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}` masks the session UUID.
- `req_[A-Za-z0-9]+` masks the request id.

A mask changes only what matching reads, so the recording still holds the original text. To see which mask a pair of calls needs, open a trace, choose "Why didn't this replay?" on step 1 and "Try this rule" on a suggestion.

A card number the user typed should never be uploaded, so add a redact pattern such as `[0-9]{4} [0-9]{4} [0-9]{4} [0-9]{4}`. The SDK replaces it with `[redacted]` before upload; matching still uses the original, so redacting doesn't stop replays. The page's "Try it" box shows both results on a sample request that never leaves the browser.

Calls already recorded keep the rules they were captured with: after saving a rule, capture them again.

## License and privacy

Apache-2.0. The SDK sends nothing beyond the capture and replay events described above: no usage analytics or telemetry.

## Versions

Versions follow semantic versioning. Before 1.0, a minor release (0.1 to 0.2) may break your code; the changelog says how, and the API supports only the latest 0.x minor. From 1.0, the API supports the previous major version for 12 months after a new major ships. See the [changelog](https://agentdevlabs.com/docs/python-changelog.md).
