Metadata-Version: 2.5
Name: weharness-tracing
Version: 0.1.4
Summary: Send your AI agent's traces to WeHarness (OpenTelemetry; metadata only unless you choose otherwise).
Project-URL: Homepage, https://weharness.io
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-agents,llm,observability,opentelemetry,tracing,weharness
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: opentelemetry-api==1.45.1
Requires-Dist: opentelemetry-exporter-otlp-proto-http==1.45.1
Requires-Dist: opentelemetry-sdk==1.45.1
Provides-Extra: acp
Requires-Dist: agent-client-protocol<0.13,>=0.12.1; extra == 'acp'
Description-Content-Type: text/markdown

# WeHarness tracing for Python

Send agent, model and tool spans to WeHarness using OpenTelemetry and OTLP/HTTP.
Requires Python 3.10 or newer and a project ingest key from the WeHarness Tracing page.

```python
import weharness_tracing as wh

# Set WEHARNESS_INGEST_KEY in your server environment, never in browser code.
wh.init(service="my-agent")
with wh.agent_run("support"):
    with wh.tool_call("lookup_order"):
        pass  # Call your tool here.
wh.shutdown()
```

The default endpoint is `https://api.weharness.io/otlp/v1/traces`.
Override it with `WEHARNESS_OTLP_ENDPOINT` if needed. Set `WEHARNESS_REVISION`
to correlate traces with a source revision.

Known prompt, output, tool-payload and stacktrace attributes are filtered by default.
This filter is not a general secret detector: do not put secrets or personal data in
span names, error messages or custom metadata. Content capture requires explicit
configuration and appropriate project settings. An ingest key sends traces; it does
not authorize reading them.

An existing OpenTelemetry tracer provider is reused. Background batches are bounded;
call `shutdown()` to flush pending spans before the process exits.

## Local evaluation runner

The Python package includes `weharness-run`. Save the cases returned by the
WeHarness MCP `evaluate_local_start` operation as `cases.json`, then run your
agent in the shape it has:

```sh
weharness-run --cases cases.json --command "python agent.py" --timeout 300 --out report.json
weharness-run --cases cases.json --function my_agent:graph --out report.json
weharness-run --cases cases.json --url http://127.0.0.1:8000/invoke \
  --start "uvicorn app:app --port 8000" --out report.json
weharness-run --cases cases.json --acp "opencode acp" --out report.json
```

- `--command` runs a CLI entry point. Each case input goes to stdin, or
  replaces `{input}` in the command. The output is what it prints.
- `--function module:attr` (or `path/to/agent.py:attr`) imports your code from
  the working directory and calls it in a separate Python process. It accepts
  plain and async functions, generators, objects with `invoke`, `ainvoke` or
  `run` (LangGraph graphs, LangChain runnables, runner objects), OpenAI Agents
  SDK `Agent` objects (run with `Runner.run`), and classes that take no
  arguments. A case input that is a JSON object is passed parsed, a LangGraph
  graph receives `{"messages": [{"role": "user", "content": ...}]}`, anything
  else gets the text; `--input-as text|json|messages` overrides this. The output
  is the return value: a string as is, an OpenAI Agents SDK result's
  `final_output`, a LangGraph state's last message content, anything else as
  JSON text. What the function prints is not part of the output. `--python`
  chooses the interpreter (default: the active virtualenv's, else the one
  running `weharness-run`).
- `--url` POSTs `{"input": "<case input>"}` to the endpoint (change it with
  `--body '{"message": {input}}'`, add headers with `--header`) and uses the
  reply as the output: JSON as JSON text, anything else as text. A non-2xx
  reply gives exit code 1 and `http_status`. `--start` starts the server under
  the runner, waits until it accepts connections, runs the attempts one at a
  time and stops it afterwards. Without `--start` the runner cannot observe or
  isolate the server's own calls (`effects` is `null`).
- `--acp "<command>"` runs an agent that speaks the Agent Client Protocol over
  stdio, such as `opencode acp`. Install the extra first:
  `pip install "weharness-tracing[acp]"` (the official ACP Python SDK). Each
  attempt starts the command, opens one session in the working directory and
  sends the case input as the prompt. The output is the agent's reply text, and
  the attempt adds `acp`: the stop reason, the tool calls, the usage and cost
  the agent reported, and each permission decision. The agent gets no file
  system or terminal capability from the runner and uses its own tools. Nobody
  answers permission requests during a run: a request that names a path outside
  the working directory is rejected, and any other is allowed. The agent keeps
  its own MCP configuration and model. It is not a Python process, so the side
  effect interceptor does not cover it (see below). `--python` chooses the
  interpreter that has the ACP SDK (default: the one running `weharness-run`).

For each repetition (`--repeats`, default what WeHarness asked, else 3;
`--parallel N` runs N attempts at a time) the report records the output, exit
code, duration, the trace ids the agent sent and the side effects it attempted.
Each attempt's spans carry `eval.run.id`, `eval.case.id` and `eval.attempt`
resource attributes. Give the report to `evaluate_local_report` through your
coding host. The runner itself uploads nothing. Its exit code indicates report
generation, not that the agent passed the behavior checks; inspect the
evaluation verdict.

### Side effects

Isolation is on by default. Every Python process the agent starts (the command
and its Python children, the function's process, a server started with
`--start`) loads an interceptor for outbound HTTP through `http.client` and
`urllib.request`, `urllib3` and `requests`, and `httpx` (which the OpenAI,
Anthropic and Google SDKs use). Before anything is sent:

- Model providers pass: OpenAI, Anthropic, Gemini and Vertex AI, Azure OpenAI,
  Bedrock, Mistral, Cohere, Groq, DeepSeek, Together, Fireworks, xAI,
  OpenRouter, Perplexity, the WeHarness model proxy, the hosts in
  `OPENAI_BASE_URL`, `ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`,
  `AZURE_OPENAI_ENDPOINT` and `WEHARNESS_MODELS_URL`, and any `--model-host`.
- WeHarness trace endpoints and the endpoints in `WEHARNESS_OTLP_ENDPOINT`,
  `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` pass;
  the trace ids in those exports become the attempt's `trace_ids`.
- `localhost` and loopback addresses pass. Private network addresses do not.
- Any other host is answered from `--mocks`, or refused: the agent's HTTP
  client raises a connection error that names the call and how to allow it.
  `--allow-host api.example.com` (or `*.example.com`) lets real calls through.

A mocks file is a JSON list, or `{"mocks": [...]}`. The first match wins:

```json
[
  {"method": "POST", "url": "https://api.stripe.com/v1/refunds", "status": 200,
   "body": {"id": "re_123", "status": "succeeded"}},
  {"url": "api.crm.example/customers/*", "headers": {"X-Mock": "1"}, "body": {"name": "Ada"}}
]
```

In `url`, `*` matches any characters; a pattern matches the URL without its
query unless it contains `?`, and a pattern without a scheme matches both
`http` and `https`. `method` defaults to any, `status` to 200, and a body that
is not a string is sent as JSON.

Each attempt's `effects` lists what it tried, grouped with a count:

```json
{"method": "POST", "host": "api.stripe.com", "path": "/v1/refunds",
 "action": "mocked", "kind": "external", "status": 200, "count": 1}
```

`action` is `allowed`, `mocked`, `refused` or `replayed`; `kind` is `model`,
`trace`, `local`, `allowed_host` or `external`. Query strings are never
reported; paths are. `effects` is `null` when the runner could not observe the
agent: a non-Python agent, a server it did not start, or `--no-isolation`.

What this does not cover: raw sockets, other HTTP clients (aiohttp, pycurl,
gRPC), processes that are not Python (Node, shell tools, containers), and
database or queue drivers. Python tools the agent runs are covered too, so
install dependencies before the run. The interceptor prevents accidental side
effects during an evaluation; it is not a security sandbox, and the agent's
code can undo it. If it cannot load, the agent's Python process stops rather
than run without it.

### Record and replay

`--record DIR` stores each model call's request (without credential headers or
query keys) and response, per case and attempt, in
`DIR/<case_id>/attempt-<n>.jsonl`. `--replay DIR` answers identical model
requests (same method, URL and JSON body) from that recording, in recorded
order, without the network, and refuses a request the recording does not hold.
Trace exports, localhost and allowed hosts behave as in any run; mocks answer
as usual. Attempts carry `"recorded": true` or `"replayed": true`. Use replay to re-run a
failure without new model calls while you change the code around the model.
Recordings contain prompts and model outputs; keep them out of version control
unless you mean to share them.

### Credentials

Your agent's provider credentials remain available, and so does `WEHARNESS_INGEST_KEY`: it only
sends your project's traces, and your agent needs it so each attempt carries its traces (WeHarness
checks tool calls and links failures to them). WeHarness credentials named `WEHARNESS_TOKEN`,
`WEHARNESS_API_KEY`, `WEHARNESS_MCP_TOKEN`, and variables starting with `WEHARNESS_SECRET` are
removed from the child environment. This is a local process runner, not a security sandbox;
run only code you trust. On POSIX, timed-out attempts receive a process-group
termination; Windows currently terminates the direct process only.

### Model access during a run

By default your agent calls its model with your own provider key, read on your
machine and never sent to WeHarness. If you ask your coding host to use
WeHarness's models instead, `evaluate_local_start` takes a model and a credit cap,
reserves the cap in credits, and returns `model_access.env` once: a model token
for that run (`WEHARNESS_MODELS_TOKEN`, `WEHARNESS_MODELS_URL`,
`WEHARNESS_MODELS_ALLOWED`) and the chosen provider's own variables pointed at
WeHarness's model proxy (for example `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL`).
Set them in the environment you start `weharness-run` in; it passes them on to
your agent unchanged. Save the cases file without `model_access`, and keep the
token out of files you commit.

The token allows only that model, within the credit cap, and stops working when
the run closes (every attempt reported, the run failed, or left unreported) or
after 12 hours. The credits its calls used are then debited and the rest of the
cap is released. Your coding host's model can see the token.
