Metadata-Version: 2.5
Name: callwitness
Version: 0.1.0
Summary: Record every tool call an AI agent makes. Block nothing.
Project-URL: Homepage, https://github.com/AditiChaudharyy14/callwitness
Project-URL: Source, https://github.com/AditiChaudharyy14/callwitness
Author: Aditi Chaudhary
License: MIT
License-File: LICENSE
Keywords: agents,ai,ai-safety,mcp,observability,security
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.8
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# Callwitness

**Record every tool call an AI agent makes. Block nothing.**

[![tests](https://github.com/AditiChaudharyy14/callwitness/actions/workflows/tests.yml/badge.svg)](https://github.com/AditiChaudharyy14/callwitness/actions/workflows/tests.yml)

A transparent MCP proxy. It sits between an agent and its tools, forwards every
byte unchanged, and writes down what happened.

No dependencies. Python 3.8+. MIT.

---

## The thing it shows you

Same tool. Same permission. Two very different actions:

```
      51B  send_email   ops@acme.com
    20085B send_email   exfil.example.net, drop@unknown.example
```

An allowlist cannot tell those apart — the agent is permitted to send email in
both cases. The difference is *how much* is leaving and *where it is going*, and
those are the two signals Callwitness records on every call.

## Why it blocks nothing

Because it should be installable in production on a Tuesday afternoon.

Callwitness cannot corrupt what an agent sends or receives: it relays every message
whether or not it can parse it, and every write to storage is wrapped so a
recorder bug can't reach the stream. That property is tested, not asserted —
see `tests/test_passthrough.py`, which asserts the proxied output is
byte-identical to running the server directly.

**It cannot delay, either.** Observation runs on its own thread behind a bounded
queue, so the relay only ever does a non-blocking hand-off. A deliberately
half-second-slow observer moves the gap between two forwarded messages by 0.05ms
— it used to move it by 4.2 seconds. The queue drops rather than growing without
limit under load, and counts what it dropped: unrecorded data nobody can see is
worse than data that was never collected.

It matters because the security industry is currently writing rules against
agent failures nobody has measured. Enforcement without data is guessing with
extra steps. Collect first.

## Install

```bash
pip install -e .
```

## Use

Wrap any stdio MCP server:

```bash
callwitness run --echo -- npx -y @modelcontextprotocol/server-filesystem /data
```

Or let it wrap the servers you already have. It finds your client's config,
shows you exactly what would change, and writes nothing until you say so:

```bash
$ callwitness install

/Users/you/Library/Application Support/Claude/claude_desktop_config.json
  filesystem
    - npx -y @modelcontextprotocol/server-filesystem /data
    + callwitness run --label filesystem -- npx -y @modelcontextprotocol/server-filesystem /data
  git
    - uvx mcp-server-git --repository /repo
    + callwitness run --label git -- uvx mcp-server-git --repository /repo
  remote-api  SKIPPED: remote server -- needs `callwitness proxy --upstream
              https://mcp.acme.com/mcp --port <port>` and a port you choose

2 servers would be wrapped. Nothing has been changed.
```

`--apply` writes it, after a timestamped backup. `callwitness uninstall --apply`
puts everything back. Running install twice does nothing the second time.

Dry-run is the default because this edits a file you did not write and a broken
MCP config means a broken agent — the one outcome this whole tool promises not
to cause. Anything it does not recognise is skipped and named rather than
guessed at.

Knows about Claude Desktop, Cursor, Windsurf, Claude Code, and project-local
`.mcp.json` / `.vscode/mcp.json`. If yours lives elsewhere:
`callwitness install --config /path/to/mcp.json`.

### Remote servers

Production agents mostly talk to remote MCP servers over Streamable HTTP. Put
Callwitness in front of one and point the client at the local address instead:

```bash
callwitness proxy --upstream https://mcp.example.com/mcp --port 8100 --echo
```

```json
{
  "mcpServers": {
    "example": { "url": "http://127.0.0.1:8100/mcp" }
  }
}
```

POST, the SSE response stream, the server-initiated `GET` stream and session
teardown are all relayed verbatim, headers included, so the `Mcp-Session-Id`
handshake works without Callwitness understanding it. Both transports share one
recorder (`CallTracker`), so a row looks the same whichever produced it.

The agent behaves exactly as before. Then look at what it did:

```bash
callwitness stats            # per-tool volume, errors, latency, destinations
callwitness tail -n 20       # the most recent calls
callwitness verify           # check nothing has been altered since it was written
callwitness export out.jsonl # everything, for analysis
```

### Then let it write the rules

The next tier was going to be YAML you write by hand. But a person typing
`max_payload: 8KB` for `send_email` is guessing at a number they have no way to
know — which is the thing this project says the industry is doing wrong.
Enforcement without data is guessing with extra steps, and a rule language is
not data. So the rules come out of the observation tier instead:

```
$ callwitness suggest --since 14d

  send_email     max_payload            3.5KB    # p99 observed 2.3KB over n=400; 1.5x headroom
  send_email     destinations_emails    3 allowed # 3 distinct emails covering 100% of traffic over n=400
  send_email     rate_limit_per_hour    21       # 10.0/hour average over 39.9 hours; 2x headroom
? fetch_url      destinations_hosts     --       # 99 distinct hosts across 150 calls -- too varied for an
                                                 #   allowlist; this reads as a general-purpose fetcher
? delete_record  insufficient_data      --       # only 6 calls observed; 30 needed before a threshold
                                                 #   means anything
```

`--format yaml` emits the same thing as a policy draft, every rule commented
with the evidence it rests on.

Note what it refuses to do. A tool below 30 calls gets no threshold, because a
p99 over n=6 is an anecdote. A tool whose destinations are too varied is flagged
for a human rather than handed an allowlist that would fire constantly. And a
destination that was never seen is not a destination that is forbidden — it may
simply not have happened yet, and the output says so rather than letting you
forget it. Every line is a hypothesis with its evidence attached, not a finding.

**Baseline poisoning.** If the bad thing already happened while Callwitness was
watching, it is in the distribution, and a plain percentile quietly raises the
ceiling to permit it. The demo above showed exactly that: a 29KB exfiltration
produced a 43KB proposed ceiling — one that would have allowed the very call
this tool exists to catch.

So ceilings come from the bulk of a distribution, not all of it. Calls far above
the median set no limit; they are named, with timestamps and destinations, and
handed to a person:

```
  send_email  max_payload   4.3KB  # p99 of the bulk is 2.9KB; 1.5x headroom.
                                   #   EXCLUDES 1 call above 11.4KB
! send_email  tail_review   1      # 1 call more than 8x the 1.4KB median. A rare
                                   #   enormous call is the most interesting thing
                                   #   here, so it sets no limit until you have
                                   #   looked at it: 28.8KB at 2026-09-09T18:09
                                   #   -> exfil.example.net
```

The reference is the median, because it is the one statistic a single enormous
call cannot move — which is the point when that call may be the attack. If more
than 10% of traffic sits above the threshold it is not a tail, it is the shape,
and nothing is excluded; misdescribing the distribution is a different failure,
and just as wrong.

This is not a solution to baseline poisoning. Nothing that learns from unlabelled
traffic has one. It is a refusal to hide it.

### Try it without an agent

No agent, no API key, no network, no Node:

```bash
python examples/demo.py                     # throwaway run, nothing kept
python examples/demo.py --keep --repeat 40  # record into your own store
callwitness suggest                             # then let it propose rules
```

The plain run uses a temporary directory so trying the tool doesn't pollute
anyone's data — but the obvious next thing to type is `callwitness stats`, and
"No data yet" is a bad first hour. `--keep` records into `~/.callwitness`, and
`--repeat` sends enough varied traffic that `suggest` has a distribution to
work from rather than an anecdote.

## Privacy

| Flag | Effect |
|---|---|
| *(default)* | Credentials in argument values are redacted before storage |
| `--no-redact` | Stores argument values verbatim, credentials included |
| `--no-args` | Stores argument *shape* only (`{"to": "<str:20>"}`), never values |
| `--max-arg-bytes N` | Caps stored bytes; the true size is still recorded |
| `--home DIR` | Where data lives (default `~/.callwitness`) |

**Redaction is on by default.** Tool arguments routinely carry API keys, bearer
tokens and connection strings, and without this every install would be a
plaintext credential store that didn't exist before Callwitness was installed. Known
key formats, credentials inside URLs, sensitively-named parameters and
high-entropy tokens are replaced with `<redacted:reason>` on the write path —
never on read, because by then the plaintext is already on disk. The true
pre-redaction byte count is still recorded, so the volume signal survives.

Destinations survive redaction on purpose: `postgres://admin:hunter2@db.internal`
stores as `postgres://admin:<redacted:url_password>@db.internal`. The host is the
signal; the password is not.

`--no-args` still records destinations — hosts, emails, IPs — because
destinations are the signal. That's deliberate, it's tested, and you should say
it out loud to anyone you ask to run this.

Everything stays on the machine that ran it. Nothing is transmitted anywhere.

### Evidence, not just a log

An append-only file is trivially editable by anyone with filesystem access —
including a compromised agent running as the same user. A record that can be
silently rewritten is a convenience, not evidence.

So every call commits to the one before it. Editing, deleting, reordering or
inserting a record breaks the chain from that point, and `callwitness verify` says
where:

```
$ callwitness verify
BROKEN  filesystem  642 records, breaks at seq 118
                    content does not match its hash: this record was edited
                    after it was written
```

Exit code 1 on a break, so it works in a cron job without anyone parsing text.

**It is tamper-evident, not tamper-proof, and the tool says so out loud.**
Someone who can write to the file can also recompute every hash after a change
and produce a chain that verifies — nothing local can stop that, because the
verifier and the attacker read the same file. What defeats it is an anchor the
operator does not control, so `verify` prints the head hash and tells you to
store it somewhere the machine cannot reach. That is a deployment decision, and
inventing one for you would be worse than naming the gap.

Records written before chaining existed are reported as predating it, not as
tampering. A verifier that cries wolf on an upgraded install is worse than no
verifier.

## What gets stored

`calls` — one row per tool call: tool, arguments, `args_bytes`, whether they
were truncated, `signals`, `duration_ms`, `is_error`, `result_bytes`, and a
result preview.

`signals` splits two things a naive scan conflates:

```json
{
  "destinations":   {"emails": ["archive@unknown-host.example"],
                     "hosts":  ["exfil.example.net"]},
  "content_counts": {"emails": 400}
}
```

**Destinations** are entities found in routing fields — `to`, `url`, `webhook`,
`attach_url` and so on. **Content counts** are how many entities appear in the
payload. Scanning the whole blob for email addresses would report four hundred
customer emails from inside a message body as "destinations" and bury the one
address the message is actually addressed to. These are different signals and
they compose: *29KB addressed to an unknown host, containing 400 email
addresses* is a shape worth stopping. *29KB containing 400 addresses, sent to
the CRM you always use* is a Tuesday.

`events` — `initialize` and `tools/list`, so you know which tools were exposed.

`sessions` — one row per wrapped process, with the exit code.

SQLite at `~/.callwitness/callwitness.db`, plus an append-only `calls.jsonl`.

## Design rule

The recorder must never corrupt the protocol stream, and must never delay it.
Every message is forwarded first, then handed to a background queue; parsing
happens on another thread, inside a `try`. If recording throws, traffic still
flows. If recording is slow, traffic still moves.

If you contribute, keep it that way. `test_a_broken_recorder_never_raises` and
`tests/test_hardening.py` are there to make sure you do.

## The experiment

`experiments/` runs the measurement this tool exists to make possible: 15 tasks
x 4 injection channels, an agent with real tools and real side effects, all
traffic recorded through Callwitness itself.

```bash
python experiments/run.py --driver scripted --out runs/pilot --fresh --repeats 4
python experiments/analyze_runs.py runs/pilot
```

That validates the pipeline with no API key and no network. Swap
`--driver llm` with a Groq free-tier key for the real thing. Protocol and
design are in [experiments/README.md](experiments/README.md).

## Where this is going

1. **Now** — observe. Record every call, block nothing.
2. **Next** — deterministic policy: the rules `callwitness suggest` proposes,
   evaluated inline, sub-millisecond, fail-open by default. The generator ships
   first on purpose; an engine that enforces numbers nobody could justify is the
   problem, not the product.
3. **Then** — context: an LLM judge, but only on calls the deterministic tier
   flags. Payload volume × destination reputation first.

Scope: MCP tool calls over stdio and Streamable HTTP. The deprecated
two-endpoint HTTP+SSE transport is not covered. Direct API calls made inside
agent code need an SDK wrapper, and that is deliberately not in v1.

## Where it came from

Out of an experiment on chain-of-thought faithfulness
([cot-hint-verbalization](https://github.com/AditiChaudharyy14/cot-hint-verbalization)),
which turned up a measurement problem: "hint verbalisation rate" reads 100% on
the reasoning trace and 12% on the user-facing answer, for the same responses.
Same data, same model, an order of magnitude apart depending only on where you
look.

A field whose headline metric moves by 10x depending on the instrument does not
need another opinion about agent risk. It needs somebody to start writing down
what actually happens.

## Tests

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT
