Metadata-Version: 2.5
Name: nookku
Version: 0.4.0
Summary: A test harness for developers of agent apps. Test your agent through Claude Code or Codex, byte for byte, and audit the record.
Project-URL: Repository, https://github.com/mohanraj00/nookku
Project-URL: Documentation, https://github.com/mohanraj00/nookku/blob/main/docs/index.md
Project-URL: Specification, https://github.com/mohanraj00/nookku/blob/main/SPEC.md
Project-URL: Changelog, https://github.com/mohanraj00/nookku/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/mohanraj00/nookku/issues
Author: Mohan Raj
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,audit,claude-code,codex,llm,testing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/mohanraj00/nookku/main/docs/assets/logo/nookku-wordmark-dark.svg">
  <img src="https://raw.githubusercontent.com/mohanraj00/nookku/main/docs/assets/logo/nookku-wordmark.svg" alt="Nookku" width="332">
</picture>

# Nookku

(நோக்கு - to observe)

A test harness for developers of agent apps.

[![CI](https://github.com/mohanraj00/nookku/actions/workflows/ci.yml/badge.svg)](https://github.com/mohanraj00/nookku/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/nookku)](https://pypi.org/project/nookku/)
[![Python versions](https://img.shields.io/pypi/pyversions/nookku)](https://pypi.org/project/nookku/)
[![License](https://img.shields.io/github/license/mohanraj00/nookku)](https://github.com/mohanraj00/nookku/blob/main/LICENSE)

**Test your chat agent through Claude Code or Codex. The [harness](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#harness) model does not retype one message or one reply.**

Nookku is a test harness for developers of agent apps. You talk to your agent in the coding harness where you already work. The [relay](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#relay) sends each message to your agent byte for byte, and shows each reply byte for byte. The model does not run while you talk. At the end, the model reads the exact record and evaluates your agent: business logic, tone, accuracy.

It adapts to your app through a thin [entry](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#entry) in `.nookku/`, and your app's code does not change. The cost is plumbing: you write the entry once, and you change it when the start or the wiring of your app changes.

## How it works

```text
tester ──> harness ──> relay ──────> tap ──stdin/stdout──> entry ──> your app
                         │            │
                         ▼            ▼
                    relay.jsonl    tap.jsonl
            (what you typed      (what the agent
             and saw)             received and sent)
                         │            │
                         └─> audit <──┘   exit 0 clean, 1 break, 2 invalid record
```

- [Harness](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#harness): Claude Code or Codex, where you type your test messages.
- [Relay](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#relay): the command `nookku hook`, which the plugin or the project hooks run in the harness. It carries each message and each reply, and the model writes neither.
- [Tap](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#tap): a proxy in front of your agent. It forwards each byte with no change, and records what the agent received and sent.
- [Entry](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#entry): a thin wrapper that starts your app and speaks one JSON line in and one JSON line out.
- [Audit](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#audit): a program that compares the two records byte for byte, and names each break.

[docs/architecture.md](https://github.com/mohanraj00/nookku/blob/main/docs/architecture.md#data-flow) shows this flow with each part, and has diagrams of one turn, the test lifecycle and the trace.

Two processes write two records, and the audit compares them byte for byte. It reports 8 break classes ([SPEC.md section 3.3](https://github.com/mohanraj00/nookku/blob/main/SPEC.md#33-break-classes)). It fails closed: it never reports clean on a record that it cannot read. 62 audit cases, 20 contract cases, 18 trace cases, 11 seal cases and 7 receiver cases in [conformance/](https://github.com/mohanraj00/nookku/blob/main/conformance/) test the spec ([cases](https://github.com/mohanraj00/nookku/blob/main/conformance/build.py), [tests](https://github.com/mohanraj00/nookku/blob/main/tests/)).

A test also records what your app did. It copies the session files of the app's own Agent SDK or Codex sessions. It records the backend calls, the direct model calls and the OpenTelemetry spans of the app. [docs/architecture.md](https://github.com/mohanraj00/nookku/blob/main/docs/architecture.md) shows each part.

## Install

```bash
uv tool install nookku
```

It needs Python 3.10 or later, and it has no runtime dependencies ([pyproject.toml](https://github.com/mohanraj00/nookku/blob/main/pyproject.toml)). The docs are for version 0.4.0. If you use verbatim-relay 0.3.x or earlier, do the [0.4.0 steps of the upgrade guide](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/upgrade.md#040), because the package has a new name ([CHANGELOG.md](https://github.com/mohanraj00/nookku/blob/main/CHANGELOG.md#040)). To upgrade from a later version of nookku, end each running test first. Then run `uv tool upgrade nookku`. [docs/how-to/upgrade.md](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/upgrade.md) gives each step.

## Quick start

Select one route: try the toy shop, connect your app, or choose a relay.

### Try the toy shop

In a clone of this repo, with the toy shop agent and the project hooks in Claude Code:

```bash
nookku init claude-code --entry "python3 examples/toy-shop/agent.py"
nookku check        # one message through the entry: PASS proves the connection, not the reply
nookku view         # in a second terminal: each reply shows here
```

In Claude Code, type the prompt `nookku start`, then your test messages, then `nookku end`. The end of the test writes `audit.json` in the [test folder](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md#test-folder) `.nookku/tests/<test-id>/`. Then the model evaluates the test and writes `report.md`. [docs/getting-started.md](https://github.com/mohanraj00/nookku/blob/main/docs/getting-started.md) shows each step with its real output.

Optional: to see the audit again, run it by hand:

```bash
nookku audit --tap .nookku/tests/<test-id>/tap.jsonl --relay .nookku/tests/<test-id>/relay.jsonl
```

### Connect your app

In the harness, in the project folder of your app, type `!nookku setup`. Then type the prompt "Follow the Nookku setup guide, and connect a test to this app." [docs/how-to/test-your-app.md](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/test-your-app.md) gives each step from the install to the results, with the expected output. To write the entry yourself, read [docs/how-to/connect-your-agent.md](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/connect-your-agent.md).

### Choose a relay

You install the [relay](https://github.com/mohanraj00/nookku/blob/main/SPEC.md#5-relays) as the plugin or as the project hooks. Both work in Claude Code and in Codex, with the same rules. Both show each reply as the reason of a blocked prompt, and in `nookku view`. [docs/how-to/choose-a-relay.md](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/choose-a-relay.md) compares them. Then read the guide of your setup: [Claude Code plugin](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/claude-code-plugin.md), [Claude Code project hooks](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/claude-code-hook-kit.md) or [Codex](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/codex.md).

## Results

| Proof | Plugin, Claude Code 2.1.296 | Plugin, Codex 0.162.1 | Project hooks, Claude Code 2.1.296 | Project hooks, Codex 0.162.1 |
|---|---|---|---|---|
| Messages reach the agent byte for byte | 10/10 | 10/10 | 10/10 | 10/10 |
| Replies reach the tester byte for byte | 10/10 | 10/10 | 10/10 | 10/10 |
| Same, with a system prompt that tells the model to rewrite both | 5/5 | 5/5 | 5/5 | 5/5 |
| Model call to the agent denied, agent receives nothing | yes | yes | yes | yes |
| Audit finds planted faults | 5/5 | 5/5 | 5/5 | 5/5 |
| After the test, the model has no memory of the conversation, and reads it from the transcript | yes | yes | yes | yes |

Data: [plugin in Claude Code](https://github.com/mohanraj00/nookku/blob/main/proofs/plugin-claude-code/results.json), [plugin in Codex](https://github.com/mohanraj00/nookku/blob/main/proofs/plugin-codex/results.json), [project hooks in Claude Code](https://github.com/mohanraj00/nookku/blob/main/proofs/hooks-claude-code/results.json), [project hooks in Codex](https://github.com/mohanraj00/nookku/blob/main/proofs/hooks-codex/results.json), [evaluation](https://github.com/mohanraj00/nookku/blob/main/docs/results.md#p5-the-model-judges-the-record-not-its-memory). Method: [docs/results.md](https://github.com/mohanraj00/nookku/blob/main/docs/results.md#1-proofs).

Under pressure, the mechanism had **0 breaks in 1,000 turns**. The test had 40 scripted conversations in each harness, 5 or 20 turns long. They had refusals, HTTP 500 errors, questions back to the tester and ambiguous messages. Each turn was a new harness call. I registered the design before the first run. Method, data and the one deviation: [docs/results.md](https://github.com/mohanraj00/nookku/blob/main/docs/results.md#2-benchmark-under-pressure).

## Docs

The how-to rows are in the order of the tasks of a test: choose, connect, run, read the results, fix. The row after them has the tasks that maintain Nookku.

| Kind | Pages |
|---|---|
| Tutorial | [Get started](https://github.com/mohanraj00/nookku/blob/main/docs/getting-started.md) |
| How-to: choose | [Choose a relay](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/choose-a-relay.md) |
| How-to: connect your app | [Test your own app](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/test-your-app.md), [Connect your agent](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/connect-your-agent.md), [HTTP tap](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/http-tap.md), [Test a streaming agent](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/test-a-streaming-agent.md), [Isolate an Agent SDK session](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/isolate-agent-sdk.md), [Add a backend](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/add-a-backend.md), [Model calls and OpenTelemetry](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/record-model-calls.md) |
| How-to: run a test | [Claude Code plugin](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/claude-code-plugin.md), [Claude Code project hooks](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/claude-code-hook-kit.md), [Codex](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/codex.md) |
| How-to: read the results | [Read the results of a test](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/read-the-results.md) |
| How-to: fix | [Troubleshooting](https://github.com/mohanraj00/nookku/blob/main/docs/troubleshooting.md), [Recover a stuck test](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/recover-a-stuck-test.md) |
| How-to: maintain | [Upgrade](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/upgrade.md), [Remove](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/remove.md), [Run the proofs](https://github.com/mohanraj00/nookku/blob/main/docs/how-to/run-the-proofs.md) |
| Reference | [CLI](https://github.com/mohanraj00/nookku/blob/main/docs/reference/cli.md), [Configuration](https://github.com/mohanraj00/nookku/blob/main/docs/reference/config.md), [Records](https://github.com/mohanraj00/nookku/blob/main/docs/reference/records.md), [Glossary](https://github.com/mohanraj00/nookku/blob/main/docs/reference/glossary.md), [SPEC.md](https://github.com/mohanraj00/nookku/blob/main/SPEC.md), [Results](https://github.com/mohanraj00/nookku/blob/main/docs/results.md) |
| Explanation | [Architecture](https://github.com/mohanraj00/nookku/blob/main/docs/architecture.md), [Limits](https://github.com/mohanraj00/nookku/blob/main/docs/limits.md), [FAQ](https://github.com/mohanraj00/nookku/blob/main/docs/faq.md), [Worked evaluations](https://github.com/mohanraj00/nookku/blob/main/docs/evaluation-example.md) |

The full map is [docs/index.md](https://github.com/mohanraj00/nookku/blob/main/docs/index.md). Read [docs/limits.md](https://github.com/mohanraj00/nookku/blob/main/docs/limits.md) before you trust a result: the deny is best effort, and the entry is not audited.

## Contribute

Read [CONTRIBUTING.md](https://github.com/mohanraj00/nookku/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/mohanraj00/nookku/blob/main/CODE_OF_CONDUCT.md). Report a security problem as [SECURITY.md](https://github.com/mohanraj00/nookku/blob/main/SECURITY.md) says.

## License

Apache-2.0. See [LICENSE](https://github.com/mohanraj00/nookku/blob/main/LICENSE).
