Metadata-Version: 2.5
Name: saddle-harness
Version: 0.1.0
Summary: Deterministic execution harness for non-deterministic local LLMs.
Author: Eliza Halpern, Eryn Lipkowitz
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: coverage>=7.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.13.5
Requires-Dist: python-multipart>=0.0.12
Requires-Dist: rich>=13.9
Requires-Dist: starlette>=0.41
Requires-Dist: uvicorn>=0.32
Description-Content-Type: text/markdown

<img width="492" alt="saddle" src="docs/saddle-logo.svg" />

# Saddle

Break your local model to harness.

Saddle is a deterministic execution harness that turns a non-deterministic local LLM into a dependable coding tool: constrained DAG planning, parallel proof-gated workers, machine-checked gates. No retries on unverified foundations.

**Status:** early. The agent-plus-auditor loop below is built and measured on a small benchmark (details under *Status* below). Don't trust it unattended yet: on harder tasks the auditor has accepted wrong work (#109).

## Phase 2: agent plus auditor

Phase 2 adds:

- `saddle auto`: one task, run autonomously in a git worktree, with a time and token
  budget. The result is a branch.
- A tiered auditor whose findings reach the model as tool results. `finish` is refused
  while the audit fails.
- A lane chip in the chat: **Ask** (the default, read-only: the model gets only read,
  list and search tools), **Edit** (opt-in, edits your folder directly, unaudited) and
  **Task · Small**, which starts the same audited run from the browser and shows an
  evidence packet compiled from the ledger.
- Run-state signals so a run finds you: tab title, favicon, an opt-in browser
  notification, and a state pill per session in the sidebar.
- Two questions a run can ask once each: whether tests may be edited when the auditor
  needs a test, and whether to extend a budget at 80%. Unanswered, each takes the old
  behaviour as its default.
- `saddle verify --anchor`, which checks each run's outcome against the
  `Saddle-Outcome` trailer on its branch.
- `saddle audit --tiered`, which runs the same battery on any diff.

- [docs/USING-SADDLE.md](docs/USING-SADDLE.md): install, the Ask / Edit / Task lanes,
  run-state signals, the two questions, how runs end, reading the packet,
  `saddle verify --anchor`.
- [docs/AUDIT-TIERS.md](docs/AUDIT-TIERS.md): the gates by tier, reason classes,
  caching.
- [docs/CLI.md](docs/CLI.md): every flag and exit code.

**Status as of 2026-09-28:**

- **Measured, no harm:** on two benchmark tasks, six seeds each, the audited agent's
  false-done rate was 0 of 6 on both, the same as the plain agent's, at 0.95x and 0.69x
  of its wall time to a finished evidence packet.
- **Measured, detection:** a rate on planted defects, not a pass. Re-audited offline on
  `48eb6d6`, the auditor refused 65 of 65 planted-bug trees, 49 of 65 for the right
  reason. Every right-reason refusal came from the agent's own tests failing; on one
  task the other gates refused correct and wrong trees alike.
- **Observed, not yet measured:** in single unscored runs on two harder tasks, the
  auditor accepted a tree that failed the task's hidden checks (#109). Measuring that
  rate is next.
- **Not measured:** detection of defects that were not planted.
- **Not built:** only the Small lane exists. Contracts are not sealed before
  implementation. The only questions a run asks are the two above.

## Layout

- `docs/ARCHITECTURE.md` — full architecture specification.
- `src/saddle/{dag,gates,evidence,runner,slice,scheduler,journal,vllm,cli}.py` — the vertical slice, in layering order `dag → gates → evidence → runner → slice`:
  - `dag.py` — DAG data model and validation.
  - `gates.py` — pure gate predicates.
  - `evidence.py` — subprocess policy and gate-runner adapters.
  - `runner.py` — node execution.
  - `slice.py` — end-to-end vertical slice orchestration.
  - `scheduler.py` — Kahn scheduler.
  - `journal.py` — proof journal.
  - `vllm.py` — vLLM client.
  - `cli.py` — command-line interface.
- `transcript.py`, `timeline.py`, `ux.py`, `tools.py`, `chat.py` — reporting and CLI support (`chat.py` present; see #23).

## Quickstart

The CLI as built (`cli.build_parser`; every command and flag in docs/CLI.md): `saddle doctor`, `saddle dag`, `saddle run`, `saddle auto`, `saddle audit`, `saddle tail`, `saddle verify`, `saddle explain`, `saddle chat` (alias `saddle web`), `saddle up`. Configure the vLLM API key via `SADDLE_VLLM_API_KEY` or `VLLM_API_KEY`. Run the suite with:
```bash
PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest -q
```

## Requirements

- Python 3.12+
- Self-operated vLLM ≥ 0.28 server with guided decoding (XGrammar) + KV offloading —
  the proven setup is [qwen38-27b-rtx3090](https://github.com/syv-ai/qwen38-27b-rtx3090)
- pytest, coverage.py, ruff, mutmut (sampled per node in Tier 1; the merge-time Tier 2 is not built)
- Linux with `bwrap` (bubblewrap) that can start: Task runs refuse to run without it.
  On Ubuntu 24.04 unprivileged user namespaces are restricted by AppArmor, so `bwrap`
  needs the upstream `bwrap-userns-restrict` profile or an equivalent.
- A user systemd manager (`systemd-run --user --scope` works) for the per-command memory
  cap; without one saddle falls back to a weaker per-process address-space ceiling.
  `SADDLE_MEMORY_MAX` sets the cap (bytes, or a number with K, M, G or T; default `6G`).

## Security model

Saddle runs code a model wrote, on your machine, as you. It confines some of that and
not all of it. Run it only on code you would be willing to run yourself.

It needs a model server: a local OpenAI-compatible chat-completions endpoint (vLLM with
structured outputs; `--base-url`, default `vllm.DEFAULT_BASE_URL`,
`http://127.0.0.1:18020/v1`). Your prompts and the files the model reads go to that
server. Point `--base-url` somewhere else and they go there instead.

**What is confined** (`sandbox.py`, `memcap.py`):

- **File tools stay in the workdir.** Every path a tool reads or writes is resolved,
  symlinks included, and refused if it lands outside the working directory
  (`sandbox.resolve_within`).
- **Commands run under `bwrap` when it can start** (`Sandbox._argv`). The root is built
  up from the system directories and the interpreter, read-only (`SYSTEM_DIRS`); HOME
  and /tmp are empty; the workdir is the one writable place, and its `.git` is
  read-only (`Sandbox._git_binds`). Each command gets its own namespaces (sharing the
  host network unless the lane takes it away), its own session and process group, and
  its process group is killed when it ends.
- **The environment is an allowlist** (`sandbox.ENV_KEEP`), so the model key in
  saddle's own environment does not reach a command.
- **A Task run (`saddle auto`) requires isolation and has no network.** It refuses to
  start when `bwrap` is missing or cannot start (`require_isolation=True`,
  `IsolationUnavailableError`), and its commands see loopback only (`network="none"`).
- **Every command the model runs has a memory cap** (`memcap.cap`): a systemd user scope with
  `MemoryMax`, no swap and at most 4096 tasks, or a per-process `prlimit --as` ceiling
  where no user systemd manager is reachable. The cap also covers the audited tree's
  test command and `mutmut run` (`evidence.tree_memory_limit`). Default 6 GiB;
  `SADDLE_MEMORY_MAX` changes it. A command the cap kills is recorded as a failure.
- **The git calls of an autonomous run and of the packet's branch actions run with
  hooks and `core.fsmonitor` off** (`sandbox.HOST_GIT_GUARD`, used in `auto.py` and
  `web/branch_actions.py`), so config planted in `.git` does not run there. saddle's
  other git calls, such as the gates' diffs (`evidence.py`), do not pass it.
- **The chat server binds to loopback by default**; any other bind address requires a
  token (`web.app.needs_token`).

**What is not confined:**

- **The auditor's gates.** `pytest`, `coverage` and `mutmut` run as you on the host,
  with your read access and your network, under the memory cap only (#104). A test the
  model wrote runs there during an audit.
- **Ask and Edit in the chat without a working `bwrap`.** Those lanes use `bwrap` when it
  starts; otherwise commands run as you and the sandbox reports `isolation: none`
  (`Sandbox.isolation`). They keep the host network either way. Ask offers the model
  read-only tools only (`tools.tools_for_mode`); Edit writes your folder directly,
  unaudited.
- **`git`, `ruff` and `coverage` bookkeeping calls** have no memory cap.

## Reading the comments

Comments and docstrings state the claim they rely on, and the evidence behind it, in
plain words. Some name a benchmark task and seed (`T5`, `t8-s1`) or a round of runs
("round 3e"); docs/BENCHMARK-RECORD.md describes the tasks and the rounds. The raw run
records behind those measurements are the author's own and are not public.

## License

Copyright (C) 2026  Eliza Halpern and Eryn Lipkowitz. See [AUTHORS](AUTHORS).

Saddle is free software under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See [LICENSE](LICENSE).
