Metadata-Version: 2.5
Name: fannypack-agents
Version: 0.3.0
Summary: Fast, reversible tool calls for LLM agents. Your tools, strapped on, ready before you reach for them.
Project-URL: Homepage, https://github.com/nikhilkulkarni1755/fannypack-agents
Project-URL: Repository, https://github.com/nikhilkulkarni1755/fannypack-agents
Project-URL: Issues, https://github.com/nikhilkulkarni1755/fannypack-agents/issues
Author: Nikhil Kulkarni
License: Apache-2.0
License-File: LICENSE
Keywords: agents,compensation,latency,llm,mcp,tool-calling,undo
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: http2
Requires-Dist: h2>=4; extra == 'http2'
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == 'mcp'
Description-Content-Type: text/markdown

# fannypack

**Fast, reversible tool calls for LLM agents.** Your tools strapped on and within
reach: the runtime already has the likely call in flight before the model
finishes asking for it, and every call it makes is recorded well enough to be
taken back afterwards.

Two claims, both measured (current release: 0.3.0):

1. **Latency shaving** — 1.2–1.8x off a tool-calling turn, taken from the parts
   of the latency that are the runtime's fault rather than the model's. Bounded
   by tool latency: with tools under ~50ms there is nothing to shave (a coding
   belt measured 1.0–1.17x); the gain grows with slower, uneven, or repeated
   calls.
2. **Selective undo** — take back one action and leave every action that never
   depended on it standing. The runtime says which of three things an action is
   (nothing to undo, reversible, irreversible) and refuses rather than pretends.

## Install

```bash
pip install fannypack-agents        # the bare name on PyPI is an unrelated package; the import is `fannypack`
```

or from source:

```bash
git clone https://github.com/nikhilkulkarni1755/fannypack-agents && cd fannypack-agents
pip install -e .                    # one dependency: httpx
```

Python 3.10+. Provider keys go in `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or
`FIREWORKS_API_KEY`; Ollama needs no key.

## Quickstart: a bank agent

Register tools with a decorator. Each one says what it does to the world, and
the ones that can be taken back say how.

```python
import asyncio
from fannypack import Agent, Effect, Pack, providers

pack = Pack()
balances = {"acct_alice_0001": 1000.0, "acct_bob_0002": 250.0}
transfers = []

@pack.tool(effect=Effect.READ_ONLY)                      # a read: cacheable, safe to run early
async def get_balance(account_id: str) -> dict:
    """Balance of one account."""
    return {"account_id": account_id, "balance": balances[account_id]}

@pack.tool(effect=Effect.IDEMPOTENT_WRITE)               # a write; the same reference twice moves money once
async def transfer(from_account: str, to_account: str, amount: float, reference: str) -> dict:
    """Move money between accounts."""
    balances[from_account] -= amount
    balances[to_account] += amount
    transfers.append(reference)
    return {"transfer_id": f"txn_{len(transfers):06d}", "from_account": from_account,
            "to_account": to_account, "amount": amount}

@pack.compensator("transfer")                            # this is what makes transfer reversible
async def reverse_transfer(result: dict) -> dict:
    balances[result["to_account"]] -= result["amount"]
    balances[result["from_account"]] += result["amount"]
    return {"reversed": result["transfer_id"]}

@pack.tool(effect=Effect.NON_IDEMPOTENT_WRITE)           # no compensator: the notes are gone
async def withdraw_cash(account_id: str, amount: float) -> dict:
    """Dispense cash at an ATM."""
    balances[account_id] -= amount
    return {"dispensed": amount}

async def main():
    agent = Agent(pack, providers.from_spec("anthropic:claude-sonnet-5"))
    run = await agent.run("Move 100 from acct_alice_0001 to acct_bob_0002 with reference 'rent-sep', "
                          "then confirm both balances.")
    print(run.answer)
    print(run.metrics.as_dict())

asyncio.run(main())
```

Schemas come from your type hints; `Annotated[str, "..."]` adds a parameter
description. No base classes, no rewrite of your functions.

### Selective undo

Suppose the transfer was a mistake.

```python
txn = next(a for a in run.actions() if a.tool == "transfer")

print(run.plan_undo(txn.id).explain())
# undo a_7f2c19 in run_4b81:
#   compensate a_7f2c19 (transfer) [target] -- compensating via reverse_transfer

await run.undo(txn.id)                      # the money is back; the balance reads that
                                            # followed are untouched -- they never depended on it
```

And suppose the model had withdrawn cash instead:

```python
cash = next(a for a in run.actions() if a.tool == "withdraw_cash")
await run.undo(cash.id)
# Irreversible: withdraw_cash registers no compensation
```

There is no `force` flag. Every action resolves to one of three states —
nothing to undo (a read), reversible (a write with a compensator), irreversible
(a write without one) — and the plan tells you which before anything runs.

A write whose inverse needs the *old* state — an edit, an overwrite — gets it
from a snapshot the runtime takes just before the write and stores on the
action:

```python
@pack.snapshotter("write_file")
def before_write(args: dict) -> dict:
    return {"content": files.get(args["path"])}

@pack.compensator("write_file")
def restore(args: dict, before: dict) -> dict:
    files[args["path"]] = before["content"]
    return {"restored": args["path"]}
```

What makes this *selective* is the ledger: it records which later calls used
which earlier results (a `transfer_id` returned by one call and passed to
another), so undoing one action cascades only through the calls that actually
consumed it. `run.why(action_id)` shows that chain; `run.verify()` confirms the
history was appended to, never rewritten.

### Restart from step N

```python
resumed = await agent.resume(run.id, from_action=some_model_turn.id)
```

The transcript up to that point is rebuilt from the ledger, and every call the
original run completed is served back from the record instead of executed
again — reads within their freshness window, writes unconditionally. Twelve of
twelve resumed bank transfers across four real models moved the money exactly
once.

### before[] → model → after[]

The policy's first use was to *pre-fire* calls into the cache; the model still
spent a turn asking for them. `Options(prefill=True)` goes the rest of the way:
the calls a request of this shape always starts with run **before the first
model call**, and go into the transcript as a turn already taken — an assistant
message that made them, and their results. The model's first real turn starts
from `after[]`, the decisions the table doesn't know.

```python
agent = Agent(pack, provider, ledger=ledger, options=Options(policy=True, prefill=True))
agent.warm()                       # learn before[] from past runs
run = await agent.run("I need $50 rn from acct_alice_0001")
# before[]: get_balance(acct_alice_0001) ran while the prompt was being built
# model:    decides withdraw_cash(50) -- writes are always the model's
```

Mapping "I need $50 rn" to `get_balance` is the hard part, and it is two
deterministic tiers, no model involved, ~100µs: an exact lookup on the
request's *shape* (its tool-vocabulary words), and, failing that, cosine
similarity against the words and character trigrams of every past goal that
led to each prescription. "Withdraw 20 dollars", "take out 40 in cash" and "I
want to withdraw some cash" all taught the same signature; "I need $50 rn"
matches it and fires; "what's the weather" does not. Only read-only calls are
ever run this way. A wrong match costs one read and is reported. When the table
knows a chain (`find_sources` → `read_source($prev.next)` → `verify_claim`),
the whole chain runs before the model reads the prompt.

Prefill is a bet -- reads now against model turns later -- and the policy
learns both sides of it, so it declines when history says the reads would
cost more than the turns they remove (a fast model with slow tools). Measured:
11.3s → 6.3s on a 3.4s-a-turn model, correctly skipped on a 0.45s one. Two
known limits: results must be spliced in as tool results (a prose "already
done" paragraph is ignored or re-fetched by every model tested), and it should
not be combined with `plans` on a task defined by a step count -- the model
loses the count.

## The vocabulary

| Term | What it means | Where |
|---|---|---|
| **Pack** | Your tools, registered once, each with an effect class | `Pack`, `Effect` |
| **Ledger** | The append-only, hash-chained record of what happened and what depended on what | `Ledger` |
| **Latency shaving** | Taking off every part of a turn's time that is the runtime's to take: the flags in the next table | `Options` |
| **Selective undo** | Take one action back; independent actions stand; irreversible ones refuse | `run.undo` |
| **Reach-ahead** | Start a *guessed* read-only call before the model asks for it | `Options.speculate` |
| **Pre-dispatch** | Start the calls a request of this shape *always* needs, from a learned table: deterministic, one lookup, the same schedule every time | `Options.policy` |
| **Prefill** | Run before[] — and the chain the table knows follows it — before the first model call, spliced in as a turn already taken; the model starts from after[] | `Options.prefill` |
| **One reach** | Several dependent calls in a single model turn — `"$1.next"` feeds step 1 into step 2; each step starts the moment its JSON closes, while the rest of the plan is still streaming | `Options.plans` |
| **Pre-check** | Does this string need a tool, and which? ~60µs, deterministic | `Classifier` |

Latency shaving is these flags, each measurable alone:

| Flag | Removes |
|---|---|
| `routing` | The tokens of every schema you *don't* need this turn (65–85% of input) |
| `stream_ahead` | The gap between "the model decided" and "the call started" |
| `parallel` | Queue time behind independent calls |
| `caching` | Repeated and duplicated I/O. A write drops every cached read about the same identifiers, so a balance is never served from before a withdrawal |
| `speculate` / `policy` | The first call's latency entirely |
| `prefill` | The model turns that would have asked for what the table already knows |
| `plans` | N−1 of the N model round-trips in a dependent chain |

`Options.baseline()` turns everything off — that is how most agent loops run
today, and what the numbers below compare against. It is an honest baseline:
measured equal to a hand-written sequential SDK loop to within 10ms on every
shape tested.

Which flags matter for which shape, measured against peer runtimes (OpenAI
Agents SDK, Pydantic AI, LangGraph) on a scripted model:

- **Repeated reads** — the cache is the largest default-on win, 1.27x at 0.5s
  tools rising to 1.59x at 2s. No peer runtime has one.
- **Uneven fan-out** (one slow call among fast ones) — stream-ahead wins by
  1.05–1.11x; on a *homogeneous* fan-out every runtime that runs calls
  concurrently ties within 20ms, and all three peers do.
- **Dependent chains** — only `plans` help, and a plan is worth about one model
  round-trip per step saved, so its value is set by the model's per-turn time:
  ~2s on Sonnet, less on faster models. Steps now run while the plan is still
  streaming (0.4–0.9s of tool time under decode on gpt-oss chains); on a
  scripted model that is a 5-step chain in 1.15s instead of 1.75s.
- **Small belts** — `routing` is inert until the pack is larger than `route_k`
  (12); it exists for the 50-tool case.

## Findings

Three rounds, four models (`claude-sonnet-5`, `claude-haiku-4-5`, `gpt-5.1`,
`gpt-oss-120b`), a 47–50 tool belt, every run correctness-gated so a
configuration cannot get faster by doing less. Full tables in
[`docs/results.md`](docs/results.md), [`docs/results-2.md`](docs/results-2.md)
and [`docs/results-3.md`](docs/results-3.md); raw samples and ledgers in
[`bench-results/`](bench-results/).

**Claim 1 — latency shaving, 1.2–1.8x.** On independent-call workloads the runtime removes its
share: gpt-5.1 `fan_out` 13.0s → 7.6s, Sonnet 18.0s → 14.4s, the clearest single
mechanism being parallel fan-out taking `queued` from 12.7s to 0. On strictly
dependent chains the ordinary mechanisms are neutral — nothing to overlap — and
*one reach* is what moves them: gpt-5.1 `chain_8` 13.7s → 7.6s, nine model
turns to two or three. Pre-dispatch takes a warm `research_chain_3` from 11.3s
to 5.8s on gpt-oss at 100% precision.

**Claim 2 — selective undo.** In the suite, undoing a transfer reverses exactly
the transfer; undoing a route change restores the route and leaves the brake
applied; undoing a cash withdrawal is refused with the reason. Restart from any
model turn replays the record: 12/12 exactly-once.

**What does not help, measured.** `compact_state` loses on most workloads.
Reach-ahead from the goal text wasted about one read per run and is off by
default. Routing *hurt* on Sonnet chains until the routed prefix was frozen and
large enough to prompt-cache. Haiku never uses one-reach plans at all. The
runtime's own time is 2–3ms a turn, under 8ms at max; the model is where the
seconds are.

**Round four, the limits** ([`docs/results-4.md`](docs/results-4.md)). Prefill
as a synthesized turn: **11.3s → 6.3s (1.78x)** on a 3.4s-a-turn model, and
correctly declined on a 0.45s-a-turn one; as a prose paragraph it never won.
Plans hold at 16 dependent steps (1.25x, two plans). A 200-tool belt: the
baseline fails the task, routing to 12 completes it in 5.9s (3.9x). The query →
before[] mapping: 6/6 on paraphrases, 0 false fires on near-misses. Where it
breaks: prefill plus plans on a step-counted task loses the count; a model
that calls one tool per turn caps every fan-out gain.

**Do models need fine-tuning to call tools faster?** No, for round-trips —
plans and pre-dispatch get that from frontier models by prompting. What a
fine-tune *would* buy, read off the ledgers with `fannypack mine`: gpt-oss
emits sequential-but-independent calls 48% of the time (plans already fix
that), Haiku writes ~15 tokens of prose before each call, gpt-5.1 and Sonnet
nothing. [`docs/research.md`](docs/research.md) has the literature.

## Run the experiments

```bash
fannypack bench --provider anthropic:claude-sonnet-5 --repeats 3 --json out.json
fannypack report out.json
fannypack compare bench-results/*.json

cd suite && uv sync            # the external suite: a bank, a car, a camera, a research chain
uv run fannypack-suite chain  --provider openai:gpt-5.1 --db /tmp/ledgers
uv run fannypack-suite policy --provider fireworks:accounts/fireworks/models/gpt-oss-120b
uv run fannypack-suite mine     /tmp/ledgers/*.db
uv run fannypack-suite classify /tmp/ledgers/*.db
```

## Elsewhere

- **As a library**: `PackServer` gives an agent loop you already have the
  ledger, the cache and selective undo without adopting `Agent`; `Router`,
  `ResultCache`, `Ledger` and `plan_undo` work on their own.
- **Over MCP** (`pip install -e '.[mcp]'`): `examples/serve_mcp.py` serves a
  pack to Claude or ChatGPT with `fannypack_undo` and `fannypack_why` as tools.
- **CLI**: `fannypack inspect | why | graph | verify | undo | policy | mine | classify`.
- **Not this**: a tracing dashboard, a durable execution engine, an agent graph
  framework, a replacement for MCP, a sandbox. [`docs/design.md`](docs/design.md)
  has the reasoning and the known limits.

If you find it useful, a star on the repo helps other people find it.

Apache-2.0.
