Metadata-Version: 2.5
Name: evenkeel
Version: 0.1.1
Summary: Failure classification, retries with hints, dollar budgets and escalation for LangChain agents. Ported from keel.
Project-URL: Repository, https://github.com/yodablocks/evenkeel
Project-URL: keel, https://github.com/yodablocks/keel
Author: yodablocks
License: MIT
License-File: LICENSE
Keywords: agents,budgets,human-in-the-loop,jev,langchain,langgraph,middleware,retries
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: langchain<2,>=1.3
Requires-Dist: langgraph<2,>=1
Requires-Dist: typing-extensions>=4.7
Provides-Extra: jev
Requires-Dist: typesafe-sdk<0.8,>=0.7; extra == 'jev'
Description-Content-Type: text/markdown

# evenkeel

Keeps a LangChain agent on an even keel when things fail: failures are classified and handled by kind,
bad output is retried with a hint, spend is tracked in dollars with a budget, and failures that need a person
pause the run for approval.

Ported from [keel](https://github.com/yodablocks/keel), a durable execution engine for AI agents.
An independent project, not affiliated with LangChain. Experimental: 0.x, expect changes.

## What it adds to LangChain's built-in middleware

| Need | Built in | EvenKeelMiddleware |
|---|---|---|
| Retries | `ModelRetryMiddleware`, `ToolRetryMiddleware` retry with the same input | Classifies the failure first: back off, retry **with a hint**, fail at once, or escalate |
| Limits | `ModelCallLimitMiddleware` counts calls | Tracks **dollars** from token usage and pauses at a budget |
| Approvals | `HumanInTheLoopMiddleware` asks before chosen tools | Escalates **when a failure needs a person**, from any model or tool call |

Use it alongside the built-ins: `ModelFallbackMiddleware` for switching models, `HumanInTheLoopMiddleware`
for approvals fixed in advance.

## Install

    pip install evenkeel          # rule-based classification
    pip install "evenkeel[jev]"   # adds TypeSafe's Jev classifier

## Use

```python
from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver
from evenkeel import EvenKeelMiddleware, ModelPrice, PriceTable

agent = create_agent(
    "openai:gpt-4o-mini",
    tools=[...],
    middleware=[
        EvenKeelMiddleware(
            budget_usd=0.50,
            prices=PriceTable({"gpt-4o-mini": ModelPrice(input_per_mtok=0.15, output_per_mtok=0.60)}),
        )
    ],
    checkpointer=InMemorySaver(),  # needed for pauses
)
```

With Jev (needs `TYPESAFE_API_KEY`):

```python
from typesafe_sdk import TypeSafeClient
from evenkeel import EvenKeelMiddleware, JevClassifier

EvenKeelMiddleware(classifier=JevClassifier(TypeSafeClient()))
```

## How failures are handled

| Kind | Example | Action |
|---|---|---|
| `transient` | 429, 503, timeout, dropped connection | Back off and retry the same call |
| `bad_output` | Malformed output, a call that breaks the schema (recognised by Jev, or raised as `BadOutputError`) | Retry with a hint saying what was wrong |
| `bad_input` | Input a tool rejects (`BadInputError`, HTTP 400 or 422, a pydantic `ValidationError`) | Model call: fail. Tool call: return the error to the model so it fixes its arguments |
| `needs_human` | Refund over a limit, a policy block | Pause with `interrupt()` and wait for a decision |
| `fatal` | A bug, bad credentials | Re-raise the original error at once |

Retries and escalations stop at `max_attempts` (default 4). Each failure is recorded (error, kind, confidence,
which classifier decided, action): in the agent's state under `evenkeel_attempts` when the call ends in a
result, and on the raised error as `error.evenkeel_attempts` when it finally fails.

Some failures never reach the middleware, because LangChain turns them into messages for the model itself:
a call to a tool that does not exist, and tool arguments that fail the tool's schema.

## Pauses and how to resume

A failure that needs a person:

```python
from langgraph.types import Command

agent.invoke(Command(resume={"decision": "retry", "hint": "approved by finance"}), config)
agent.invoke(Command(resume={"decision": "fail"}), config)
agent.invoke(Command(resume={"decision": "skip", "message": "Refunded by hand."}), config)  # tool calls only
```

A `retry` with a `hint` on a tool call sends the reviewer's words to the model, which can call the tool again; a
tool cannot take a hint itself. A call that still needs a person after one approved retry fails. An answer
that is not one of these values does not raise: the run pauses again, with an `error` field in the payload
saying what was wrong.

Over budget:

```python
agent.invoke(Command(resume={"approve": True, "budget_usd": 1.00}), config)
agent.invoke(Command(resume={"approve": False}), config)  # ends the run
```

The budget applies to each run: every new `invoke` on a thread starts with zero spend and the configured
budget.

## Limitations

- **Budgets can overshoot by one call.** The check runs before a model call, but a call's cost is known only
  after it. keel has the same limitation.
- **No built-in prices.** You supply them; a model without a price, or a response without token usage, is an
  error rather than a silent $0. Dated or versioned names (`gpt-4o-mini-2024-07-18`, `claude-3-5-sonnet-20241022`, `gpt-4-0613`, `...-latest`)
  use the price of their base name; any other variant (`gpt-4o-mini` against `gpt-4`, `gpt-4-turbo`) needs its own
  entry.
- **On resume after an escalation, the failing call runs once more** before the decision applies, because
  LangGraph re-runs the paused node. For a tool with side effects, make it idempotent or use `skip`.
- **Classification is only as good as the classifier.** Rules only recognise explicit signals (status codes,
  network errors, evenkeel's error classes): on keel's 75 error messages from public SDK issues they pick the
  right kind 29 times and the right action 39 times. With Jev the results are below.

## Classifier accuracy

Measured with `evals/jev_eval.py` on keel's labelled error sets: 75 error messages from public SDK issues and
60 sampled from 13 other repositories. Jev pinned to `jev-1.13.0`, one run on 2026-10-14. "Right action" counts
a verdict as right when the default policy does the same thing as for the label (`bad_input` and `fatal` both
fail).

| Classifier | Public set, right kind | Public set, right action | Fresh set, right kind | Fresh set, right action |
|---|---|---|---|---|
| Rules only | 29 / 75 (39%) | 39 / 75 (52%) | 43 / 60 (72%) | 48 / 60 (80%) |
| `JevClassifier` (rules for explicit signals, Jev for the rest, rules below 0.5 confidence) | 50 / 75 (67%) | 63 / 75 (84%) | 49 / 60 (82%) | 56 / 60 (93%) |

Caveats: the labels are one person's (keel's author, who also wrote the classifier prompt), most messages come
from development rather than production, and Jev's answers vary by about one case between runs. On the public
set Jev answered 66 cases and the rules 9 (Jev below 0.5 confidence), of which 7 were wrong. keel's TypeScript
cascade scored 46 to 47 and 59 to 60 on the same set ([its results](https://github.com/yodablocks/keel#failure-classification-rules-vs-jev));
the difference is in how often Jev falls below the confidence floor, and its cause has not been established.

## Develop

    uv sync --all-extras
    uv run pytest
