Metadata-Version: 2.4
Name: resilientforge
Version: 0.3.0
Summary: Persistent, cross-run failure memory for tool-calling agents.
Project-URL: Homepage, https://github.com/amir2628/resilientforge
Project-URL: Issues, https://github.com/amir2628/resilientforge/issues
Author: ResilientForge contributors
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: anthropic>=0.30
Requires-Dist: chromadb>=0.5
Requires-Dist: openai>=1.30
Requires-Dist: pydantic>=2.0
Requires-Dist: typer>=0.12
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.100; extra == 'dashboard'
Requires-Dist: uvicorn>=0.23; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: cloudpickle>=3.0; extra == 'dev'
Requires-Dist: httpx>=0.24; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: isolation
Requires-Dist: cloudpickle>=3.0; extra == 'isolation'
Provides-Extra: langgraph
Requires-Dist: langgraph>=0.2; extra == 'langgraph'
Provides-Extra: semantic
Requires-Dist: sentence-transformers>=2.2; extra == 'semantic'
Description-Content-Type: text/markdown

# ResilientForge

[![CI](https://github.com/amir2628/resilientforge/actions/workflows/ci.yml/badge.svg)](https://github.com/amir2628/resilientforge/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](./LICENSE)

ResilientForge is a small, framework-agnostic Python library that sits on top of an
agent's existing tool-calling loop and adds **persistent, cross-run failure memory**.

When a wrapped agent hits a tool failure or violates a defined invariant,
ResilientForge captures the failure, checks a local failure oracle for a
previously-successful fix, applies it if found — no model call needed — and,
if it has to generate a new fix via a model call, writes that fix back to
the oracle so the next occurrence of the same failure *shape* is resolved
without another model call, even if the specific values involved are
different. Once a fix has proven itself reliably enough times, it's
promoted into a **standing guard** that fixes the arguments *before* the
tool is even called — the failure stops recurring at all, instead of
recurring once and then being fixed on retry.

**Status:** v0.2.0, Alpha — not yet published to PyPI. Full phase-by-phase
development history is in [`CHANGELOG.md`](./CHANGELOG.md); see
[`docs/architecture.md`](./docs/architecture.md) for how it actually works
under the hood, and "Validation" below for how it's held up against a real
external agent.

## Quickstart

```bash
pip install resilientforge
```

```python
from resilientforge import wrap

def create_event(date: str, title: str = "Event") -> dict:
    import re
    if not re.match(r"^\d{4}-\d{2}-\d{2}$", date):
        raise ValueError(f"could not parse date '{date}'")
    return {"date": date, "title": title, "status": "created"}

def reflect(context):
    # Stand-in for a real model call — see integrations/raw_tool_loop.py's
    # create_anthropic_reflect() for a real Anthropic-backed one.
    return {
        "strategy": "reformat_argument",
        "transforms": [{"argument": "date", "transform": "parse_relative_date_to_iso"}],
    }

wrapped = wrap(create_event, reflect=reflect)

wrapped.invoke(date="next Friday", title="Standup")
# -> {"date": "2026-07-17", "title": "Standup", "status": "created"}
# (recovered: date wasn't ISO 8601, reflect() proposed reparsing it, retried, succeeded)

wrapped.invoke(date="next Tuesday", title="Retro")
# -> {"date": "2026-07-21", "title": "Retro", "status": "created"}
# (recovered via the recipe learned above — reflect() was NOT called again)
```

That's the raw tool-calling loop integration — the simplest entry point,
and the reference implementation. For a real Anthropic tool-use loop, an
OpenAI function-calling loop, or a LangGraph `ToolNode`, see
`integrations/raw_tool_loop.py` / `integrations/langgraph_adapter.py` and
the runnable demos in `examples/`:

```bash
python examples/raw_loop_demo.py
python examples/langgraph_demo.py
python examples/guards_demo.py
```

## Does it work? Real numbers, not a claim

The table below is generated by `pytest tests/failure_injection` — seven
scenarios reproducing the real-world failure patterns this project targets
(malformed tool-call JSON, a missing required field, a natural-language date
where ISO 8601 was expected, a transient timeout, a wrong-typed argument, a
longer-running recurring-date scenario built specifically to demonstrate
guard promotion, and a scenario whose correct fix can't be guessed from the
arguments alone, built to demonstrate speculative branching), each run once
with **no** recovery mechanism (baseline) and once wrapped with
ResilientForge, with different literal values per trial:

| Scenario | Trials | Baseline recovery | Recovery (ResilientForge) | Avg attempts to recovery | Oracle hit rate (after 1st occurrence) | Guard promoted | Prevention rate | Avg candidates considered |
|---|---|---|---|---|---|---|---|---|
| malformed_json_args | 5 | 0% | 100% | 0.6 | 100% | yes | 100% | 0.2 |
| missing_required_field | 5 | 0% | 100% | 0.6 | 100% | yes | 100% | 0.2 |
| natural_language_date | 5 | 0% | 100% | 0.6 | 100% | yes | 100% | 0.2 |
| transient_timeout | 5 | 0% | 100% | 1.0 | 100% | no | 0% | 0.2 |
| wrong_type_argument | 5 | 0% | 100% | 0.6 | 100% | yes | 100% | 0.2 |
| recurring_date_guard | 8 | 0% | 100% | 0.4 | 100% | yes | 100% | 0.1 |
| ambiguous_fix_candidates | 6 | 0% | 100% | 2.2 | 0% | no | 0% | 2.2 |

The "oracle hit rate" column is Phase 1's proof: after the *first*
occurrence of a given failure shape is recovered once (which needs one
model call), every later occurrence — even with completely different
literal values — resolves via the oracle with **zero additional model
calls**. The next two columns are Phase 2's proof: once a fix has been
applied reliably enough times (3, by default), it's promoted into a
**standing guard** that fixes the arguments *before* the tool is even
called — `transient_timeout` is the one scenario where no guard is
promoted, because its correct "fix" is just a blind retry with no argument
to change. `recurring_date_guard` is the dedicated demonstration: 8 trials,
the first 3 cross the promotion threshold reactively, the remaining 5 use
dates never seen before and are all *prevented* outright, not merely
recovered from. The last column is Phase 3's: `ambiguous_fix_candidates`
still recovers 100% of trials via real, per-candidate verification, at a
real, reported cost — unlike every other scenario, its oracle hit rate is
0%, because considering multiple candidates means a model call is made
every round regardless of whether a recipe already exists. Regenerate this
table yourself with `pytest -s tests/failure_injection` (also written to
`tests/failure_injection/reports/latest.md`) — these numbers were
regenerated fresh against the current code, including the signature-
normalization fixes described below (they didn't move: none of these
seven scenarios exercise HTTP status text or hex byte literals).

## Validation: tested against a real external agent, not just our own scenarios

The numbers above come from scenarios this project wrote itself. To check
whether the oracle's recovery and signature-matching logic hold up against
real, unscripted failures, we ran 3 rounds of validation against
[`langchain-ai/react-agent`](https://github.com/langchain-ai/react-agent) —
a real external LangGraph agent, wrapped with a real web-search tool and a
real URL-content-extraction tool, zero engineered failures.

- **Round 1** found a real bug before the actual research question could
  even be tested — async tool support was completely missing from
  `integrations/langgraph_adapter.py`. Fixed, then confirmed cross-session
  recurrence-recognition on the one real failure shape that showed up (a
  network timeout).
- **Round 2** widened the real failure surface (a second, real tool) and
  found two opposite bugs in signature normalization: a **false-merge**
  (two genuinely different real HTTP errors collapsed into one signature)
  and a **missed-match** (one real problem split across two signatures) —
  plus a third finding, a fix recorded as "recovered" that was actually
  structurally inert.
- **Round 3** fixed and confirmed all three against the exact real cases
  that exposed them, and in the process found a fourth instance of the
  same inert-fix bug via a different code path — also fixed and confirmed,
  in an addendum to the same report.

Every real bug this process found was found by testing against something
real, not synthetic — and every one was fixed and re-confirmed against the
exact real case that exposed it, not just asserted fixed. This is
validation against **one real external agent, across 3 rounds** — not a
claim of broader battle-testing. Full detail, including what each round
does and doesn't prove:
[round 1](docs/real_world_validation.md) ·
[round 2](docs/real_world_validation_round2.md) ·
[round 3 + addendum](docs/real_world_validation_round3.md).

## Standing guards: prevention, not just recovery

Once a fix has succeeded reliably enough times (3, by default), it's
promoted into a **standing guard** that proactively fixes the arguments
*before* the tool is even called — the failure stops happening at all,
instead of happening once and then being fixed on retry:

```python
wrapped = wrap(create_event, reflect=reflect, guard_promotion_min_occurrences=3)

wrapped.invoke(date="next Friday", title="A")   # recovers via reflect()
wrapped.invoke(date="next Tuesday", title="B")  # recovers via the fast-path recipe
wrapped.invoke(date="next Monday", title="C")   # recovers, and this 3rd success promotes a guard

wrapped.invoke(date="next Wednesday", title="D")
# -> succeeds on the FIRST attempt — no failure is even recorded for it,
#    and reflect() is not called: the guard fixed the date before the call.
```

Guards are stored the same way recipes are — locally, per oracle — and can
be inspected/revoked with the CLI, or described as text for you to splice
into your own system prompt (ResilientForge never touches a prompt itself
— see the "Standing guards" section in
[`docs/architecture.md`](./docs/architecture.md)):

```bash
resilientforge guards list
resilientforge guards describe
```

Add invariants to catch failures that don't raise an exception (e.g. a
result that's missing a required field):

```python
from pydantic import BaseModel
from resilientforge import Invariant, wrap

class EventResult(BaseModel):
    title: str
    attendees: list[str]

wrapped = wrap(
    my_agent,
    invariants=[Invariant.from_pydantic_model("valid_event", EventResult)],
)
```

See [`docs/writing_invariants.md`](./docs/writing_invariants.md) for the
full interface (deterministic, Pydantic-schema, and LLM-judged invariants,
plus `on_violation="recover"|"abort"|"warn"`).

## Speculative branching: considering more than one fix

By default, a failed call gets exactly one candidate fix per attempt. Ask
for more with `num_branches`, and the tool is still only ever called for
real **once per attempt** — candidates are ranked by a recipe's real
`success_rate` when one exists, so this never risks a duplicate real-world
side effect:

```python
wrapped = wrap(create_event, reflect=reflect, num_branches=3)
```

If your tool has no problematic real-world effect regardless of which
arguments it's called with (a pure computation, a read-only lookup, a
validation — **not** anything that books, sends, charges, or deletes for
real), opt in to `side_effect_free=True` and ResilientForge will actually
try each candidate for real, in ranked order, until one fully passes your
invariants — genuine verification, not a guess:

```python
wrapped = wrap(compute_something, reflect=reflect, num_branches=3, side_effect_free=True)
```

See the "Speculative branching" section in
[`docs/architecture.md`](./docs/architecture.md) for the full design,
including why the flag isn't called `idempotent`.

## Sandboxed isolation: a hang or crash shouldn't take down your agent

`isolate=True` runs every real tool call in a freshly-spawned subprocess,
with an optional wall-clock deadline — a hang or a crash becomes a normal
recoverable failure instead of blocking or killing your process:

```python
wrapped = wrap(create_event, reflect=reflect, isolate=True, call_timeout=5.0)
```

This protects the *caller*, not the world the tool touches — it cannot
undo a real-world side effect the tool already performed before it hung
or crashed (no code-level sandbox can). It also requires `tool_fn` to be
picklable (checked eagerly, at `wrap()` time): a module-level function or
a bound method works out of the box; a locally-defined closure or lambda
needs `pip install resilientforge[isolation]` (adds `cloudpickle`, which
can serialize those too — falls back to it automatically when installed).
POSIX-only, best-effort resource ceilings are available too:

```python
wrapped = wrap(create_event, reflect=reflect, isolate=True,
                max_memory_mb=512, max_cpu_seconds=10)
```

See the "Sandboxed isolation" section in
[`docs/architecture.md`](./docs/architecture.md) for the full design,
including why resource caps are best-effort even on POSIX systems (an
empirically-confirmed, not hypothetical, caveat) and why this isn't
available through the LangGraph adapter.

## Local dashboard: the oracle's contents, in a browser

```bash
pip install resilientforge[dashboard]
resilientforge dashboard --oracle-path .resilientforge
```

A small, read-only, GET-only view of the same recipes/guards/failure
history the CLI already exposes — binds to `127.0.0.1` by default,
`fastapi`/`uvicorn` are only pulled in by this extra. Try it against seeded
example data:

```bash
python examples/dashboard_demo.py
resilientforge dashboard --oracle-path examples/.resilientforge_dashboard
```

![ResilientForge Dashboard](docs/images/dashboard.png)

Every number in that screenshot is real, produced by actually running the
recovery loop against `examples/dashboard_demo.py`'s seeded data, not
mocked for the picture — recipes and their success rates, which guards
have been promoted, and the raw failure history including the one call
ResilientForge correctly gave up on instead of fabricating a fix. See the
"Local dashboard" section in [`docs/architecture.md`](./docs/architecture.md)
for the full design.

## Observability: watch the recovery loop as it runs

The dashboard shows the oracle's contents *after the fact*. `metrics` is
a live counterpart — a callable that sees events as they actually
happen: every real tool call, how a recovery ultimately resolved, and
when a guard fires, gets promoted, or gets revoked.

```python
from resilientforge import wrap
from resilientforge.telemetry import LoggingMetricsHook

wrapped = wrap(create_event, reflect=reflect, metrics=LoggingMetricsHook())
```

`LoggingMetricsHook` is a zero-dependency reference implementation using
stdlib `logging` — configure the `resilientforge.metrics` logger the
normal way to send it wherever your logs already go, or write your own
`MetricsHook` (any `Callable[[MetricEvent], None]`) for Prometheus,
Datadog, OpenTelemetry, or anything else. See `examples/walkthrough_demo.py`
for it running live alongside the printed recovery narration, and the
"Observability" section in [`docs/architecture.md`](./docs/architecture.md)
for the full event list.

## Inspecting the oracle

```bash
resilientforge list                    # recipes learned so far
resilientforge list --failures         # raw failure history
resilientforge inspect <signature>     # full detail for one recipe (prefix match OK)
resilientforge prune --dry-run         # preview what pruning would remove
resilientforge stats                   # counts + resolution-status breakdown
resilientforge dashboard               # read-only web view of all of the above

resilientforge guards list             # standing guards
resilientforge guards inspect <tool> <argument>
resilientforge guards revoke <tool> <argument>
resilientforge guards prune --dry-run  # unattended maintenance -- see docs/architecture.md
resilientforge guards describe         # text to splice into your own system prompt
```

## Installation

```bash
pip install resilientforge               # raw Anthropic/OpenAI tool-loop adapter
pip install resilientforge[langgraph]    # + the LangGraph adapter
pip install resilientforge[dashboard]    # + the local web dashboard
pip install resilientforge[isolation]    # + isolate=True support for closures/lambdas
pip install resilientforge[semantic]     # + an optional semantic embedder (~1GB — see docs/architecture.md
                                          #   before installing: it didn't outperform the default on our own benchmark)
```

## Development

```bash
pip install -e ".[dev,langgraph,dashboard,isolation]"
pytest tests/unit tests/integration          # fast, no network — default CI gate
pytest tests/failure_injection                # the recovery-rate proof above
pytest -m live                                 # opt-in, real API calls
```

See [`CONTRIBUTING.md`](./CONTRIBUTING.md) for more, including how to add a
new failure-injection scenario.

## License

Apache 2.0 — see [`LICENSE`](./LICENSE).
