Metadata-Version: 2.4
Name: open-dream-rsi
Version: 0.3.8
Summary: Open-source implementation of the Dream-RSI (Recursive Self-Improvement through Evolving Worlds) architecture.
Author: patrykorwat
License-Expression: MIT
Project-URL: Homepage, https://github.com/patrykorwat/open-dream-rsi
Project-URL: Repository, https://github.com/patrykorwat/open-dream-rsi
Keywords: llm,agents,self-improvement,offline-rl,dreamer
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Open Dream-RSI

> **📄 Preprint:** *Open Dream-RSI: An Open-Source Library for Recursive
> Self-Improvement Around a Frozen LLM, with Replay-Gated Learned Policies and
> a Curated Knowledge Base* — P. Orwat, 2026.
> [**PDF**](paper/main.pdf) · [**LaTeX source**](paper/main.tex)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![CI](https://github.com/patrykorwat/open-dream-rsi/actions/workflows/ci.yml/badge.svg)](https://github.com/patrykorwat/open-dream-rsi/actions/workflows/ci.yml)
[![Tests](https://img.shields.io/badge/tests-359%20passing-brightgreen)](tests/)

## What is this?

Every AI agent pays the same tax over and over: it fails a task, works out
what went wrong, and then throws that understanding away when the session
ends. Open Dream-RSI keeps what was learned — **without touching the model**.

The LLM stays **frozen**. What improves is everything *around* it:

- the **exploration policy** (even re-written as Python code by the LLM
  itself, gated by replay before it is trusted),
- a library of **verified solutions** (recipes) reused as warm starts,
- **curated lessons** distilled from failures — every one of them promoted
  only after it *measurably* helps on replay,
- and the **world model** the policy is scored against: a Discovery Tree of
  every attempt, enriched with replay-safe error-class facts by the Sentinel
  engine.

Between runs the system "dreams": it replays thousands of strategies over
the recorded history in-process, at **zero API cost**, and keeps only what
beats the incumbent. An open implementation of *Recursive Self-Improvement
through Evolving Worlds* (Dream-RSI, Zheng et al.,
[arXiv:2609.14858](https://arxiv.org/abs/2609.14858)).

**Zero runtime dependencies** — Python 3.10+ stdlib only, ~9.8k lines of
library code, ~5.4k lines of tests, no build step.

**Not another memory plugin.** Experience-memory systems record what the
agent learned and reuse it as-is. ODR is an *evolving worlds* loop: every
attempt is recorded into a Discovery Tree, and between runs the system
dreams — replaying thousands of strategy variants against that recorded
world. Nothing is promoted on the model's word alone: a candidate policy,
recipe or lesson replaces the incumbent only after surviving counterfactual
replay.

**Evaluation policy:** this project makes performance claims on exactly
**one benchmark — goose × TravelPlanner**, public and scored by the
benchmark's own evaluators ([below](#benchmark-goose--travelplanner-public-externally-scored)).
The scripted suites (`bench`, `bench-policy`) are **deterministic
self-checks** — behavioural regression contracts for the machinery — and
carry no model-quality claim.

---

## Try it — no API key, no install

From a fresh clone, two commands show the whole loop on your machine. The
demo spins up a mock OpenAI server in-process, so nothing leaves your laptop:

```bash
git clone https://github.com/patrykorwat/open-dream-rsi.git
cd open-dream-rsi
python examples/live_loop_demo.py          # full dream cycle, ~10 s
python -m open_dream_rsi bench --cycles 10 --markdown   # deterministic self-check
```

The demo solves one task against a mock model that *fails on purpose* the
first time, records the failure into a Discovery Tree, "dreams" over the
recorded history between cycles, and reuses the verified fix afterwards —
watch the cycle lines: the first cycle pays the failure tax (6 API calls),
every later cycle solves from memory in 3. The benchmark prints the
cold-baseline comparison (53% fewer calls at equal solve quality, re-run on
every commit in CI).

**Then point it at a real model** — any OpenAI-compatible endpoint, keys from
the environment only, model name auto-detected: see [Quick start](#quick-start).

## How it works

```
                        ┌───────────────┐
                        │  Frozen LLM   │
                        └──────┬────────┘
                               │ proposals
              online loop      ▼      offline dream
        ┌──────────── DiscoveryTree ──────────── ReplaySimulator ──────────┐
        │                     │                          │                 │
        │              node.sentinel              counterfactual          │
        │              (Sentinel facts,           rollout scoring         │
        │               replay-safe)                    │                 │
        │                     │                         ▼                 │
        │                     └────→ Policy improvement ────→ next cycle  │
        │                                                                   │
   SENTINEL MODE (tool-execution layer)                                     │
        │          │             │                                          │
   error-class   nudge       permute-gate ──→ block tool calls             │
   ledger        (once/session)              past 80% of budget ───────────┘
```

1. **Online execution** — the agent attempts each task with the real LLM and
   a sandboxed verifier, appending every attempt to a per-task
   *Discovery Tree* (code, test feedback, score, one-line plan, and — when a
   Sentinel engine is wired — structured error-class facts and an explicit
   termination reason).
2. **Offline dreaming** — instead of paying for real-world rollouts, the
   agent "dreams" over the recorded history: thousands of strategy variants
   are scored in an in-process replay simulator at zero external-call cost.
   The best parameters, programs and recipes are persisted and steer the
   next cycle.

Beyond parameter-level dreaming, the loop closes section 3 of the paper
("dreaming with code"): each cycle the LLM may **rewrite the exploration
policy itself** as a small Python program, `choose_action(frontier, step)`.
Candidates are statically validated (AST gate), executed only in a hardened
subprocess sandbox, and scored by **counterfactual replay rollout** on the
recorded tree. A candidate replaces the incumbent only on evidence — and
the incumbent is re-scored on the *current* tree, not trusted at its stored
score. A crashing or cheating policy can never break the loop: expansion
falls back to the greedy baseline.

---

## Quick start

```bash
git clone https://github.com/patrykorwat/open-dream-rsi.git
cd open-dream-rsi
pip install -e .
```

Or without a clone: `pip install
git+https://github.com/patrykorwat/open-dream-rsi`. **PyPI releases are
tag-driven**: pushing a `v*` tag (e.g. `v0.3.8`) builds the sdist/wheel,
validates metadata, and publishes the `open-dream-rsi` project via trusted
publishing (`.github/workflows/publish.yml`) — no stored credentials.
Until the first tag is published, install from the repo. The name
`dreamrsi` on PyPI belongs to an unrelated project; this library is the
open implementation of the Dream-RSI paper (official repo:
[zhengkid/Dream-RSI](https://github.com/zhengkid/Dream-RSI)).

**Point it at a model — or don't.** Any OpenAI-compatible endpoint works;
keys are read only from the environment. The *model name is
auto-detected* from the endpoint (`GET /v1/models`), so pointing at a local
vLLM/Ollama needs only the base URL:

```bash
export OPENAI_BASE_URL=http://127.0.0.1:8000/v1   # vLLM / Ollama / any gateway
export OPENAI_API_KEY=***                         # any token for local servers
# no ODR_LLM_MODEL needed: the first model the endpoint serves is used.
# Pin ODR_LLM_MODEL only to select among several served models.
```

Queue a task in `tasks.json`:

```json
[{
  "task_id": "add1",
  "category": "math",
  "prompt": "Implement add(a, b) returning the sum.",
  "tests": [{"call": "add(2, 3)", "expected": 5}],
  "max_attempts": 3
}]
```

Run the supervisor — once, as a daemon, or as the recommended automated
`dream` cycle (one command that imports session evidence, runs the promotion
gate when the evidence justifies it, and publishes earned lessons as skills):

```bash
python -m open_dream_rsi --memory .dream_rsi dream \
    --sessions ~/.hermes/state.db --skills-out ~/.hermes/skills/odr-curated
python -m open_dream_rsi loop --tasks tasks.json --interval 300   # daemon mode
python -m open_dream_rsi loop --tasks tasks.json --once           # single cycle
python -m open_dream_rsi status                                   # what it learned
```

No key? Everything deterministic still runs: `bench`, `bench-policy`,
`gate-replay` and the dashboard all use a scripted mock client. A full
in-process walkthrough with a mock OpenAI server: `examples/live_loop_demo.py`.

Don't want the CLI at all? **The same loop runs as an MCP server inside your
existing coding agent** — that is the fastest way in:

```bash
python3 -m open_dream_rsi mcp --tasks ./tasks.json --memory ./.dream_rsi
```

…the exact config block for your agent lives in
[docs/integrations.md](docs/integrations.md).

---

## Plug in your favourite agent

The loop speaks **MCP** (Model Context Protocol). One config block and your
everyday agent can queue tasks for the dreamer, pull back verified recipes
and consult curated lessons — **zero extra LLM setup**: the dreamer resolves
its own brain (env → local goose config → localhost vLLM), and asks the
endpoint itself which model to use.

| Tool | What it does |
|---|---|
| `odr_status` | what the loop has learned (policies, recipes, events) |
| `odr_recipes` | best **verified** solution for a category — use as warm start |
| `odr_lessons` | curated failure lessons for a category / search query |
| `odr_add_task` | queue a task (prompt + tests, or prompt + criteria) — test-less tasks are completion-judged |
| `odr_run_once` | run one improvement cycle now (bounded API budget) |
| `odr_dream` | the automated cycle: import host session evidence, distill curated lessons, run the promotion gate, publish skills |

Tools are registered prefixed per client (e.g. Hermes
`mcp_open_dream_rsi_odr_run_once`). **No model calls tools on its own
initiative** — trigger them via a recipe's `instructions:`, a hook, or an
explicit ask ("check odr_recipes before you try again"). The `AGENTS.md`
prompting block that makes the loop proactive lives in
[docs/integrations.md](docs/integrations.md).

Per-host setup — OpenCode, Goose (auto-setup script), Hermes (incl. Docker),
Codex CLI, Claude Code, Cowork/claude.ai connectors, DeepSeek Harness — plus
the zero-configuration brain chain and troubleshooting:
**[docs/integrations.md](docs/integrations.md)**.

---

## Providers

| Provider | Preset | Base URL | Key env var |
|---|---|---|---|
| OpenAI | `openai` | `https://api.openai.com/v1` | `OPENAI_API_KEY` |
| Cursor (via local proxy) | `cursor` | `http://127.0.0.1:3000/v1` | `CURSOR_API_KEY` (optional) |
| Cursor Agent CLI (headless, your login) | `cursor-agent` | — (no HTTP) | — |
| Local (vLLM / Ollama / LM Studio) | `local` | `http://127.0.0.1:8000/v1` | `OPENAI_API_KEY` |
| Borrowed from goose config | `goose` | resolved from `~/.config/goose` | — |

Precedence is always: explicit overrides → environment → **endpoint model
discovery** (first id from `GET /v1/models`) → preset defaults; a pinned
model the endpoint no longer serves self-heals by re-discovering once.
Cursor has no public OpenAI-compatible chat endpoint — the `cursor` preset
targets a local proxy over the Cursor login, and since 0.3.0 the default
zero-proxy brain is the `cursor-agent` CLI. Full prose and CLI install:
**[docs/integrations.md](docs/integrations.md)**.

Library-level use (no CLI):

```python
from open_dream_rsi import (DiscoveryTree, DreamEngine, ReplaySimulator,
                            DreamAgent, LLMConfig, OpenAICompatibleClient)

client = OpenAICompatibleClient(LLMConfig.from_preset("local"))
tree = DiscoveryTree()
dreamer = DreamEngine(simulator=ReplaySimulator(tree))
dreamer.run_offline_optimization(iterations=100)
agent = DreamAgent(dreamer=dreamer, client=client)
```

---

## What persists between runs (`--memory` dir)

| Artifact | File | Effect |
|---|---|---|
| Dreamed policy parameters | `policies.json` | next cycle starts from the last one's optimum |
| Promoted policy programs | `policy_codes.json` | LLM-written `choose_action` steers expansion |
| Recipe library | `recipes.json` | verified solutions replayed as warm starts |
| Knowledge base | `lessons.json` | curated, replay-gated lessons (declarative text) |
| Discovery trees | `trees/` | offline dreaming always has history; nodes optionally carry replay-safe Sentinel facts |
| Audit log | `events.jsonl` | append-only record of every decision |

Switches worth knowing: `--no-policy-code`, `--no-knowledge`, `--no-thoughts`,
`--no-judge` ablate the four optional machinery layers; `--evolution-stop`
(opt-in) freezes policy regeneration after an anytime-valid betting-martingale
alarm — no promotion evidence left in a category stops the spending, the last
promoted policy stays active; `--budget N` caps API calls per cycle (dreaming
stays free).

---

## Automated dreaming: when does the loop actually run?

The seam is `dream_once()` (CLI `odr dream`, MCP `odr_dream`, or a host
hook). **You do not set a schedule — the module decides the tier per call**,
so firing it after every task is safe and costs nothing extra when there is
nothing new to learn:

| Tier | Triggered when | What runs |
|---|---|---|
| **maintenance** | quiet call (no new evidence, recent last dream) | ingest scan (cheap), ALM stale/supersede/archive/GC, skills republish — **zero LLM** |
| **full** | `min_new_evidence` (default 3) new failure-shaped sessions, OR `max_age_seconds` (default 6 h) since the last full dream | + online attempts, dreaming, paired-replay promotion gate (`budget`, default 20 API calls) |

A busy day dreams a few times, a quiet day once, an idle day never;
concurrent calls collapse on a single-instance lock. **Probes are queued
automatically**: every failure-shaped session appends its *own request* to
`tasks.json` as a `probe:<session>` task (`max_attempts=1`, appended after
manual tasks), so the promotion gate never depends on hand-written task
files — the evidence you already produced is the counterparty.

For a mass install the whole thing is two artifacts, no host glue:

```bash
python3 -m pip install git+https://github.com/patrykorwat/open-dream-rsi  # library + CLI + MCP server
cp -r plugins/hermes_odr_trigger $HERMES_HOME/plugins/odr-trigger
hermes plugins enable odr-trigger                # restart to load the hook
# optional overrides:
hermes config set plugins.entries.odr-trigger.settings.provider local
hermes config set plugins.entries.odr-trigger.settings.base_url http://.../v1
```

---

## Benchmark: goose × TravelPlanner (public, externally scored)

This is the **only benchmark** in this project — everything else below the
heading line is machinery self-check. A real agent (goose v1.53) plans real
itineraries against the official **TravelPlanner** offline database
(`osunlp/TravelPlanner`) over a stdlib MCP sandbox, scored by the benchmark's
*own* commonsense and hard-constraint evaluators (nothing in the scoring path
is ours), on the model `Qwen3.8-Flash-Next` served by vLLM. Same endpoint and
model across every study in this repo and the paper.

Frozen protocol: iterations only on `train.csv` (45 tasks); one preregistered
eval pass on official validation rows 102–181 (79 tasks, 60 hard),
configuration unchanged from the dev split; official test split never
touched. Arms interleaved task-by-task per pass (paired, McNemar-ready).
The gate arm is *choice removal*, not persuasion: past 36 of 45 tool calls
the sandbox refuses to serve (all text channels off). The gate alone is the
entire arm — no policy, recipe or lesson text enters these episodes — so the
delta measures the fixed call-cutoff intervention itself, not learned
content; the learned machinery is validated separately on the scripted
suites above.

| split | arm | delivered plans | commonsense pass | hard pass | final pass |
|---|---|---|---|---|---|
| train (45) | cold | 15/45 (33%) | 4 | 3 | 6.7% |
| train (45) | gate | 41/45 (91%) | 18 | 9 | 20.0% |
| **eval (79)** | cold | 18/79 (23%) | 12 | 6 | 7.6% |
| **eval (79)** | gate | **74/79 (94%)** | 40 | 17 | **21.5%** |

McNemar on paired delivery: eval 56↔0 discordant, p=1.4e-17 (train 28↔2,
p=4.3e-7). 95% Wilson intervals on the eval split: delivery 22.8%
[14.9, 33.2] → 93.7% [86.0, 97.3]; final pass 7.6% [3.5, 15.6] → 21.5%
[13.9, 31.8]. 79 eval tasks, one model, one benchmark: the delivery
contrast is the decisive effect; the final-pass point estimates carry wide
intervals and no cross-model replication yet.

![goose × TravelPlanner: delivery + failure anatomy](docs/figures/fig_tp.png)

Why it works: **every single undelivered cold episode (91/91 across both
splits) died pinned at the harness call cap with the sandbox data already
in hand** — the binding failure under budget pressure is *termination*, not
error recovery, and three text-in-context channels measured earlier
(pull tool, Stop-hook note, budget nudge) converted zero of those episodes.
Removing the option to keep exploring is the only lever that moved delivery.
Honest caveats: the *choice* of gate over persuasion was made knowing
results on the earlier 101-task validation pool (selection era, archived —
the eval pass is a preregistered confirmation, not a first exposure); the
gate's 5 residual eval failures are episodes that called through the
refusals to the cap anyway. Everything behind these numbers is in this
directory: `benchmarks/travelplanner/` — the sandbox MCP server, the
episode runner, the official-evaluator wrapper, the exact split CSVs and
the raw per-episode records. `python3 score.py results/submissions/…`
reproduces the table above to the task from the committed records; the
directory README documents prerequisites and the one-command rerun. The
figure re-renders from `fixtures/tp_v3_summary.json` via
`paper/make_figures.py`. Paper §Evaluation.

---

## Machinery self-checks (deterministic, key-free)

Not benchmarks — behavioural regression contracts: a scripted solver makes
the model arm a constant, so every delta is attributable to the library's
gates, sandboxes, promotion and retrieval logic. They exist so a broken
mechanism fails CI instead of silently mutating a published number, and they
re-run from a fresh clone in seconds.

### `bench` — the dreaming loop against a cold baseline

| arm | solves | API calls | calls / task·cycle | dream its |
|---|---|---|---|---|
| cold_baseline (fresh memory each cycle) | 50 | 150 | 3.0 | 0 |
| dream_rsi_loop (policies + recipes + dreaming) | 50 | 71 | 1.42 | 3000 |

**53% fewer API calls at equal solve quality** (re-measured on this commit:
identical to the published table).

```bash
python -m open_dream_rsi bench --cycles 10 --markdown
```

### `bench-policy` — decoy traps (exploration contracts)

Solve-rate self-checks saturate on easy suites and cannot tell good
exploration from luck, so `bench-policy` hides each fix behind a
*low-scoring* branch (passes 1/3 tests) past a *plausible decoy* (passes 2/3
forever). A score-greedy loop locks onto the decoy and starves; escaping
requires structural exploration. Six arms, one scripted solver, one budget:

| arm | solves | solve rate | API calls | calls / solve | policy calls | curator calls |
|---|---|---|---|---|---|---|
| greedy | 0/120 | 0% | 480 | ∞ | 0 | 0 |
| epsilon_greedy | 29/120 | 24% | 405 | 13.97 | 0 | 0 |
| **evolved_policy** | **93/120** | **78%** | **249** | **2.68** | 26 | 0 |
| **knowledge_curator** | **87/120** | **72%** | 330 | 3.79 | 0 | 27 |
| **thought_guided** | **120/120** | **100%** | **164** | **1.37** | 0 | 0 |
| **sentinel_guided** | **102/120** | **85%** | 219 | 2.15 | 15 | 0 |

```bash
python -m open_dream_rsi bench-policy --cycles 8 --format md
```

The per-arm explanations and ablations (ledger-only, engine-off), the pinned
invariants, and the `gate-replay` promotion-rule replayer:
**[docs/self-checks.md](docs/self-checks.md)**.

![Decoy-trap suite: per-cycle solve rate, six arms](docs/screenshots/odr_policy_bench.png)

---

## What the loop learned about memory (honest results)

The knowledge curator distils verifier failures into short, validated,
deduplicated **lessons** — pure text, never executed. Live-model replay,
however, proved that fresh lessons are *hazardous by default*:

- Structurally perfect lessons (passed every schema/caps gate) dropped
  solve-rate **18/20 → 6/20** on a greedy decoder: "keep checking"-style
  advice turns greedy decoding into an over-exploration loop.
- The same facts rewritten as **declarative background** (no imperative
  verbs, explicit stop clause): 18/20 — indistinguishable from cold
  (McNemar p=1.0). The hazard is imperative mood, not content.
- Seven further published injection strategies (abstract workflows,
  exemplars, evidence certificates, reactive-on-error notes,
  end-of-prompt position) scored **zero** paired gains against a
  contemporaneous cold baseline; moving safe text to the end of the prompt
  was actively harmful (11/20, p=0.008 — recency makes the model *act* on
  facts that were inert mid-prompt).

The shipped design encodes all of that: lessons are `staging` candidates
that must pass a paired-replay gate (net ≥ 1, **zero** solve→fail
regressions, explicit stop clause) before any prompt sees them; proposal
prompts frame the KB as declarative background; and a per-category
**headroom verdict** (skip curation when recent solve-rate ≥ 0.7 *and*
calls/solve ≤ 5) spends zero curator/gate calls where the cold baseline
already wins. Guidance pays a perturbation tax — the loop only pays it
where there is measurable headroom to spend it against.

### Verdicts must be replayable (why lessons anchor on tool calls)

Tool results are the only evidence class with cheap, objective ground
truth — exit code, timeout, empty payload, retrieved content — so the
paired-replay gate can re-execute the same moment and decide. Prose
quality ("was this plan good?") has no such oracle: an LLM judge cannot
be re-run into a decision, only re-sampled into noise — and that noise
is measured, not hypothetical. Trusting freshly distilled lessons
*without* replay dropped a greedy solver from 18/20 to 6/20, and the one
noise-promoted lesson that ever reached production (an anodyne "answer
more directly") won its coin flip precisely because it rarely hurt —
hence the confirmed-strict-win promotion. The loop therefore learns
where truth is cheap and treats everything else as unevidenced material,
not knowledge. The strongest non-tool signal — the user's own in-session
corrections — is deliberately not auto-imported yet; it is tracked as
the next first-class channel under the same staging → replay-gate
discipline.

---

## Learning vs lifecycle vs history

The repository keeps four planes strictly apart:

| Plane | Owner | Question it answers |
|---|---|---|
| **Learning** | Dream-RSI pipeline (dreamer, replay gate, lesson gate) | Does this artifact *deserve* to be active? |
| **Current artifact state** | `ArtifactLifecycleManager` (`lifecycle.py`) | What is true about this artifact *now*? |
| **Transition history** | append-only `artifacts/events.jsonl` | How did it get here? |
| **Immutable evidence** | Discovery Trees + the event logs | Why do these artifacts exist, and what world produced them? |

The ALM manages the four artifact types (`policy_parameters`,
`policy_program`, `recipe`, `lesson`) through an explicit state machine
(`CANDIDATE → VALIDATED → ACTIVE → STALE/SUPERSEDED/QUARANTINED → ARCHIVED`,
terminal `REJECTED`), with hard rules: activation is *only* an explicit
promotion call carrying the evidence; every state change appends exactly
one immutable event **before** exposing the new state; the materialized
view must always equal `rebuild_artifact_state(events)` (a hand-edited view
is rebuilt from the log, never trusted); merges create NEW artifacts with
lineage and never mutate sources; deletion never prunes the event log.
Discovery Trees and audit events are **evidence, not artifacts** — the ALM
refuses to register them. Inspect the store:
`odr artifacts [--type lesson --state ACTIVE --history]`.

---

## Importing real sessions: Hermes and Cursor (curator's second door)

The loop distils lessons from failures it produces itself. `sessions.py`
opens a second legitimate door: **transcripts from host agents are evidence
you already paid for**. Imports feed the *same* pipeline as loop-collected
failures — structural gate, staging-by-default, paired replay promotion.
Nothing imported is trusted; activation is still earned. Transcripts are
treated as **hostile input**: opened read-only, credential-scrubbed, parsed
defensively — and lessons that never passed the promotion gate **never reach
a prompt**.

### From Hermes sessions

Hermes persists every session to `<HERMES_HOME>/state.db` (SQLite:
`sessions` + `messages`). Export, then distil:

```bash
python -m open_dream_rsi --memory .dream_rsi sessions export \
    --source hermes --db ~/.hermes/state.db --limit 50 \
    --out episodes.jsonl

# inspect what WOULD be distilled — no LLM calls, no writes:
python -m open_dream_rsi --memory .dream_rsi sessions distill \
    episodes.jsonl --dry-run --limit 5

# real run: staging lessons into the KB (per-category = episode cwd name)
python -m open_dream_rsi --memory .dream_rsi sessions distill episodes.jsonl
```

The next `odr loop` cycle gates the staging entries exactly like
loop-collected ones; `odr artifacts --type lesson --history` shows the
lifecycle events behind every activation.

### User corrections (natural-language evidence, issue #6)

Host sessions also carry corrections stated in prose — "nie tak, użyj
istniejącego klienta", "don't create another HTTP client" — which error-marker
scanning never sees. `corrections.py` extracts them deterministically (no
extra LLM call per message), pairs each with the assistant turn it targets,
and labels four classes: `correction`, `approach_redirect`, `preference`,
`new_requirement`. Only the first three are distillation-ready; ordinary
follow-ups, new requirements and ambiguous negatives stay unclassified.
Provenance survives end to end: evidence actions read
`user_correction:<host>:<session>#<msg_index>`, and each record keeps source
host, session id, workspace and whether a later turn suggests the session
recovered (`outcome: confirmed|unverified`). Message text arrives already
secret-redacted from the readers; nothing is added back.

The trust model is unchanged: **a user correction is evidence worth testing,
not verified truth.** Candidates are deduplicated deterministically (Jaccard
on content words) *before* any paid curator call — repeats within one episode
do not strengthen evidence — then flow through the same curator prompt (with
an explicit "user wording is evidence, not verified truth" note), enter
`STAGING`, and face the same paired-replay gate. A correction-only session is
no longer skipped: its probe task is the **pre-correction** user request, so
the user's corrected answer never leaks into replay context. Every extraction
is auditable via the `user_correction_found` event in `events.jsonl`; source
model identity is provenance, never a trust score, and there is no
cross-model evaluation matrix (one target: the configured runtime model).

### Backfill: re-distilling burned sessions

A dream cycle consumes a host session the first time it distills it. While
the endpoint was broken (404 churn, connection refused, empty payloads), a
session could be marked seen while producing ZERO lessons: `seen_sessions`
grew, the knowledge base did not — the evidence was burned. Since 0.3.0
transport failures defer sessions instead of consuming them, but sessions
burned by the old behaviour stay invisible. `scripts/backfill_history.py`
audits that damage and repairs it:

```bash
# damage report (dry-run; reads only session ids + cwd, never transcripts):
python scripts/backfill_history.py plan --memory .dream_rsi \
    --state-db ~/.hermes/state.db

# make burned sessions unseen again (timestamped backup written first):
python scripts/backfill_history.py requeue --memory .dream_rsi \
    --state-db ~/.hermes/state.db --category radoskoppl --apply

# trim the replay-probe backlog to the newest N (manual tasks always kept):
python scripts/backfill_history.py prune-probes 60 --memory .dream_rsi --apply

# run cycles back-to-back through the real CLI (the dream lock collapses
# overlaps; transport failures defer, never burn):
python scripts/backfill_history.py drain 5 --memory .dream_rsi \
    --state-db ~/.hermes/state.db --provider local --budget 200
```

A session counts as burned when it is marked seen, still present in the
store, and never produced a single lesson. Rejections meaning "no failure
evidence in this transcript" are excluded — requeueing those would loop
forever with zero yield. Tests: `tests/test_backfill_history.py`.

### From Cursor chats

Cursor keeps conversations in `globalStorage/state.vscdb` (Linux:
`~/.config/Cursor/User`, macOS: `~/Library/Application Support/Cursor/User`,
Windows: `%APPDATA%\Cursor\User`), with `workspaceStorage` mapping chats to
project folders for the category. The reader handles the undocumented
layout and the bubble-ordering rules; re-verify after major Cursor
upgrades.

```bash
python -m open_dream_rsi --memory .dream_rsi sessions export \
    --source cursor --out episodes.jsonl --limit 100
python -m open_dream_rsi --memory .dream_rsi sessions distill episodes.jsonl
```

Notes: the live DB is write-ahead-logged — `immutable=1` may miss the
newest chats, so quit Cursor or copy `state.vscdb`+`-wal`+`-shm` together
and point at the copy for a complete snapshot. Chat history contains
everything ever pasted into it: export redacts credential-shaped substrings
by default (`--no-redact` exists for local processing only) — review
`episodes.jsonl` before sharing it.

### Curator lessons as Hermes skills

```bash
python -m open_dream_rsi --memory .dream_rsi skills export \
    --out ~/.hermes/skills/odr-curated
```

Renders only **active** (promotion-gated) lessons into Hermes-style skills —
one directory per category, declarative bullets, never imperative commands
(the measured hazard). The export is one-way and regenerated wholesale:
edit `lessons.json` (or distil again), not the skills.

### Sharing lessons: git-backed stores

`share.py` turns curated knowledge into a portable, verifiable store that
other instances (or other people) can consume — a directory of JSON files,
optionally a git checkout, so GitHub is just storage:

```bash
# one-time: point the store at a repo (clone/pull handled for you)
python -m open_dream_rsi --memory .dream_rsi lessons export \
    --store git@github.com:you/dream-rsi-lessons.git

# on the consumer side — someone else's store becomes your staging:
python -m open_dream_rsi --memory .dream_rsi lessons import \
    --store https://github.com/you/dream-rsi-lessons.git
```

Trust model: **export ships only what earned activation** (staging and
rejected lessons excluded by construction, secrets scrubbed, sha256
manifest); **import lands as staging, never active** (checksum mismatch
aborts the whole import — tampering detected, not trusted; each lesson
still needs *your* paired-replay gate to reach a prompt). A local directory
works identically (`--store ./store`) — git is just the transport.

### Full Hermes wiring (what exists today)

| Path | Direction | What it does |
|---|---|---|
| `odr mcp` (stdio) / `--http` | Hermes → ODR | Hermes calls `odr_run_once` / `odr_dream` / `odr_status` as tools |
| `plugins/hermes_sentinel/` | Hermes → ODR engine | Sentinel observes tool results, annotates, gates |
| `plugins/hermes_odr_trigger/` | Hermes → ODR engine | session-end hook spawns the detached `odr dream` cycle (no shell/cron) |
| `plugins/deepseek_harness/` | DeepSeek Harness (`dsh`) → ODR | Cordis overlay: the ODR MCP server behind `@deepseek-ai/dsh-mcp-client` |
| `hooks/` + `.cursor-plugin/` | Cursor → ODR (no MCP) | Plugin manifest (GitHub import) + hooks (sentinel notes, session-end dream; default brain: the cursor-agent CLI) + `task add` queue — for Cursor installs where MCP servers are blocked |
| `odr sessions export/distill` | Hermes → curator | session history becomes staging lessons |
| `odr skills export` | curator → Hermes | active lessons become discovered SKILL.md files |
| `odr dream` | any trigger | one automated cycle: evidence → gate → ALM → skills |

---

## Security model

Candidate code — both task solutions and policy programs — runs in a
hardened subprocess (`open_dream_rsi/sandbox.py`): `python -I`, scrubbed
environment (only `PATH` — API keys never cross the wall), a timeout that
kills the **entire process group**, and POSIX address-space / CPU /
thread-count limits. Policy code additionally passes an AST gate *before*
any spawn: required entry point, no imports, no dunder access, no
`__builtins__` reference (name plus constructed-key indexing is itself an
escape route).

**This is defence in depth, not a container.** The subprocess does not
restrict filesystem or network access, and an AST gate cannot be provably
complete against code that is not statically gated. For hostile task
sources set `ODR_SANDBOX_CMD` to a
[bubblewrap](https://github.com/containers/bubblewrap)/nsjail wrapper
(e.g. `bwrap --unshare-all --die-with-parent --ro-bind / /`) or run the
loop inside a container.

On non-POSIX hosts (Windows) the same `python -I` child runs, but there are
no rlimits and the timeout kills only the direct child — treat Windows as
out of scope for hostile task sources. When `ODR_SANDBOX_CMD` is unset,
each fresh process logs a one-time warning naming exactly this gap.

---

## Live dashboard

One command, no dependencies, stdlib http server:

```bash
python -m open_dream_rsi dashboard                   # mock LLM, port 8765
python -m open_dream_rsi dashboard --provider cursor # real model
python -m open_dream_rsi dashboard --host 0.0.0.0    # LAN access
```

Dark single-page UI: KPIs (cycles / solved / API budget / dream iterations),
live event feed, per-task attempt boards with score bars and code diff-downs,
dreamed-policy gauges per category and the learned recipe library.

The dashboard is a plain unauthenticated stdlib HTTP server whose control
routes (`/api/pause`, `/api/reset`) mutate the running loop — keep `--host`
on loopback unless the LAN is trusted.

---

## Positioning

- **The paper this implements:** Dream-RSI
  ([arXiv:2609.14858](https://arxiv.org/abs/2609.14858)); official repo
  [zhengkid/Dream-RSI](https://github.com/zhengkid/Dream-RSI), explainer
  [dream-rsi.com](https://dream-rsi.com/).
- **Peer implementations** (this repo is an independent open one, not the
  first): [TheAstrayDev/dream-rsi-sdk](https://github.com/TheAstrayDev/dream-rsi-sdk)
  (adapter SDK, strict replay; LLM-written policies on their roadmap),
  [robinber/dream-rsi-spark](https://github.com/robinber/dream-rsi-spark)
  (local Qwen + CUDA kernel exploration), plus skill variants
  ([hermes-dream-rsi](https://github.com/lesterppo/hermes-dream-rsi),
  [dream-rsi-skill](https://github.com/Harkit2004/dream-rsi-skill),
  [Pi-RSI](https://github.com/mailbobg/Pi-RSI)).
  What differentiates this repo: an **always-on autonomous supervisor**
  with budget guards, **LLM-written exploration policies** gated by
  counterfactual replay (section 3 closed), a **public-benchmark
  measurement** (goose × TravelPlanner) rather than self-scored claims, a
  deterministic decoy-trap self-check suite, persistent cross-run memory,
  and a **negative-results section on memory** that the loop itself
  enforces.
- **OpenRSI / Frontis-MA1** ([FrontisAI/OpenRSI](https://github.com/FrontisAI/OpenRSI))
  is a different layer: they post-train *weights*; this library optimises
  the *exploration policy around a frozen model* — no training, no GPU, no
  weight access. Complementary: their trained improver could be the frozen
  LLM behind our client.

---

## Module architecture

| Module | Role |
|---|---|
| `core.tree` | DiscoveryTree — hypotheses, results, actions, thoughts, Sentinel facts, termination reason |
| `core.simulator` | replay simulation without touching the environment |
| `core.dreamer` | offline optimisation over recorded history |
| `core.agent` | LLM agent driven by the rewarded policy |
| `core.policygen` | policy generation, AST gate, rollout scoring, promotion |
| `core.curator` | lesson distillation, headroom verdict, gate verdict |
| `core.judge` | completion judge for test-less (`criteria`) tasks |
| `loop` | `AutoRSIRuntime` — the autonomous supervisor (opt-in `sentinel_engine`) |
| `dream` | `dream_once` — automated cadence seam: evidence ingest, tier decision, gate + ALM, skills publish; lock-guarded |
| `lifecycle` | `ArtifactLifecycleManager` — state machine, event-sourced history, lineage, GC |
| `sessions` | Hermes `state.db` / Cursor `state.vscdb` readers, redaction, lessons→skills bridge |
| `memory` | `DreamMemory` — policies, recipes, lessons, trees, events |
| `sandbox` | `run_isolated` — the single untrusted-code execution boundary |
| `tools` | `CodeVerifier` — sandboxed verification of candidate solutions |
| `llm` | OpenAI-compatible client (stdlib `urllib` transport; model discovery + 404 self-heal) |
| `cli` | `loop / status / dashboard / bench / bench-policy / gate-replay / mcp / proxy / sentinel` |
| `mcp`, `mcp_server` | MCP stdio server for loop tools and sentinel checks |
| `proxy` | loopback OpenAI-compatible proxy borrowing goose's upstream |
| `dashboard` | zero-dependency live web UI |
| `gate_replay` | offline validation of the lesson promotion rule |
| `evolution` | anytime-valid stopper for policy evolution (betting martingale over promotion outcomes; freezes regeneration, keeps the last promoted policy) |
| `sentinel` | host-agnostic error-class engine + `SentinelObservation` world contract |
| `utils.goose` | goose config resolver (CLI + desktop dialects, keychain) |
| `plugins/*` | Hermes / Claude Code / goose / DeepSeek Harness / Cursor adapters |
| `benchmarks/travelplanner` | **the benchmark**: sandbox MCP server, episode runner, official-evaluator wrapper, split CSVs, raw published episode records |

## Tests

```bash
python -m unittest discover -s tests      # full suite, ~2 min
python scripts/fast_tests.py              # fast subset (slow bench/sandbox modules excluded), ~15 s
```

359 tests at this commit, re-counted by CI on every push (the badge above
shows the last green run, not a live count). Stdlib-only: benchmark
behavioural contracts
(greedy must solve nothing; the gated arm must dominate ε; the rollout must
be prefix-only), sandbox escape regressions, gate semantics, MCP protocol,
goose config dialects, sentinel channel semantics (nudge fires once per
budget, merges with recurrence notes, clean calls count toward the budget;
the permute-gate serves until ceil(0.8·budget) then refuses forever, per
session; the 30-day production replay must reproduce its recorded note count
exactly), model discovery and stale-pin self-heal contracts, and the
structured seam (cross-session counts never reach `to_world_dict()`; the
frontier-feature attribution pattern — beats greedy with metadata, ties
without; the vendored Hermes engine pinned byte-for-byte to upstream by
`test_vendored_engine_matches_upstream`).

## License

MIT — see [LICENSE](LICENSE).
