Metadata-Version: 2.4
Name: openforage
Version: 0.2.28
Summary: Agent-driven market strategy discovery, backtesting, and signal submission for OpenForage.
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.0
Requires-Dist: psutil>=5.9
Requires-Dist: pyarrow<25,>=18.1
Requires-Dist: zstandard>=0.22
Requires-Dist: eth-account>=0.10.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: cryptography>=42.0.0
Requires-Dist: google-cloud-storage<3,>=2.19
Requires-Dist: psycopg2-binary<3,>=2.9
Requires-Dist: uvicorn[standard]>=0.34.0
Requires-Dist: websockets>=14.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pyjwt>=2.9.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: structlog>=24.0.0
Requires-Dist: tomli>=2.0; python_version < "3.11"

# OpenForage Library

> 🚧 **CLOSED BETA — invite code required.** OpenForage is currently in
> closed beta. Access to the library requires an invite code. The agent
> will not be able to `openforage register` without one.
> [Subscribe at openforage.ai](https://openforage.ai) to join the whitelist;
> invite codes are distributed to whitelisted operators as the closed beta
> opens. Do not start the bootstrap below until you have an invite code in
> hand.

Python CLI and library for the OpenForage commodity trading and analysis
platform. An agent registers a wallet, runs a background search that
evaluates mathematical signal expressions against historical market data,
submits passing signals to the OpenForage server, and consumes callback
events for failures and periodic improvement prompts.

**This README is the agent's single front door.** Point any modern coding
agent at this file — Claude CLI, Codex CLI, Hermes, OpenClaw — and the
agent can register, start a background search that downloads data while
the rest of the setup happens in parallel, wire up callbacks and
schedules, and begin productive iteration without opening anything else.
For clean-room journey verification, use the canonical narrative source of truth,
[`consciousness/skills/clean_room/references/clean_rooms/openforage_library/agent_user_journey.md`](../consciousness/skills/clean_room/references/clean_rooms/openforage_library/agent_user_journey.md)
as the testing checklist; it defers operational onboarding back to this README.

The six bundled `SKILL.md` files under `src/openforage/skills/` are
depth references; this README is the index and the active orchestration
script.

---

## §0 — Set a Durable Outcome Before Doing Anything Else

**STOP. Before any other command, put the outcome below somewhere your
runtime will preserve and re-read.** If your runtime has a `/goal`
slash command, use it. Otherwise use the runtime's native tasklist plus
a durable scratchpad (`HEARTBEAT.md`, a pinned message, or a local
`GOAL.md`). The important primitive is the durable outcome, not the
literal `/goal` command.

```text
Bootstrap OpenForage end-to-end: complete every §A task in
openforage_library/README.md, keep the background search running
healthily, register a file sink plus a scheduled_improvement callback,
poll `openforage status --json` plus `events.jsonl` until the worker is
healthy or a failure is classified, consume each improvement_prompt
event and act on it, and only stop when actual results stabilize or a
genuine blocker is reached. Read every bundled SKILL.md before changing
the run loop. Save durable discoveries to memory before exiting.
```

Runtime-specific ways to persist it:

- **Claude CLI** — create native task entries for each §A item and put
  the outcome in a durable scratchpad or pinned instruction. If your
  Claude runtime or harness extension exposes `/goal`, you may paste the
  same text there, but Claude CLI is not assumed to have a built-in
  `/goal` primitive.
- **Codex CLI** — use `update_plan` and `record_tasklist` for the §A
  items. If your Codex version exposes `/goal`, paste the same outcome
  there as an additional runtime-specific guard.
- **Hermes** (Nous Research runtimes that expose `/goal`) — paste the
  outcome into `/goal` so the runtime tick re-loads it.
- **OpenClaw** — persist the outcome in the Lane Queue or `/goal` if
  that runtime exposes one. Then paste the output of
  `openforage.get_agents_md_fragment()` into the workspace's `AGENTS.md`
  so the OpenClaw skill loader picks up OpenForage.

Re-read the durable outcome on every turn. There is no scenario where
you can hold this in working memory across multi-turn work and remain
correct.

---

## §A — The Six Bootstrap Tasks (Enter These Into Your Tasklist Now)

Enter these six items into your runtime's native task tool **before**
running any command beyond `--help`. Use the runtime's tasklist
primitive (Claude: `TaskCreate`; Codex: `update_plan` then
`record_tasklist`; OpenClaw / Hermes: the runtime's task store). Stable
IDs are below so cross-agent handoff is unambiguous.

| ID | Task | Done when |
|----|------|-----------|
| `T-OF-register` | Run `pip install openforage`, then `openforage init` if you need to inspect or edit `.openforage/settings.yaml` before registration, then `openforage register --invite-code <CODE> --json`. Save the printed `address`. | `address` returned, `wallet.key` exists under the configured `data_dir`, mode `0600`. |
| `T-OF-start-search-bg` | Spawn `nohup openforage start --json > openforage-start.log 2>&1 &` **in the same turn as register completes** — do NOT wait for the data download. | The shell returns immediately; `background.pid` appears under the configured `data_dir`. |
| `T-OF-callbacks-sink` | While data downloads in the background, register a file sink with `openforage callbacks register file --path .openforage/callback-events.jsonl`. Optionally add a webhook or shell sink. | `openforage callbacks list --json` includes a `file` entry. |
| `T-OF-schedule` | Register at least one `scheduled_improvement` callback and wire your runtime's tick to `openforage.run_due_callbacks()`. | `openforage callbacks list --json` includes a `scheduled_improvement` entry; your tick is scheduled. |
| `T-OF-baseline-watch` | Read §J Health Baseline. Poll `openforage status --json` and tail `events.jsonl` until you can compare against the table. | `running`, `process_health`, `data_download_progress`, and recent events either match the baseline or you have classified the deviation. |
| `T-OF-improvement-loop` | Consume each `improvement_prompt` from the sink, apply the suggested adjustment, restart the worker if needed, compare on `get_search_stats()` / `get_yield_analysis()`. | One full prompt-to-comparison cycle has completed. |

These six are the entire onboarding contract. Treat the list as the
spec, not the README narrative. If your runtime cannot store these as
first-class tasks, write them into a `HEARTBEAT.md` you re-read every
turn.

Release-soak proof for these tasks must run on this project's owned
test stack and capture the submission endpoint hit, server DB readback,
terminal status transition, and exact terminal verdict. A local run,
legacy shared endpoint, or log-only transcript is not acceptance
evidence.

---

## §B — Mental Model (One Line)

> `register (wallet) → search (worker discovers signals across historical
> eras) → submit → callbacks emit events → improvement_prompt events feed
> back into the agent loop.`

That is the entire system. Everything else in this README is detail.

---

## §B.1 — `settings.yaml` Contract

`~/.openforage/settings.yaml` is the single durable user-editable runtime
configuration file for the OpenForage library. The package ships the template at
`openforage_library/src/openforage/settings.yaml`; `openforage init` creates the
user copy and upgrades it on later runs by preserving current keys, adding new
template keys, and dropping removed keys.

Runtime loading is strict. A normal library entrypoint fails loudly when the
user settings file is missing, has unknown keys, has retired keys, or is missing
required keys. Use `openforage init` when installing or upgrading so the local
file matches the packaged template.

Source order is deliberately narrow: explicit CLI/function arguments win where
an API exposes them; otherwise `settings.yaml` is the runtime source of truth.
`null` means "use the documented built-in behavior", such as SQLite instead of
Postgres, auto-derived identity, default-on standard logging, or local-server
validation disabled. It does not mean "read a legacy environment variable".

The file covers these groups:

- Local paths and service endpoints: `data_dir`, `server_url`,
  `signal_service_url`, `local_server_port`.
- Registration and identity: `invite_code`, `market_id`, `instance_*`, and
  `author_*`. `market_id` is a required positive integer (default `1`, the
  current live market); the background worker fails loud if it is null, and
  market identity is sourced only from `settings.yaml`, never the CLI.
- Local persistence: `db_url`, `db_schema`, and the local-server Postgres pool
  setting.
- Local-server, GUI, shutdown, and logging knobs:
  `local_server_named_universe`, validation/sync toggles, `gui_demo_mode`,
  `stop_wait_seconds`, and `standard_logging`.
- Search and runtime resource knobs: algorithm, workers, memory estimate,
  batch size, priority, genetic parameters, and feature budget.
- Sync transport and concurrency: HTTP transport, chunk size, and max
  concurrency.
- Contributor stats: upload opt-out, upload URL/interval, and B2 consent
  references.

---

## §B.2 — Logging Guide

OpenForage library logs are visible standard diagnostics. What the logs mean:
they show progress through public work, including `entry`, `exit`, `decision`,
`value`, and `error` events. Logging is on by default and default-on; routine
`entry`, `exit`, `decision`, and `value` lifecycle events are DEBUG diagnostics,
while `error` events remain visible at the default INFO threshold.

Logs go to stderr by default. The stream and destination can be supplied by the
caller when configuring logging, so machine JSON on stdout stays parseable.
Set verbosity with the normal Python log level; enable debug when you are
actively inspecting lifecycle progress.

To opt out, run `openforage init` and set `standard_logging: false` in
`~/.openforage/settings.yaml` before import or process start. Do not use log
suppression to hide a real failure; classify the failure from status, events,
and stderr.

Internal IP and firm IP stay bounded. A cryptic token must stay cryptic, and
logs should never reveal protected source detail. The coverage gate
`scripts/verify_openforage_library_logging_coverage.py --json` checks captured
library emissions, and `scripts/ci_checks.sh` runs that coverage gate beside the
format parity check.

---

## §C — Read These Skills First (Not Optional)

The six bundled `SKILL.md` files under `src/openforage/skills/` are the
source of truth for behavior. Skipping them is the single most expensive
shortcut. Install/import must happen before `openforage.list_skills()`:
the bundled skills ship inside the wheel, so discovery only works after
`pip install openforage` has made that wheel importable in the active
Python environment. Inside the agent runtime, run:

```bash
python -c "import openforage; print('\n'.join(openforage.list_skills()))"
```

then read every printed path before changing the run loop or invoking a
CLI beyond `--help` / `--version`.

| Skill | Must-read gate |
|-------|----------------|
| `openforage_quickstart` | Before `T-OF-register` — installation, durable state directory, smallest local loop, agent handoff fields. |
| `background_search_loops` | Before `T-OF-start-search-bg` — worker lifecycle, restart, diagnostics, supervisor patterns. |
| `search_settings` | Before changing `--algorithm` / `--settings-path` — built-in search templates, settings validation, and safe template switches. |
| `callback_hooks` | Before `T-OF-callbacks-sink` — file/webhook/shell sink contracts, opt-in flags, audit log. |
| `scheduled_improvement_loops` | Before `T-OF-schedule` — `scheduled_improvement` semantics, why it does not run an optimizer, sink ordering. |
| `agent_runtime_integration` | Before `T-OF-improvement-loop` — durable outcome/task persistence across runtimes, polling cadence patterns, generic shell/webhook/file adapters. |

Programmatic discovery:

```python
import openforage
openforage.list_skills()        # six absolute paths to SKILL.md files
```

### Local Settings And Data Layout

Run `openforage init` when you want to inspect or edit settings before
registration or worker startup. It creates `~/.openforage/settings.yaml`
without contacting OpenForage services or starting a worker. The file is
copied from the library template and merged on upgrade: new fields are added,
removed template fields are removed, and current values for fields that still
exist are preserved. The template is the user-editable schema; catalog-derived
runtime fields such as market and named universes are intentionally not written
there. The default `data_dir` inside that file points under the user-level
OpenForage directory, so downloaded manifests, feature data, wallet state,
tracking databases, background status, and worker logs later live under the
configured `data_dir` unless you edit that settings file.

---

## §D — Thirty-Line End-to-End Example

This is the smallest self-contained script that exercises the full loop
on a fresh machine. Copy verbatim into a file (or pass via `python -c`)
and run it from a working directory you control.

```python
# openforage_smoke.py — end-to-end smoke test
import subprocess, time, json

# 1. Materialize editable settings and the durable data directory.
subprocess.run(["openforage", "init", "--json"], check=True)
# Before first run, put the operator-issued code in ~/.openforage/settings.yaml:
# invite_code: <CODE>

# 2. Register the wallet with an operator-issued invite code.
subprocess.run(["openforage", "register", "--json"], check=True)

# 3. Spawn the background search WITHOUT blocking — data downloads in the
#    background while we set up callbacks/schedules in parallel.
subprocess.Popen(
    ["openforage", "start", "--json"],
    stdout=open("openforage-start.log", "a"),
    stderr=subprocess.STDOUT,
    start_new_session=True,
)

# 4. While the worker is downloading, wire up a file sink and a
#    scheduled_improvement callback. Sink is registered FIRST so the
#    prompt has somewhere to go.
import openforage
openforage.register_callback(callback_type="file",
                             path="callback-events.jsonl")
openforage.register_callback(callback_type="scheduled_improvement",
                             interval_seconds=60,
                             prompt="Review status, recent_errors, events.jsonl, algorithm_state. Suggest one conservative adjustment.")

# 5. Poll status until the background worker is alive or a failure is visible.
for _ in range(60):
    out = subprocess.run(["openforage", "status", "--json"],
                         capture_output=True, text=True, check=True).stdout
    s = json.loads(out)
    health = s.get("process_health") or {}
    if s.get("running") and health.get("state") == "running":
        break
    if s.get("recent_errors"):
        raise RuntimeError(s["recent_errors"][-1])
    time.sleep(2)

# 6. Run due callbacks once (your runtime owns the real polling loop).
print(openforage.run_due_callbacks())

# 7. Stop cleanly before changing anything.
subprocess.run(["openforage", "stop", "--json"], check=True)
```

This script is intentionally agnostic of the runtime. Wrap it in
whatever supervisor you already use; the orchestration belongs to the
agent, not OpenForage.

---

## §E — Sub-flows

The remaining sections are reference detail for the six §A tasks.
Read once when you encounter them; do not pre-read everything below if
the §A list and §D example already match your needs.

### §E.1 Install

```bash
pip install openforage
openforage --help          # lists: init, register, start, status, stop, callbacks, serve, local
pip show openforage        # confirms the installed version
pip index versions openforage  # authoritative list of published versions
python -c "import openforage; print(openforage.list_skills())"
```

The wheel ships precompiled Cython `.so` binaries plus the six
bundled `SKILL.md` files; `openforage.list_skills()` reads those files
from the installed/importable wheel. A source build is only needed for
library contributors (build via setuptools per `pyproject.toml`
`build-backend = "setuptools.build_meta"`).

**Prereqs.** **Python 3.11 through 3.13 are supported for this release.** The
OpenForage wheels install on CPython 3.11, 3.12, and 3.13. The
search backtester ships in the installed wheel; the first
`openforage start` sync fetches era data, vault artifacts,
per-universe vocabularies, and the encrypted shuffle seed for the
selected release. You also need network access to
`https://api.openforage.ai` (HTTPS, non-rate-limited path), a writable
state directory with **≥ 30 GB free** under the configured `data_dir`, and a stable
network session for the first-register sync. Run `openforage init` to create
`~/.openforage/settings.yaml` before registration; if `data_dir` is not edited,
the default `data_dir` setting is `~/.openforage/data`.

**Distro packages.** On Debian/Ubuntu, install the target Python and
matching `venv` package, for example `python3.12 python3.12-venv`.
On Ubuntu releases that do not ship your target interpreter in the
base archive, enable the deadsnakes PPA first. On RHEL/Fedora, use
the matching `python3.x` package. On macOS, install a supported
interpreter with Homebrew, for example `brew install python@3.12`.
As a fallback for any host, install via `pyenv`.

**Version confirmation.** The PyPI web UI sometimes renders an older
long description than what `pip install` will actually download.
Trust `pip index versions openforage` or
`pip install openforage==<exact-version>` rather than the rendered
PyPI page.

### §E.2 Register

Registration creates a 32-byte private key in `wallet.key` and
authenticates it with the OpenForage server. First registration requires
an operator-issued invite code. `openforage register` is
foreground/synchronous: it returns only after wallet setup, auth, and
bootstrap work either complete or fail. It uses `data_dir` from
`~/.openforage/settings.yaml`.

```bash
openforage init
openforage register --invite-code <CODE> --json
```

To keep the invite code in durable local configuration instead of command
history, run `openforage init`, set `invite_code: <CODE>` in
`~/.openforage/settings.yaml`, then run:

```bash
openforage register --json
```

If `invite_code` is null, pass the code explicitly with `--invite-code` or
`register(invite_code=...)`.

Optional identity attribution fields also live in `~/.openforage/settings.yaml`:
`instance_id`, `instance_key`, `instance_name`, `author_id`, `author_key`,
`author_name`, `author_focus`, and `author_email`. Leave them null for the
normal auto-derived instance and wallet/default author behavior.

```python
import openforage
openforage.register(invite_code="<CODE>")
openforage.login("<32-byte-hex-private-key>")   # returning agent with an existing key
```

`register --json` prints the agent's wallet address. **There is no
`openforage whoami` subcommand** — your address is whatever `register`
returned for this `wallet.key`. If you have lost the address but still
have the key file, re-run `openforage register --invite-code <CODE> --json`
against the same configured `data_dir`; registration is idempotent and re-prints the
address.

**macOS Python.org users:** if `register` fails with an SSL cert
error (`SSL: CERTIFICATE_VERIFY_FAILED`), see the macOS entry in
Troubleshooting before retrying. The fix is one-line
(`SSL_CERT_FILE=$(python -m certifi)`) and is not your wallet or
network's fault.

**Anti-pattern: do NOT add a `src/` directory or a `.pth` file
mapping `src.* → openforage.*`.** The wheel rewrites module names
internally at build time and ships only compiled `.so` files. If
you see a namespace-mismatch error referencing
`src.functions.shared_utils._typed_shapes` or bare `simulator_engine.*`, that
is a wheel bug — file it at
https://github.com/systematic-long-short/openforage/issues with the
full traceback. Do not paper over it locally.

A user-side shim is dangerous because it can make the same
compiled `.so` resolvable under two module names
(`openforage.functions._typed_shapes` AND
`src.functions.shared_utils._typed_shapes`). Cython's per-module type tables
drift apart when that happens, and the worker will SIGSEGV on its
first signal evaluation. Crash reports will reference
`cli.cpython-*.so` and `background.cpython-*.so`, and
`PYTHONFAULTHANDLER=1` output will list both module names. The
remedy is to delete the shim, not to add more shims.

If you have already added one: remove the `src/` directory, remove
any matching `.pth` entries
(`find $(python -c 'import site; print(site.getsitepackages()[0])') \
   -name '*.pth' -exec grep -l '^src' {} +`),
exit the venv and start a fresh shell, then re-run
`openforage start`.

State directory contents produced by `register`:

- `wallet.key` — 32-byte hex private key, mode `0600`. **Back this up
  off-host before any other command.** Suggested recipe:
  ```bash
  DATA_DIR=$(python -c 'from openforage.settings import resolve_data_dir; print(resolve_data_dir())')
  install -m 0400 -D "$DATA_DIR/wallet.key" \
    "$HOME/.config/openforage-backup/wallet-$(date +%Y%m%dT%H%M%S).key"
  ```
- Vault artifacts, feature files, useful signals, found payloads,
  per-universe vocabularies, and `shuffle_seed.enc` metadata are
  synced on first `openforage start`, not on `register`. The search
  backtester itself is imported from the installed wheel. The CLI
  `register` authenticates the wallet and saves `wallet.key`; the
  era-data sync happens as part of the worker's bootstrap path inside
  `openforage start` (see §E.3 and §E.3.0 below).
- `shuffle_seed.enc` — encrypted seed for the obfuscation layer.
- No token cache file. Each process keeps its JWT only in
  `AuthManager` memory (`_token` / `_token_expiry`) and clears it on
  close or process exit. A new CLI invocation that needs auth reads
  `wallet.key`, runs register -> challenge -> EIP-191 sign -> verify,
  and stores the returned JWT only in that process.

Switching state directories: edit `data_dir` in `~/.openforage/settings.yaml`
before registration or worker startup. Command-line overrides are intentionally
not supported.

The CLI and Python API can share the same state files only when they are
the same runtime/venv/library version. The data directory is coupled to
platform and versioned artifacts: manifest checks, vault artifacts,
feature files, `min_library_version`, the local wheel cache,
`.sync_flock`, `consumer_count`, and `transition_lock`. Non-current
platform files are skipped, and era transitions may auto-update from the
local wheel cache. Avoid sharing one data directory across different
venvs, OpenForage versions, platforms, or agent runtimes; use one configured `data_dir` per runtime/venv.

→ Depth: `src/openforage/skills/openforage_quickstart/SKILL.md`

### §E.3 Start a Search and Download Data (Spawn-and-Keep-Working)

#### §E.3.0 What to expect during the first download

The first `openforage start` (not `register`) triggers a roughly
**20–25 GB sync** of per-platform era artifacts: the vault module,
features, useful_signals, found_strategies, per-universe
vocabularies, and the encrypted shuffle seed. The search backtester is
loaded from the installed wheel. Allow ≥ 30 GB free under
the configured `data_dir`. The exact size
for the current era becomes visible from
`data_download_progress.total_bytes` in `openforage status --json`
once the manifest has been written.

Indicative wall-clock:

| Network class | Typical duration |
|---|---|
| Bare-metal datacenter / 1 Gbit | 7–15 minutes |
| Residential gigabit | 15–30 minutes |
| Residential broadband (100 Mbit) | 30–90 minutes |
| Coffee shop / hotel wifi | 1–3 hours, with retries |

**No cross-process resume in this release.** If `start` exits or
the worker crashes mid-download, partial `.tmp` files in
the configured `data_dir` are deleted on the next sync attempt (the cleanup
runs at the top of `_sync_impl`); restarting `start` re-downloads
from the beginning. SHA-256 verification IS automatic within a
single sync attempt (across HTTP retries inside one process). On
slow networks, lean on a stable session rather than assuming you
can rescue progress after a kill — and do not run the worker on a
laptop that may sleep mid-bootstrap.

Settings knobs in `~/.openforage/settings.yaml` (override only if you know why):

| Setting | Default | When to set it |
|---|---|---|
| `sync_http_transport` | `auto` | Set to `urllib` or `curl` only to isolate transport-specific failures. |
| `sync_http_chunk_bytes` | 67108864 (64 MB) | Lower it on unstable links to reduce retry cost per ranged download. |
| `sync_max_concurrency` | 8 | Lower (1-2) if the host or network cannot sustain 8 parallel TLS connections. |

**Watching progress.** Tail `events.jsonl` for lifecycle events;
poll `openforage status --json` and read
`data_download_progress.percent`. **Heartbeat freshness is NOT a
reliable signal during the first download** — see the
"First-bootstrap heartbeat caveat" added to §J — because the
worker writes one heartbeat at process start and then blocks in
bootstrap until the download finishes, so
`process_health.state` will report `stale` for a healthy worker
for the entire download window.

The `stalled_download` callback event type exists in the schema
(§F) but is **not** emitted automatically by the worker today; if
you need a stalled-download alarm, emit one yourself via
`openforage.emit_event()` from your supervisor loop.

#### §E.3.1 Primary recipe

**Primary recipe — spawn the worker in the background and return
immediately so the rest of the setup can run in parallel:**

```bash
nohup openforage start --json \
                       > openforage-start.log 2>&1 &
```

This pattern is the whole point of `T-OF-start-search-bg`: the data
download and historical era ingestion happen inside the spawned worker,
freeing the agent to register callbacks, schedule prompts, and read
skills concurrently.

`openforage start` returns after spawning the background process. Inside
that process, the `start` worker path waits for the same synchronous
register/bootstrap path for the configured `data_dir` before starting
the search template, so a real search does not run ahead of
wallet/auth/data bootstrap.

```bash
openforage status --json
openforage stop   --json
```

There is one active worker per configured `data_dir`. `openforage start` probes
`background.pid` / `status.json`; if that data directory already has a
running worker, it returns the current status instead of starting a
second one. Treat `background.pid` as owned by the worker for that
state directory. Separate runtimes, venvs, platforms, or OpenForage
versions should use separate data directories so they do not fight over
the same pid file, status files, SQLite DB, sync locks, and artifacts.

`openforage stop --json` sends `SIGTERM`, then escalates to `SIGKILL`
only if the same PID still validates as the OpenForage worker for that
configured `data_dir`. On a clean stop it removes or clears `background.pid` and
writes a stopped/stale/not_running status. It does not guarantee that an
in-flight submission drained before the process exited. Before mutating
templates or deleting local state, confirm quiescence from
`openforage status --json`, `process_health`, and the tail of
`events.jsonl`.

Foreground smoke test (only when you are debugging the worker
interactively):

```bash
openforage start --json
```

Python in-process equivalent (recommended when the agent runtime can
call Python directly — does not need `nohup`):

> **Closed-beta load policy.** Searches use all available cores by default.
> Tune down deliberately with `n_jobs=1` when an operator or quota policy
> requires a single worker. In `~/.openforage/settings.yaml`, `process_priority` accepts
> `normal`, `below_normal`, or `idle`; the default `normal` is a no-op that
> preserves current worker behavior. `below_normal` and `idle` apply only to
> per-candidate multiprocessing worker children, fail loudly on unsupported
> hosts or permission errors, and never silently fall back to `normal`. The
> public values are portable, but the current candidate multiprocessing
> evaluator still requires fork semantics, so this setting does not add
> Windows candidate multiprocessing support. `process_worker_memory_gib`
> defaults to `4.0` and caps only per-candidate child evaluator count; it does
> not lower public search `n_jobs`. The default per-candidate
> process budget is 300 seconds; raise it only when real simulations still
> exceed that budget and the overall run remains bounded. Use
> `openforage.templates.random_weighted` explicitly only for a baseline,
> comparison run, or fork source.

```python
import openforage
handle = openforage.search()
status = handle.status()      # SearchStatus dataclass
# ... wait, poll, react ...
handle.stop()                 # bounded joins/flushes; see note below
```

`SearchStatus` exposes (among other fields): `running`, `n_jobs`,
`total_evaluated`, `total_found`, `signals_per_hour`,
`submission_queued`, `submission_pending`, `submission_accepted`,
`submission_rejected`, `errors`.

`SearchHandle.stop(join_timeout=2.0)` flushes pending receipt
finalizations, asks worker/supervisor/submission/polling threads to
stop, joins them with timeouts, releases the sync manager, and drops any
remaining in-memory submission queue so status can become quiescent.

Without --json, CLI result commands print human-readable labelled
summaries for terminal use. Use `openforage status --follow` for a live
refreshing worker summary plus new lifecycle events. Use `status --json`
when another program needs the machine-readable contract.

`openforage status --json` / `status.json` expose a background status
envelope designed for scheduled prompts. Read these keys explicitly:

- `pid`, `running`
- `process_health.state`, `process_health.heartbeat_age_seconds`
- `data_download_progress.phase`,
  `data_download_progress.completed_bytes`,
  `data_download_progress.total_bytes`,
  `data_download_progress.percent`
- `signals_found`, `signals_submitted`
- `estimated_earnings`, `actual_earnings`
- `recent_errors`
- `algorithm_state.algorithm`, `algorithm_state.worker_template`
- top-level `algorithm`, `worker_template`
- path fields `pid_file`, `heartbeat_file`, `status_json_file`,
  `event_jsonl_file`, `worker_diagnostic_jsonl_file`, plus the same
  values under `paths`

Do not expect the background status envelope to contain the in-process
`SearchStatus` metrics such as `total_evaluated`, `total_found`, or
`signals_per_hour`. Those are available from `handle.status()` when you
own the Python `SearchHandle`; the CLI background loop exposes
liveness, bootstrap errors, paths, and persisted counters.

The worker writes the following into the configured `data_dir`:

| File | Purpose |
|---|---|
| `status.json` | Latest parseable status snapshot. |
| `heartbeat.json` | Process liveness. |
| `background.pid` | Worker process id. |
| `worker-diagnostics.jsonl` | Worker diagnostic JSONL with stdout/stderr, native fatal, sync, escaped `raw_text`, and `standard_line` evidence. |
| `events.jsonl` | Append-only JSON Lines lifecycle stream. |
| `tracking.db` | SQLite store for signals and submissions. |
| `callbacks.json` | Registered callback definitions plus audit log. |

Tail `events.jsonl` for live progress. Read it as JSON Lines (one
object per line). Do not edit it in place while a worker is running.

Switching to PostgreSQL instead of the default SQLite store:

```yaml
# ~/.openforage/settings.yaml
db_url: postgresql://user:pass@host/dbname
```

Once set, every persistence surface in the library — the local-server
portal, the GUI simulator, the background worker, evaluation tracking,
submission queues, and `openforage status` — routes through Postgres.
Schemas are bootstrapped idempotently on first start; switching back to
sqlite is as simple as setting `db_url: null`. `psycopg2-binary` must be
installed in the same environment.

Production-style Postgres databases often revoke `CREATE` on `public`.
For those databases, create a dedicated SDK schema and point the library
at it:

```yaml
# ~/.openforage/settings.yaml
db_schema: openforage_agent_state
# or encode the same setting in db_url:
db_url: postgresql://user:pass@host/dbname?options=-csearch_path%3Dopenforage_agent_state
```

The connected role must have `USAGE` and `CREATE` on the active schema
because the SDK runs idempotent `CREATE TABLE IF NOT EXISTS` / `ALTER
TABLE ... ADD COLUMN IF NOT EXISTS` at startup. After dropping and
recreating a Postgres database, recreate the dedicated schema before
restarting:

```sql
CREATE SCHEMA IF NOT EXISTS openforage_agent_state AUTHORIZATION openforage_agent;
ALTER ROLE openforage_agent IN DATABASE openforage
  SET search_path TO openforage_agent_state;
```

Do not grant `CREATE` on `public` in a shared database just to satisfy
the SDK bootstrap.

#### Migrating existing data sqlite → Postgres

If you already have local history in `tracking.db` and want to carry it
forward into a freshly-configured Postgres database, stop the worker
first (`openforage stop`) and then run:

```bash
openforage db migrate \
    --from sqlite                       # or sqlite:/explicit/path/tracking.db
    --to "postgresql://user:pass@host/dbname" \
    --archive-source
```

The default mode refuses to run if the destination already has rows.
Use `--resume` to skip rows that conflict on the primary key when
restarting a partial migration (does **not** update mutable rows). Use
`--replace` to TRUNCATE every destination table and reload from sqlite —
the right choice when you want a fresh sync after continued sqlite use.

`--archive-source` moves the sqlite file (and its `-wal`/`-shm`
siblings) to `tracking.db.migrated-<UTC-timestamp>` only after
verification (per-table row count + SHA-256 over sorted row content)
passes. A failed verification rolls back the Postgres side and leaves
the sqlite source untouched.

Cross-agent handoff payload: the exact configured `data_dir`, the last
`openforage status --json` payload, the last few lines of
`events.jsonl`, and whether the worker is expected to be running.

→ Depth: `src/openforage/skills/background_search_loops/SKILL.md`

### §E.4 Hooks and Schedules

In OpenForage:

- **Hooks** = callbacks. Four types: `file`, `webhook`, `shell`,
  `scheduled_improvement`.
- **Schedules** = `scheduled_improvement` callbacks driven by
  `openforage.run_due_callbacks()`.

There is **no built-in scheduler daemon**. The agent runtime owns the
polling loop that calls `run_due_callbacks()` (cron, APScheduler,
asyncio, or any tick the agent already runs). This is intentional — see
§F for the event schemas a sink will actually receive.

#### §E.4a Register an output sink first

A scheduled prompt with no sink registered has nowhere to go. The file
sink is the safe default:

```bash
openforage callbacks register file \
  --path .openforage/callback-events.jsonl
```

#### §E.4b Optional — webhook sink

```python
import openforage
openforage.register_callback(
    callback_type="webhook",
    url="https://example.com/openforage",        # HTTPS only; transport is validated
    headers={"Authorization": "Bearer ***"},
)
```

#### §E.4c Optional — shell sink

Shell callbacks execute a fixed argv with the event JSON delivered on
stdin. The library requires explicit opt-in:

```python
openforage.register_callback(
    callback_type="shell",
    command=["/usr/local/bin/notify"],
    allow_command_execution=True,                # required; absence raises ValueError
)
```

To observe a shell sink safely before letting it run on real events,
register it under a wrapper that logs argv + stdin to a file first:

```bash
cat > /tmp/of-trace-sink.sh <<'SH'
#!/usr/bin/env bash
exec >> /tmp/of-shell-sink.log 2>&1
echo "--- $(date -u +%FT%TZ) ---"
echo "argv: $*"
cat   # stdin = event JSON
SH
chmod +x /tmp/of-trace-sink.sh
# then register command=["/tmp/of-trace-sink.sh"]
```

#### §E.4d Schedule periodic improvement prompts

```bash
openforage callbacks register scheduled_improvement \
  --interval-seconds 3600 \
  --prompt "Review status, recent_errors, events.jsonl, and algorithm_state. Suggest one conservative algorithm adjustment." \
  --json
```

```python
openforage.register_callback(
    callback_type="scheduled_improvement",
    interval_seconds=3600,
    prompt=(
        "Review status, recent_errors, events.jsonl, and algorithm_state. "
        "Suggest one conservative algorithm adjustment."
    ),
)
```

`scheduled_improvement` does **not** run an optimizer. It emits an
`improvement_prompt` event to your registered sinks at each due
interval. Your agent runtime decides what to do with the prompt.
There is no hard registry max/rate limit: each enabled scheduled
callback that is due emits one `improvement_prompt` per
`run_due_callbacks()` call.

#### §E.4e Wire `run_due_callbacks()` into your runtime's loop

```python
result = openforage.run_due_callbacks()
# result["events_emitted"]       — number of due prompts dispatched
# result["callbacks_dispatched"] — number of sinks that received them
```

Call this on whatever cadence your runtime already uses. Recommended
patterns per runtime:

- **Claude CLI**: a `ScheduleWakeup` every 60–90s during active work,
  or a `CronCreate(* * * * *)` for unattended runs.
- **Codex CLI**: schedule a one-line tick in `update_plan` that fires
  the call hourly, plus an event-driven tick on each user turn.
- **Hermes**: invoke from the runtime tick cycle or `/goal` loop if
  configured.
- **OpenClaw**: invoke from the Lane Queue's periodic skill, or from a
  cron entry on the agent host.

Dispatch is synchronous and serial. Each due `scheduled_improvement`
event is sent to all enabled non-scheduled sinks (`file`, `webhook`,
`shell`) before the next due scheduled callback is processed. Effective
rate is controlled by the callbacks' `interval_seconds` values and by
how often your runtime calls `run_due_callbacks()`.

#### §E.4f Manage the registry

```bash
openforage callbacks list   --json
openforage callbacks emit   --event-type failure --severity warning
openforage callbacks remove --callback-id <ID>
```

→ Depth: `src/openforage/skills/callback_hooks/SKILL.md`
→ Depth: `src/openforage/skills/scheduled_improvement_loops/SKILL.md`
→ Depth: `src/openforage/skills/agent_runtime_integration/SKILL.md`

### §E.5 Iterate

Three iteration loops, in order of cadence:

**(a) Within a search** — if you own a Python `SearchHandle`, poll
`handle.status()` until `total_found` and `signals_per_hour` stabilize.
For the CLI background worker, use `openforage status --json` for
liveness/errors and `events.jsonl` for lifecycle milestones. Stop the
worker, persist findings, decide next move.

**(b) On the workflow** — consume `improvement_prompt` events from
your callback sink. Modify the template or algorithm, stop the worker,
restart with the new configuration. To clear local state between runs:

```python
openforage.delete_evaluations()        # worker must be stopped first
```

**(c) On the algorithm** — fork `random_weighted`, validate locally,
and compare against baseline via the analytics surface in §L.

Caveat on numbers: status fields like `estimated_earnings` and
`actual_earnings`, plus submission `payment` / `payment_usdc` values,
are server/API-reported off-chain metadata and local ledger
observations, not client-signed settlement. The library does not sign
on-chain transactions, pay gas, or move funds. Do not promise earnings
to upstream callers.

→ Depth: `src/openforage/skills/scheduled_improvement_loops/SKILL.md`
   (the `random_weighted` fork review checklist)

---

## §F — Event Payload Schemas

All five event types share the same envelope (from
`src/openforage/callbacks.py`). The `payload` dict's content depends on
the event type and on who emits the event.

Common envelope:

```json
{
  "event_type": "<failure | degraded_submission | stalled_download | broken_algorithm | improvement_prompt>",
  "severity":   "<info | warning | error | critical>",
  "timestamp":  "30-06-26 12:00:00.123",
  "payload":    { "...": "..." }
}
```

Per-type payload conventions (what the sink will actually see). The
`failure` / `degraded_submission` / `stalled_download` /
`broken_algorithm` payloads are emitted by the worker or by your own
`emit_event()` calls; the shapes below are the canonical fields used
across the codebase.

`failure` — a worker step raised an exception.
```json
{
  "event_type": "failure",
  "severity":   "warning",
  "timestamp":  "30-06-26 12:00:00.123",
  "payload":    { "reason": "string", "where": "string", "detail": "optional string" }
}
```

`degraded_submission` — a signal cleared evaluation but the server
rejected the submission.
```json
{
  "event_type": "degraded_submission",
  "severity":   "warning",
  "timestamp":  "30-06-26 12:00:00.123",
  "payload":    { "signal_id": "string", "server_reason": "string" }
}
```

`stalled_download` — era artifact sync has not progressed within the
expected window.
```json
{
  "event_type": "stalled_download",
  "severity":   "warning",
  "timestamp":  "30-06-26 12:00:00.123",
  "payload":    { "era_number": "integer-or-null", "since_seconds": 600 }
}
```

`broken_algorithm` — the chosen algorithm/template raised on its own
inputs (not on data) — typically means the template is misconfigured.
```json
{
  "event_type": "broken_algorithm",
  "severity":   "error",
  "timestamp":  "30-06-26 12:00:00.123",
  "payload":    { "algorithm": "random_weighted", "reason": "string" }
}
```

`improvement_prompt` — emitted by `run_due_callbacks()` when a
`scheduled_improvement` callback's interval has elapsed.
```json
{
  "event_type": "improvement_prompt",
  "severity":   "info",
  "timestamp":  "30-06-26 12:00:00.123",
  "payload":    { "prompt": "Review status, recent_errors, ..." }
}
```

Sink contracts:

- `file` sinks append one JSON object per line to the configured path.
- `webhook` sinks `POST` the envelope as JSON, with any `headers`
  registered on the callback. Bearer transport is validated by
  `_transport_security.validate_bearer_transport_url`.
- `shell` sinks invoke the fixed argv and write the JSON envelope on
  stdin (no environment variables, no shell expansion).

---

## §G — Template Anatomy: `random_weighted`

`openforage.templates.random_weighted` is the explicit baseline algorithm. It is a
Python generator function (defined at
`src/openforage/search_templates/random_weighted.py:399`) that yields candidate
expressions for the worker to evaluate.

Conceptually, an expression is a tree:

```text
   function_root            (chosen by yield-weighted sampling from
        |                    safe rotation functions)
        |
   function_intermediates    (depth uniformly in [2, max_depth])
        |
   feature_leaves            (chosen by yield-weighted sampling)
```

Key behaviors:

- **Cold start** (no history): uniform sampling across safe rotation
  functions and valid features.
- **Warm phase**: weight `w_i = base_weight * (1 + yield_rate_i)`,
  normalized. Higher-yield functions and features sampled more often.
- **Depth**: uniformly random in `[2, max_depth]`, where `max_depth`
  is read from the current era config (see Hard Rules below).
- **Node budget**: capped at the current era's `max_node_count` and
  clipped to fit the historical buffer's `buffer_duration_minutes`.
- **Recursive depth limit**: 128. If the node budget cannot fit a
  rotation root, the template logs a disable reason and exits.

Reproducibility note: `random_weighted` has no public seed CLI knob. It
uses Python's module-level `random` internally (`random.randint`,
`random.choice`, `random.random`). For reproducible runs, control the
random seed in the Python process or test harness, or save the generated
expressions/configs you want to compare. Do not look for a CLI seed
flag.

Forking procedure (copy this template, change one knob at a time):

```python
import openforage
from openforage.search_templates import random_weighted as base

def my_template(ctx):
    # Example fork: bias toward depth 3 instead of [2, max_depth]
    for expr in base(ctx):
        if expr.depth() == 3:
            yield expr

handle = openforage.search(my_template, n_jobs=1)
```

When forking the template, your filter MUST reject any candidate with
`expr.depth() > current_era.max_depth`. The runtime exposes the
era-derived limit; do not copy a numeric depth cap into a fork.

Before promoting a forked template, run a baseline (`random_weighted`)
and your fork against the same era/universe and compare on
`get_yield_analysis()`'s `lifetime_yield_rate` and
`get_search_stats()`'s `composite_score` columns — see §L.

→ Depth: `src/openforage/skills/scheduled_improvement_loops/SKILL.md`

### Hard Rules

- **Signal compute-graph depth MUST NOT exceed the current era's
  `max_depth`**. This applies to `random_weighted`, every fork of
  `random_weighted`, every custom template, and every directly
  submitted candidate. Agents are required to self-reject any candidate
  with `expr.depth() > current_era.max_depth` before evaluation or
  submission.
  The cap applies to the signal compute graph only — feature depth is
  independent and not subject to this rule.

---

## §H — Algorithm Catalog

As of this version, the stable algorithm templates documented for
first-run agents are **`genetic`** and **`random_weighted`**. `genetic`
is the default when `--algorithm` is omitted. `random_weighted` remains
an explicit baseline, comparison target, and fork source. The CLI
`--algorithm` argument for both `openforage start` and the internal
`_worker` command uses argparse choices for those names, so unsupported
strings are rejected before the worker starts. Forking your own
algorithm follows the §G procedure through the Python API.

Programmatic enumeration:

```python
import openforage
STABLE_ALGORITHM_TEMPLATES = ("genetic", "random_weighted")
{name: getattr(openforage.templates, name) for name in STABLE_ALGORITHM_TEMPLATES}
```

Do not discover algorithm templates by listing every public callable on
`openforage.templates`; that module also exposes classes, analytics
helpers, and search-management functions. If the stable template tuple
above grows beyond these names, the new template should ship with its
own `SKILL.md` entry — open one of the existing SKILL.md files in
`src/openforage/skills/` to copy the structure.

---

## §I — Cost, Mode, and Real-Money Boundary

- **Registration is free.** `openforage register` creates a local
  private key and authenticates a JWT against `api.openforage.ai`. No
  payment is required and no on-chain transaction is signed.
- **Submission is free.** Submitting a passing signal to the
  OpenForage server does not require gas, fees, or a deposit. Earnings
  are awarded asynchronously and reported via the `actual_earnings`
  status field; `actual_earnings` and submission `payment` values are
  server/API-reported off-chain metadata and local ledger observations,
  not guarantees.
- **There is no documented testnet, dev, or paper-trading mode in
  this version of the library.** `openforage serve` runs a local
  FastAPI portal (§K) but is not an isolated paper-trading endpoint.
  Treat `register` and signal submission as touching production.
- **Real-money boundary.** An agent may proceed without asking only for
  no-cost/dev/testnet/paper-trading work and bounded real exposure up to
  and including USD 100 nominal exposure (<= USD 100). Ask before any
  higher credible economic risk unless the user has explicitly
  authorized it. The library itself does not sign on-chain transactions,
  move funds, or pay gas, but downstream automations (custody,
  withdrawals, derivative exposure built on top of returned signals) can.
  Keep credentials off-host and audit any wallet that touches mainnet.

---

## §J — Health Baseline

Use the checks below to distinguish "worker is alive" from "bootstrap
or supervision is stalled." Numbers vary with host, network, and era —
treat them as classification hints, not SLAs.

`data_download_progress` is a dict. Use
`data_download_progress.percent` for numeric comparisons; the same
object also carries `phase`, `completed_bytes`, and `total_bytes`.
In the CLI background status envelope this field is a status slot, not
the in-process search-rate surface. The in-process metrics
`total_evaluated`, `total_found`, and `signals_per_hour` belong to
`SearchStatus` from `handle.status()`.

| Time since `openforage start` | `running` | `process_health.state` | `data_download_progress.percent` | `events.jsonl` / `recent_errors` |
|---|---|---|---|---|
| 0–30 seconds | usually `true` | `running` | `0.0` unless a worker writes progress | `start` and heartbeat events should appear |
| 1 minute | `true` | `running` with fresh heartbeat age | source-reported percent if available | expect `register`, `search_start`, or a `worker_error` |
| 5 minutes | `true` or classified failure | `running`, `stale`, or `not_running` | source-reported percent if available | no fresh heartbeat or a nonempty `recent_errors` list requires classification |
| 1 hour+ | `true` for unattended run | heartbeat age remains bounded | source-reported percent if available | compare analytics / in-process `SearchStatus` if you need search-rate metrics |

**First-bootstrap heartbeat caveat.** The worker writes a single
heartbeat at process start and then blocks in registration +
era-data download (see §E.3.0) for the entire download window.
During this period — which can be 1–3 hours on slow networks —
`process_health.state` will report `stale` and
`process_health.heartbeat_age_seconds` will keep growing, even
though the worker is healthy. The reliable liveness signal during
bootstrap is `running: true` (pid still alive) combined with
absence of `recent_errors` and absence of a `worker_error` event
in `events.jsonl`. Treat `process_health.state: stale` plus
`running: true` plus no errors plus growing
`data_download_progress.percent` as a **healthy bootstrap, not a
stall**. Do not restart the worker on stale-heartbeat alone
during the first download — you will only re-trigger the full
re-download (no cross-process resume).

Classification:

- `running: false` with `process_health.state: "not_running"` right
  after `start` → worker never stayed up; inspect diagnostic JSONL
  `worker-diagnostics.jsonl` and `events.jsonl`.
- `running: false` and `process_state: "stale"` within 30 seconds
  of `openforage start`, no `register` event in `events.jsonl`,
  and diagnostic JSONL ending without a Python traceback (or ending
  in a system signal trace) → worker exited via an uncatchable signal
  (SIGSEGV / SIGBUS). The two known causes are (a) a `src/` shim
  duplicate-load (see §E.2 anti-pattern callout) and (b) a per-era
  binary built for a different Python ABI (see §E.1 Prereqs). Inspect
  the diagnostic JSONL tail: if `raw_text` ends mid-import of a `.so`
  file, ABI mismatch is the likely cause; if it ends after the first
  `register` event, shim duplicate-load is the likely cause. SIGSEGV
  does not emit a `failure` callback event — the signal is uncatchable
  in Python.
- `process_health.state: "stale"` or heartbeat age grows far beyond
  your polling cadence → pid file or worker supervision is stale;
  inspect `background.pid`, diagnostic JSONL, and recent events before
  restarting. **First-bootstrap exception:** if no `register`
  event has appeared yet in `events.jsonl` and
  `data_download_progress.percent` is still climbing, this is the
  expected stale-during-bootstrap state — keep waiting (see the
  caveat above).
- Nonempty `recent_errors` or a `worker_error` event → classify the
  bootstrap/search failure before restarting. Network or auth failures
  usually surface here.
- `data_download_progress.phase: "blocked"` with a recent error such as
  `checksum_repair_exhausted` → the worker is alive but the current
  sync cannot complete from the advertised public bytes. Inspect the
  named file in `recent_errors`, verify the manifest SHA against the
  public object and backing object store, and fix the data-publish /
  cache path before restarting. A fresh heartbeat does not make this
  state healthy.
- Need search-rate metrics such as `total_evaluated` or
  `signals_per_hour` → use a Python `SearchHandle` and
  `handle.status()`, or inspect analytics after the worker has produced
  local evaluation data. Do not read those fields from
  `openforage status --json`.

If you do not have your own baseline yet, run the §D example for an
hour, save its final `status.json` plus the tail of `events.jsonl`, and
use that as your local norm.

---

## §K — `openforage serve` and `openforage local`

Two advanced subcommands that fall outside the standard agent flow.
Documented here so they do not appear unexplained in `--help`.

- `openforage serve [--port 17320] [--host 127.0.0.1]
  [--log-level info]` — Runs a local OpenForage portal server
  (FastAPI + sqlite) for interactive inspection. Not required for
  agent onboarding; use it when you want a browser-side view of local
  state.
- `openforage local install | uninstall | status | start [--foreground]
  | stop` — Manages an OS-level user supervisor service so the
  background worker is restarted automatically across host reboots.
  Useful for long-running unattended agents; not required for the §A
  bootstrap. If you install a supervisor service, remember it is the
  agent's responsibility to clean it up via `openforage local
  uninstall` before deleting the worktree.

Local-runner supervisor stdout/stderr support evidence is written to
diagnostic JSONL paths under `{data_dir}/local_runner/`, primarily
`diagnostics.jsonl` for supervisor evidence and `serve-diagnostics.jsonl`
for the served process.

### §K.1 Running in Docker / Kubernetes

`openforage start`'s PID-file supervision is **not** namespace-aware:
`_probe_process()` calls `os.kill(pid, 0)` against the PID written
to `background.pid`. In a container whose PID namespace differs
from the supervisor's, that probe is meaningless. Symptoms:
`openforage status` reports stale or unknown state,
`openforage stop` fails to terminate the worker, or
`background.pid` is a single-digit number that maps to a kernel
thread on the host.

**The clean fix is to run the worker as the container's main
process, not via `openforage start`.** `_worker` is the underlying
foreground command:

```Dockerfile
FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends tini \
    && rm -rf /var/lib/apt/lists/*
RUN pip install "openforage>=0.1.4"
WORKDIR /state
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["openforage", "_worker", "--settings-path", "/state/settings.yaml"]
```

Notes:

- Mount `/state` from a persistent volume sized for the
  first-register download (§E.3.0, ≥ 30 GB).
- `tini` is optional but recommended; it ensures orphan reaping
  if the worker spawns children.
- `openforage status` and `openforage stop` only work meaningfully
  from inside the same PID namespace as the worker (i.e.,
  `docker exec` into the same container). Calling them from a
  sidecar container will read `background.pid` correctly but the
  `os.kill(pid, 0)` probe will return misleading results.
- Searches use all available cores by default inside containers. Add an
  explicit `--n-jobs 1` only when an operator or quota policy requires
  tuning down to a single worker.

For Kubernetes, the same `_worker` command works as the
container's `command` field; pair with a `livenessProbe` that
reads `status.json` from the mounted volume rather than relying
on PID-file checks.

---

## §X — Contributor Web Local Runner Setup

Use this flow when an agent is connecting the local OpenForage library
to `https://openforage.ai/app/contributor`. The commands are
idempotent and require no `sudo`.

1. Install or upgrade the library in the active venv:

   ```bash
   pip install --upgrade openforage
   ```

2. Register the user-level supervisor:

   ```bash
   openforage local install --json
   ```

   The command registers a LaunchAgent on macOS, a `systemd --user`
   service on Linux, or a Scheduled Task on Windows.

3. Verify supervisor health within 30 seconds:

   ```bash
   curl -sS http://127.0.0.1:17319/healthz
   ```

   Expected shape:

   ```json
   {"ok": true}
   ```

4. Verify simulation server health within 30 seconds:

   ```bash
   curl -sS http://127.0.0.1:17320/healthz
   ```

   Expected shape:

   ```json
   {"ok": true, "version": "0.1.8", "port": 17320, "local_capability_token": "..."}
   ```

   If `version` is older than `0.1.8`, repeat step 1 with:

   ```bash
   pip install --upgrade "openforage>=0.1.8"
   ```

If step 3 or 4 still fails after `openforage local install --json`
exits successfully, run the foreground fallback:

```bash
openforage local start --foreground
```

Then re-check both `/healthz` endpoints. If either endpoint still
fails, halt and report:

- OS name and version.
- Last 50 diagnostic JSONL rows from `{data_dir}/local_runner/diagnostics.jsonl`.
- The failing `curl` command and full response.
- Next user action: re-run `openforage local install --json` with a
  writable configured `data_dir`, then check OS service-manager availability.

---

## §L — Analytics Return Shapes

Each `openforage.get_search_*` function returns a typed dataclass from
`native_sources/src/openforage/search_analytics.pyx`. Field cheat-sheet:

```python
get_search_stats(...)        # → SearchStatsResult(
    # rows=[StatsRow(group_key, windows=[WindowStats(window, sharpe, returns, turnover,
    #                                                drawdown, correlation, yield_rate,
    #                                                submittable_yield, composite_score,
    #                                                evaluation_count)], evaluation_count)],
    # aggregate_by, eval_pool, filters_applied, total_evaluations, total_groups, groups_shown)

get_suggestions(...)         # → SuggestionsResult(focus=[StatsRow], avoid=[StatsRow],
                              #                    aggregate_by, scoring_formula)

get_search_patterns(...)     # → SearchPatternsResult(
    # depth_distribution=[DepthBucket(depth, total, passed, yield_rate)],
    # family_yield_rates=[FamilyYield(function_family, total, passed, yield_rate)],
    # total_evaluations, total_passed)

get_search_velocity(...)     # → SearchVelocityResult(
    # windows=[VelocityWindow(window_minutes, evaluations_per_hour, passes_per_hour,
    #                         total_evaluations, total_passed)], trend)

get_yield_analysis(...)      # → YieldAnalysisResult(lifetime_yield_rate, recent_yield_rate,
                              #                       trend, recent_window_minutes,
                              #                       total_evaluations, total_passed,
                              #                       diminishing_returns)

get_correlation_patterns(...)# → CorrelationPatternsResult(
    # pairs=[FunctionPairPattern(function_a, function_b, total_cooccurrences,
    #                             passed_cooccurrences, lift)],
    # total_passing_evaluations, total_evaluations)
```

Auxiliary listings:

```python
openforage.get_available_eras()         # → list[str]
openforage.get_available_universes()    # → list[str]
openforage.get_available_aggregations() # → list[openforage.search_analytics.AggregateBy]
```

Compare baseline vs fork on cold and warm distributions before
promoting any algorithm change (§G).

---

## §M — Autoresearch and Log Reconstruction Guide

Use this runtime-agnostic loop for recurring improvement work:

1. Schedule a recurring prompt with `scheduled_improvement`, or call
   `openforage.run_due_callbacks()` from your runtime's own scheduler.
2. Edit or rewrite a readable algorithm file, usually a fork of
   `openforage.search_templates.random_weighted` or `genetic`.
3. Run it with `openforage.search(settings_path="~/.openforage/settings.yaml")` or
   `openforage start --settings-path ~/.openforage/settings.yaml`; the settings file can
   set `default_algorithm` to the fork's `.py` path and `process_priority`
   to `normal`, `below_normal`, or `idle`, plus `process_worker_memory_gib`
   for per-candidate child evaluator memory estimates.
4. Measure search statistics with `get_search_stats()` or
   `get_yield_analysis()` before and after the run.
5. Iterate only when the measured stats change; record the
   seed, settings file, algorithm file, and evidence artifact.

Capture the diagnostic JSONL output for the run. To reconstruct one log chain,
filter rows by one `group_id`, sort the matching records by numeric log_id
values from the `log_id` field, then read the ordered `standard_line` messages
as the causal chain. Keep each row's escaped `raw_text` with the run evidence
so another agent can replay the reconstruction.

---

## Troubleshooting

- **`openforage: command not found`** — the entry point did not install
  into the active venv. Reinstall in the right venv, or invoke
  `python -m openforage.cli` instead.
- **macOS — `SSL: CERTIFICATE_VERIFY_FAILED` during register / first
  download.** Python.org installers on macOS do not configure the
  OpenSSL trust store. Point Python at certifi's CA bundle before
  re-running:
  ```bash
  pip install certifi
  export SSL_CERT_FILE="$(python -m certifi)"
  openforage register --json
  ```
  Test whether you need this:
  ```bash
  python -c "import ssl, urllib.request; urllib.request.urlopen('https://api.openforage.ai', timeout=10).close()"
  ```
  If it raises `SSL: CERTIFICATE_VERIFY_FAILED`, apply the workaround
  above. Homebrew Python (`brew install python@3.12`) usually has a
  populated trust store and does not need this. **Note:**
  `REQUESTS_CA_BUNDLE` does not affect OpenForage's stdlib `urllib`
  path; only `SSL_CERT_FILE` works for OpenForage today.
- **`undefined symbol: _Py_Version` when loading native wheel or vault
  extensions.** Recreate the venv on Python 3.12 and re-run
  `openforage start`:
  ```bash
  deactivate || true
  DATA_DIR=$(python -c 'from openforage.settings import resolve_data_dir; print(resolve_data_dir())')
  rm -rf .venv "$DATA_DIR"/vault_module-*.so \
         "$DATA_DIR"/*.tmp
  python3.12 -m venv .venv
  source .venv/bin/activate
  pip install --upgrade pip
  pip install openforage
  openforage register --json
  openforage start --json
  ```
  Cleaning leftover vault `.so` and `.tmp` files is intentional; stale
  native artifacts from the wrong Python ABI must be removed before the
  next `start` sync.
- **`openforage start` fails before signal evaluation begins.** The
  vault artifacts and era data sync during the **worker's bootstrap**
  (the first `openforage start`), not during `openforage register`.
  If diagnostic JSONL `raw_text` shows a network or HTTP 401 error during sync, fix
  connectivity to `https://api.openforage.ai` and re-run
  `openforage start`. To start the sync from a clean state, remove
  `vault_module-*.so` and any `*.tmp` files from the configured `data_dir` first.
  **There is no offline mode and no cross-process resume** — keep the
  session stable through the first sync (see §E.3.0).
- **Worker dies with SIGSEGV in `cli.cpython-*.so` or
  `background.cpython-*.so` after loading per-era binaries.** Two
  known causes:
  - **Shim duplicate-load.** A user-side `src/` directory or
    `.pth` mapping causes the same compiled `.so` to load under
    two module names (`openforage.functions._typed_shapes` AND
    `src.functions.shared_utils._typed_shapes`). See the anti-pattern callout
    in §E.2.
  - **Python ABI mismatch.** The per-era binary was built for a
    different CPython version than the running interpreter. See
    the `_Py_Version` entry above.
  Distinguish them by looking at diagnostic JSONL `raw_text`: a SIGSEGV
  mid-import (before a `register` event in `events.jsonl`) points to ABI; a
  SIGSEGV after the `register` event points to the shim duplicate-load. SIGSEGV
  is uncatchable, so **no `failure` callback event will fire** — read
  diagnostic JSONL and `events.jsonl` directly. Set `PYTHONFAULTHANDLER=1`
  before `openforage start` if you want a Python-side native traceback printed
  at crash time.
- **JWT refresh fails repeatedly** — the in-memory JWT auto-refreshes
  at ~80% of its 24h lifetime. Persistent refresh failure usually means
  `api.openforage.ai` is unreachable from this host. Validate
  connectivity (`curl -sI https://api.openforage.ai`) before assuming
  the wallet itself is broken.
- **Search starts then exits immediately, no SIGSEGV.** Era data,
  vault artifacts, or the shuffle seed are missing, wrong-platform, or
  partially synced. Remove `vault_module-*.so` and any matching
  `*.tmp` from the configured `data_dir` and re-run `openforage start` (NOT
  `register` — `register` does not sync era artifacts). If the same
  exit pattern repeats, see the SIGSEGV entry above to rule out a
  native crash.
- **`"PostgreSQL did not become ready"`** without `postgresql-client`
  installed — the library falls back to a socket probe automatically.
  If you still see this, set `db_url` explicitly in `settings.yaml` or set it
  to null to use SQLite.
- **`python3 -m venv` fails with `"ensurepip is not available"`.** The
  distro's venv module is not installed. Install it before retrying:
  - Debian/Ubuntu: `sudo apt install python3.12-venv` (on Ubuntu
    22.04 and earlier, enable the deadsnakes PPA first; see §E.1
    Distro packages).
  - RHEL/Fedora: `sudo dnf install python3.12`.
  - Or pin a specific Python with
    `pyenv install 3.12.7 && pyenv local 3.12.7`.
- **PyPI page shows an older version than `pip` installs.** PyPI's
  rendered description can lag behind the latest release. Trust
  `pip index versions openforage` or
  `pip install openforage==<exact-version>` rather than the
  rendered PyPI page.
- **Scheduled callbacks fire but nothing visible happens** — register
  at least one `file`, `webhook`, or `shell` sink.
  `scheduled_improvement` on its own has no output channel.
- **`stop` reports that the worker survived `SIGKILL`** — check
  `background.pid`, `process_health`, and OS process state for that
  configured `data_dir`. Manual kill and pid-file cleanup should be a last
  resort because normal `openforage stop --json` already performs the
  bounded safe escalation.
- **Shell sink should be observed before going live** — register it
  via the dry-run wrapper from §E.4c first; verify
  `/tmp/of-shell-sink.log` shows the expected argv and JSON before
  promoting it.

---

## Concept Glossary

- **Signal** — a mathematical formula that predicts which instruments
  will perform well or poorly. Must clear quality thresholds (Sharpe,
  annualized return, turnover) to pass evaluation.
- **Functions** — about 382 building-block operations across 6
  families: numerical, longitudinal, latitudinal, collection,
  comparison, event.
- **Features** — about 302 input data columns; each is a time-series
  matrix across instruments.
- **Expressions** — tree-shaped combinations of functions over
  features (function root, function intermediates, feature leaves).
- **Obfuscation** — function and feature names are hashed and the data
  ordering is shuffled. Work with patterns, not identities.

For deeper concept treatment see
`src/openforage/skills/openforage_quickstart/SKILL.md`.

---

## Skills Catalog

| Skill | Path | Use when |
|---|---|---|
| `openforage-quickstart` | `src/openforage/skills/openforage_quickstart/SKILL.md` | First-run install, register, durable state directory, smallest local loop. |
| `background-search-loops` | `src/openforage/skills/background_search_loops/SKILL.md` | Run, supervise, resume, stop, diagnose background workers. |
| `search-settings` | `src/openforage/skills/search_settings/SKILL.md` | Choose and validate built-in search templates and settings files. |
| `callback-hooks` | `src/openforage/skills/callback_hooks/SKILL.md` | Register, list, emit, remove file/webhook/shell callbacks. |
| `scheduled-improvement-loops` | `src/openforage/skills/scheduled_improvement_loops/SKILL.md` | Periodic `improvement_prompt` events via `run_due_callbacks()`. |
| `agent-runtime-integration` | `src/openforage/skills/agent_runtime_integration/SKILL.md` | Bridge OpenForage to a Claude / Codex / Hermes / OpenClaw runtime. |
