# v5.1.0 — server / storage module split

**Status:** planned. Lands in PR #66 after v5.0.1 ships from PR #65.
**Semver:** minor (5.0.x → 5.1.0). Import paths change for downstream consumers; the public MCP surface is unchanged.
**Motivation:** `server.py` (4 268 LOC, 97 top-level defs, 39 `@_tool` decorations) and `storage.py` (3 736 LOC, ~115 public methods on `StorageEngine`) are the two largest source files in the repo and the dominant rebase-conflict surface for any concurrent change.

## Context

Inventory captured against master tip `042f42b`:

| File | LOC | Top-level defs | Notes |
|---|---:|---:|---|
| `yadgar/server.py` | 4 268 | 97 | 39 `@_tool` decorations (15 plain, 24 `power=True`) |
| `yadgar/storage.py` | 3 736 | 7 module-level + ~115 `StorageEngine` methods | All methods on one class |

Stage 11 (v5.0) decomposed `retrieval/core.py::recall` and `causal_discovery.py::pc_algorithm` safely because Stage 6 wrote **characterization tests** pinning their behavior. `server.py` and `storage.py` have no equivalent. Splitting them without that gate multiplies the kind of copy-paste oversight that produced **C-1** (auth middleware missing from SSE transport — found in the 2026-05-16 review).

The gate must be built first.

---

## Stage 1 — Characterization tests (gate, do this first)

Pin current behavior of every route handler and every `StorageEngine` method whose split is in scope.

### Route tests (new file `yadgar/tests/test_server_routes_characterization.py`)

Snapshot tests against `mcp_server.streamable_http_app()`:

- `/health` — response shape + status code; both auth modes
- `/metrics` — Prometheus exposition format; `YADGAR_METRICS_ENABLED=0` returns 404
- `/hooks/auto-capture` — POST body round-trip; sanitization applied; rate-limit fires; batched flush at 5
- `/hooks/prompt-recall` — DLQ alert text composition
- `/hooks/session-context` — bearer-auth required; markdown body returned
- `/api/graph` — node + edge shape
- `/api/stats` — counts
- `/api/system` — metrics cache snapshot
- `/api/heat-histogram` — bin count
- `/api/metrics/consolidation-log` — log entry shape
- `/api/graph/neighborhood/{node_id}` — hop expansion

### Storage tests (new file `yadgar/tests/test_storage_characterization.py`)

Pin behavior across the methods that have the highest blast radius:

- `insert_memory` + `delete_memory` cascade — verify FK cleanup across `entity`, `vector`, `memory_transition`, `memory_similarity_link`, `wiki_crossref` tables
- `batch_writes` under concurrent load (`asyncio.gather` × N writers)
- `_run_migrations_locked` idempotency — run twice, assert same row count and no errors
- `search_wiki_fts_scored` scoring stability — fixture corpus → expected ranked IDs with `math.isclose(rel_tol=1e-9)` parity check
- `recreate_vector_table` rollback semantics — inject failure mid-transaction, assert sidecar backup is readable

### Circular-import guard

A test that does `import yadgar.server` cold, without engine initialization. The split must preserve the lazy-import pattern that `file_queue.py` and `consolidation.py` already use to avoid module-load-time cycles.

```python
def test_server_module_import_does_not_pull_file_queue():
    import sys
    sys.modules.pop("yadgar.file_queue", None)
    sys.modules.pop("yadgar.consolidation", None)
    import yadgar.server  # noqa: F401
    assert "yadgar.file_queue" not in sys.modules
    assert "yadgar.consolidation" not in sys.modules
```

Tests must pass green on the pre-split master tip before any source moves. They are the spec.

---

## Stage 2 — server.py split (low-coupling groups first)

Target structure:

```
yadgar/server/
├── __init__.py              # re-exports public API for one-release deprecation
├── _state.py                # module-level singletons + locks (Group 1)
├── deps.py                  # _get_* lazy accessors (Group 3)
├── middleware.py            # CORS + bearer auth + surrogate guard (Group 4)
├── lifespan.py              # init_engines, shutdown, main, _load_default_rules (Group 2)
├── events.py                # SSE event queue + bounded set + make_event_stream (Group 6 split)
├── capture.py               # _action_batch + drainer (Group 13)
├── git_utils.py             # _resolve_project_root, _detect_branch, default-branch (Group 8)
├── routes/
│   ├── __init__.py
│   ├── health.py            # /health, /metrics, /graph view (Group 5)
│   ├── hooks.py             # /hooks/* (Group 6, minus events)
│   └── api.py               # /api/* read-only data routes (Group 7)
└── tools/
    ├── __init__.py
    ├── memory.py            # memorize, recall, forget, validate_memory (Group 9)
    ├── admin.py             # check_invariants, vacuum_now, consolidate_now, reembed_all, memory_stats (Group 10)
    ├── project.py           # project_brief, bootstrap_project, update_active_work (Group 11a)
    ├── wiki.py              # wiki_add, wiki_query, wiki_read, wiki_get, wiki_update, wiki_approve, wiki_discard, wiki_list, wiki_lint, wiki_refresh_stale, wiki_cleanup_merged_branches, wiki_drafts (Group 11b)
    └── ops.py               # add_rule, get_rules, checkpoint, restore, anchor, install_hooks, sync_instructions, dlq_inspect, dlq_requeue, memory_get, memory_update, seed_project (Group 12)
```

### Order

| Stage | Groups moved | Risk |
|---|---|---|
| 2a | `deps.py` (Group 3) | Lowest — read-only accessors |
| 2b | `git_utils.py` (Group 8) | Low — pure computation, `_project_roots` read only |
| 2c | `routes/api.py` (Group 7) | Low — no callers in other groups |
| 2d | `routes/health.py` (Group 5) | Low |
| 2e | `middleware.py` (Group 4) + `events.py` (Group 6 partial) | Medium |
| 2f | `routes/hooks.py` (Group 6 main) | Medium |
| 2g | `lifespan.py` (Group 2) + `_state.py` (Group 1) | Medium — touches every singleton |
| 2h | `tools/admin.py` + `tools/project.py` + `tools/wiki.py` | Medium |
| 2i | `tools/memory.py` (Group 9) | High — `memorize` alone is ~380 LOC, writes `_action_batch` |
| 2j | `tools/ops.py` (Group 12) | High — heterogeneous; consider further re-split in v5.2 |
| 2k | `capture.py` (Group 13) | High — circular-dep risk with `tools/memory.py` |

Each stage = one commit. Characterization tests must stay green after each.

---

## Stage 3 — storage.py split

Target structure:

```
yadgar/storage/
├── __init__.py              # re-exports StorageEngine for one-release deprecation
├── engine.py                # StorageEngine + __init__ + _db + _verify_health + _restore_from_backup + batch_writes + close + _q + _q_timeout + _send_chunk + _build_chunk_body
├── migrations.py            # _init_schema + _run_migrations + _run_migrations_locked + _migration_00*
├── memory.py                # memory query + write methods
├── vector.py                # vector table CRUD + search + recreate
├── episode.py               # insert_episode + get_*_episodes + prune
├── graph.py                 # entity + relationship + memory_similarity_link + causal_edge
├── wiki.py                  # wiki_page + wiki_draft + wiki_crossref + search
├── engram.py                # transition + slot allocation
├── checkpoint.py            # checkpoint + epoch + protect
├── profile.py               # profile + belief
├── prospective.py           # prospective_memory + narrative + astrocyte_process
├── rules.py                 # memory_rule CRUD
├── action_log.py            # action_log
├── cluster.py               # cluster CRUD
├── maintenance.py           # file_hash + consolidation_log + db_size
└── project.py               # upsert_project_init + upsert_active_work
```

Split via class-mixin composition or extracted module-level functions taking `self` (less common; mixins are the standard pattern for splitting a god-class):

```python
# yadgar/storage/engine.py
class StorageEngine(
    MemoryMixin,
    VectorMixin,
    EpisodeMixin,
    GraphMixin,
    WikiMixin,
    # ...
):
    def __init__(self, ...):
        ...
```

Each mixin is one file with the methods that operate on that table.

### Order

| Stage | Mixin extracted | Risk |
|---|---|---|
| 3a | `vector.py` (`VectorMixin`) | Low — clean interface, called only by memory query + reembed |
| 3b | `episode.py` (`EpisodeMixin`) | Low — single table, no FK writes from outside |
| 3c | `cluster.py` (`ClusterMixin`) | Low |
| 3d | `wiki.py` (`WikiMixin`) | Low — single-table cluster |
| 3e | `action_log.py` (`ActionLogMixin`) | Low |
| 3f | `engram.py` (`EngramMixin`) | Medium |
| 3g | `migrations.py` (`MigrationsMixin`) | Medium — runs at startup |
| 3h | `rules.py` + `prospective.py` + `profile.py` + `checkpoint.py` | Medium |
| 3i | `graph.py` (`GraphMixin`) | High — `delete_memory` FK cascade across entity, vector, transition |
| 3j | `memory.py` (`MemoryMixin`) | Highest — core table, ~730 LOC of mixed query + write methods |

---

## Backwards-compat shim

`yadgar/server.py` becomes a thin module that re-exports the public surface:

```python
# yadgar/server.py — kept for one release as a deprecation shim
from yadgar.server import mcp_server, main  # noqa
from yadgar.server.lifespan import init_engines, shutdown  # noqa
from yadgar.server.tools.memory import memorize, recall, forget, anchor, checkpoint, restore  # noqa
# ... full re-export of every public name

import warnings
warnings.warn(
    "Importing from yadgar.server is deprecated; use yadgar.server.tools.* or yadgar.server.lifespan.* directly. "
    "Compat shim will be removed in v5.2.",
    DeprecationWarning,
    stacklevel=2,
)
```

Same pattern for `yadgar/storage.py` re-exporting `StorageEngine`.

Plan to remove the shim in v5.2 once downstream is verified migrated.

---

## Files touched

| File | Action |
|---|---|
| `yadgar/server.py` | Delete (replaced by deprecation shim at the same path importing from `yadgar/server/__init__.py`) |
| `yadgar/server/__init__.py` | New — re-exports public API |
| `yadgar/server/*.py` | New — split per Stage 2 table |
| `yadgar/server/routes/*.py` | New |
| `yadgar/server/tools/*.py` | New |
| `yadgar/storage.py` | Delete (replaced by deprecation shim) |
| `yadgar/storage/__init__.py` | New — re-exports `StorageEngine` |
| `yadgar/storage/*.py` | New — split per Stage 3 mixin |
| `yadgar/tests/test_server_routes_characterization.py` | New — Stage 1 gate |
| `yadgar/tests/test_storage_characterization.py` | New — Stage 1 gate |
| `MIGRATION_NOTES.md` | Add v5.1 section documenting recommended import-path migration |
| `pyproject.toml` | Bump `5.0.x → 5.1.0` |
| `docs/CHANGELOG.md` | Add `[5.1.0]` entry under Changed |

---

## Verification

After every stage commit:

```bash
# syntax
.venv/bin/python -c "import ast; [ast.parse(open(p).read()) for p in __import__('glob').glob('yadgar/**/*.py', recursive=True)]; print('OK')"

# import shim still works
.venv/bin/python -c "from yadgar.server import memorize, recall, mcp_server; from yadgar.storage import StorageEngine; print('shim OK')"

# circular-import guard
.venv/bin/python -c "import sys; sys.modules.pop('yadgar.file_queue', None); sys.modules.pop('yadgar.consolidation', None); import yadgar.server; assert 'yadgar.file_queue' not in sys.modules"

# characterization tests
YADGAR_TEST=1 YADGAR_ALLOW_ROOT=1 .venv/bin/python -m pytest \
  yadgar/tests/test_server_routes_characterization.py \
  yadgar/tests/test_storage_characterization.py \
  yadgar/tests/test_retrieval_core_characterization.py \
  yadgar/tests/test_causal_discovery_characterization.py \
  -o addopts="" --timeout=60 -q
```

Full test suite green at every stage. Behavior pinned by characterization tests must not drift.

---

## Stage 4 — Test-suite performance

Current state (read-only investigation 2026-05-16):

- **57+ min** CI wall-time on 1 526 tests / 92 files
- **CI runs `-n auto` on a 2-vCPU runner** — that's 2 workers, not the `-n 4` declared in `pyproject.toml`
- **34 files marked `pytest.mark.xdist_group("server_globals")`** (Stage 13 mask). 761 of 1 526 tests = **50% of the suite** serialized into one worker
- **`_isolate_surrealdb` autouse fixture** calls `_init_schema()` per test: 1 526 × 30–50 ms ≈ 30–76 s overhead
- **`embeddings` session fixture exists in `conftest.py`** but `test_curation.py` and `test_consolidation.py` define **function-scoped overrides** that rebuild `EmbeddingEngine` per test. `test_curation.py` also constructs one at module import.
- **`--reruns 2 --reruns-delay 2` applied globally in CI** — no tests have `@pytest.mark.flaky`, so the global rerun is overkill
- **`surreal_server`** itself is correctly session-scoped, single-process, SIGKILL escalation on teardown — no change needed

### Wall-time opportunities (ranked by impact)

| # | Change | Est. saving | Risk |
|---|---|---:|---|
| 1 | Promote CI runner from 2-vCPU `ubuntu-latest` to 4-core (or self-hosted) — match `-n 4` from pyproject. Removes the half-suite tail wait. | 15–25 min | Infra change; needs runner config |
| 2 | Cache `_init_schema()` per DB namespace in `_isolate_surrealdb` — `_SCHEMA_INITIALIZED: set[str]` guard so already-init'd DBs skip the round-trip | 5–15 min | Low; pure caching |
| 3 | Remove function-scoped `embeddings` overrides in `test_curation.py` and `test_consolidation.py`; drop module-level `_engine = EmbeddingEngine()` at `test_curation.py:19` | 3–8 min | Low; just deletes lines |
| 4 | Switch CI `--dist` to `worksteal` while keeping `loadgroup` for `xdist_group`-marked tests — lets the free worker steal idle work from the serialized one | 2–4 min | Low; needs flag tweak |
| 5 | Mark characterization suites `@pytest.mark.slow` + skip-by-default; opt-in via `pytest -m slow` for the v5.1 split work and final-PR audits only | 2–5 min | Low; tests still run on demand |
| 6 | Drop `--reruns 2 --reruns-delay 2` globally; add `@pytest.mark.flaky(reruns=2)` to specific xdist-isolation flakes if they reappear | 0–2 min steady; removes latency tax on actual failures | Low |
| 7 | **Fix the state leak that forced 34 `xdist_group` markers** (task #15, still pending). Once fixed, remove the markers and let the full suite parallelize | 10–20 min | High effort; proper fix |

### Target

Bring 57 min suite → **under 15 min** by combining items 1–4 (no leak fix required). Adding item 7 brings it under 10 min.

### Subprocess-leak hardening (test surreal-server)

**Observed 2026-05-16:** 93 stale `surreal start` subprocesses on local host, ~12 GB RSS, on a 64 GB box. Two contributors:

| Source | How it leaks | Fix |
|---|---|---|
| Local agent worktrees running pytest | Worktree gets reaped by Claude harness mid-run; surreal subprocess survives | PID-file tracking + cross-session reap |
| Self-hosted CI runner accepting concurrent jobs | `concurrency.group: ${{ workflow }}-${{ ref }}` cancels only within same ref; master + PR + tag pushes run three concurrent surreals | Drop `-${{ ref }}` suffix → global single-slot |

#### `surreal_server` hardening (`yadgar/tests/conftest.py`)

```python
import atexit, os, signal
from pathlib import Path

_PID_DIR = Path("/tmp/yadgar-test-surreal")
_PID_DIR.mkdir(exist_ok=True)


def _count_alive_pids() -> int:
    alive = 0
    for pf in _PID_DIR.glob("surreal-*.pid"):
        try:
            pid = int(pf.read_text().strip())
            os.kill(pid, 0)  # existence check
            alive += 1
        except (FileNotFoundError, ValueError, ProcessLookupError):
            pf.unlink(missing_ok=True)
    return alive


@pytest.fixture(scope="session", autouse=True)
def surreal_server(tmp_path_factory):
    if (stale := _count_alive_pids()) > 5:
        pytest.exit(
            f"Refusing to spawn — {stale} stale surreal processes alive. "
            f"Run scripts/kill-stale-surreals.sh first."
        )
    # ... existing spawn body ...
    (_PID_DIR / f"surreal-{port}.pid").write_text(str(proc.pid))

    def _cleanup():
        try:
            os.kill(proc.pid, signal.SIGKILL)
        except ProcessLookupError:
            pass
        (_PID_DIR / f"surreal-{port}.pid").unlink(missing_ok=True)

    atexit.register(_cleanup)
    yield ...
    _cleanup()
```

#### `scripts/kill-stale-surreals.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail
before=$(pgrep -fc "surreal start" || true)
for pid in $(pgrep -f "surreal start"); do
  args=$(ps -o args= -p "$pid" 2>/dev/null || true)
  case "$args" in
    *"bind 0.0.0.0:8000"*) : ;;   # production daemon — keep
    *) kill -9 "$pid" 2>/dev/null || true ;;
  esac
done
rm -f /tmp/yadgar-test-surreal/*.pid
echo "Surreal: ${before} -> $(pgrep -fc 'surreal start' || true)"
```

#### CI concurrency change (`.forgejo/workflows/ci.yaml`)

```yaml
# Before — cancels only within same ref; cross-ref parallel on self-hosted
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

# After — global single-slot per workflow
concurrency:
  group: yadgar-ci-${{ github.workflow }}
  cancel-in-progress: false   # queue, don't cancel
```

Tradeoff: queue depth grows on push storms, but at most one CI surreal alive at any moment on the self-hosted runner.

#### Document in `docs/claude-workflow.md`

> **Agent worktree subprocess leak guard.** When dispatching `Agent(isolation="worktree", run_in_background=true)` for any task running pytest, the `surreal_server` fixture spawns a subprocess. If the agent task is killed (timeout, harness reap, OOM), that subprocess survives. Before launching a new agent batch, run `scripts/kill-stale-surreals.sh` if `pgrep -fc "surreal start"` returns more than ~3.
>
> The fixture writes a PID file at `/tmp/yadgar-test-surreal/surreal-<port>.pid` so subsequent test sessions detect and reap stale instances at startup. Sessions abort if more than 5 stale survive.

### Stage 4 commits

| Commit | Subject |
|---|---|
| `test: cache _init_schema per DB namespace` | item 2 |
| `test: drop per-test EmbeddingEngine in test_curation/consolidation` | item 3 |
| `ci: --dist worksteal, drop global reruns` | items 4 + 6 |
| `test: @pytest.mark.slow on characterization suites` | item 5 |
| `ci: bump runner to 4-core (if available) or self-host` | item 1 — operator action |
| `test: PID-file + atexit cleanup in surreal_server fixture` | leak hardening |
| `test: pre-session stale-surreal abort` | leak hardening |
| `scripts: kill-stale-surreals.sh helper` | leak hardening |
| `ci: tighten concurrency group to global` | leak hardening |
| `docs(claude-workflow): subprocess leak guard procedure` | leak hardening |
| `test: remove xdist_group markers (depends on leak fix)` | item 7 — already shipped in PR #65 via cherry-pick of `6b567f1` / `156ce69` / `692bc49` |

### Verification

Capture before/after wall-time on three runs each:

- `time .venv/bin/python -m pytest yadgar/tests/ -o addopts=""` (local, no xdist) — single-worker baseline
- `time .venv/bin/python -m pytest yadgar/tests/` (with pyproject addopts `-n 4 loadgroup`)
- CI run on PR — full pipeline timing

After hardening, verify leak gone:

- Spawn 10 test sessions via worktree-isolated agents, all killed mid-run
- `pgrep -fc "surreal start"` should drop to 1 (production) within 30 s
- `ls /tmp/yadgar-test-surreal/*.pid` should be empty

Record before/after wall-time in `docs/CHANGELOG.md` `[5.1.0]` entry as a concrete delta.

---

## Out of scope for v5.1

- Splitting `yadgar/consolidation.py` (1 500+ LOC) — deferred to v5.2 once split pattern is proven on server/storage.
- Splitting `yadgar/retrieval/core.py` further — Stage 11 already decomposed `recall()` into helpers; module size is acceptable.
- Mega-function decomposition of `memorize` (~380 LOC) — separate concern; tackle in v5.2 after split.
- Removing the deprecation shims — v5.2.
