# Testing

All the test surfaces in this repo — unit tests, end-to-end harnesses,
benchmarks, and CI — in one place. Use this to pick the right thing to run
for the change you just made.

## Quick start — which test should I run?

| Change I made | Run this |
|---|---|
| Anything touched by CI (ruff + mypy + pytest) | `python -m pytest tests/` |
| Memory core, MCP tool schemas, dispatch logic | `python bin/test_memory_bridge.py` |
| Chatlog subsystem (ingest / queue / redact / status) | `python -m pytest tests/test_chatlog_*.py` |
| `bin/slm_intent.py` or the SLM profiles | `python -m pytest tests/test_slm_intent.py` |
| MCP proxy dispatch | `python -m unittest bin.test_mcp_proxy_unit` |
| Retrieval quality on a real dataset | See published LME-S report — harnesses are maintained internally |

Full test inventory and what each exercises follows.

---

## Layout

Three distinct test surfaces, each with different run semantics:

```
tests/                        # pytest — CI target, no external deps
bin/test_*.py                 # standalone e2e harnesses (print PASS/FAIL)
benchmarks/                   # retrieval quality evaluation on real datasets
```

`pyproject.toml` pins `testpaths = ["tests"]`, so `pytest` on the repo root
only runs the `tests/` suite. The `bin/test_*.py` scripts are invoked
directly (`python bin/test_foo.py`) and exit with a summary. The benchmark
harnesses are evaluation drivers, not pass/fail gates — they produce
hypotheses, judge scores, and CSV/JSONL artifacts for offline analysis.

### CI wiring

`.github/workflows/ci.yml` runs three jobs on every push + PR to `main`:

1. **Lint (Ruff)** — `ruff check bin/ memory/`
2. **Type check (Mypy)** — `mypy bin/ --ignore-missing-imports`
3. **Test** — matrix of `{ubuntu, macos, windows} × {3.11, 3.12}`, each
   runs `pytest tests/`

Neither the `bin/test_*.py` harnesses nor the benchmarks run in CI (they
need LM Studio, a populated DB, or a real embedding endpoint).

---

## 1. pytest suite (`tests/`)

**Total**: 152 passed / 2 skipped across 18 files (as of 2026-04-21).
**Requirements**: none — all DB and LLM access is mocked or uses `tmp_path`.

| File | Tests | What it covers |
|---|---|---|
| `test_auth_utils.py` | 3 | Master-key retrieval, Fernet encrypt/decrypt, device-salt persistence |
| `test_chatlog_config.py` | 11 | Config resolution hierarchy (env > ContextVar > file > default), legacy-mode deprecation, caching |
| `test_chatlog_cost_report.py` | 4 | Cost aggregation by provider/model, token counting, null-cost exclusion |
| `test_chatlog_ingest_formats.py` | 10 | Claude Code + Gemini CLI schema parsing, provider inference, ingest idempotency |
| `test_chatlog_migrations.py` | 8 | Schema bootstrap + rollback, index creation, FTS table, triggers |
| `test_chatlog_perf.py` | 5 | Enqueue throughput, flush latency, batch-write latency, metadata overhead |
| `test_chatlog_redaction.py` | 13 | API-key / JWT / AWS / GitHub token scrubbing, PII, custom regex, compile cache |
| `test_chatlog_roundtrip.py` | 7 | Write → flush → search, conversation listing, metadata preservation |
| `test_chatlog_status.py` | 9 | Status JSON schema, row counts, queue depth, spill info, cold-start perf |
| `test_chatlog_status_line.py` | 9 | One-line status indicators, queue/spill warnings, severity ordering |
| `test_chatlog_write_queue.py` | 12 | Item validation, bulk queueing, spill file creation, cost fields |
| `test_content_safety.py` | 2 | Malicious-pattern detection (eval / exec / SQLi) + benign-content allowlist |
| `test_embedding_utils.py` | 3 | Vector pack/unpack, cosine similarity, batch cosine with heterogeneous dims |
| `test_llm_failover.py` | 3 | Model-size parsing, embed-model discovery, LLM selection heuristic |
| `test_pg_sync_fk_safety.py` | 3 | FK-safe embedding sync: parent presence, orphan deferral |
| `test_recency_bonus.py` | 8 | Recency-bonus scoring, date interpolation, undated handling, stable tie-break |
| `test_slm_intent.py` | 11 | Gate off → None, profile loader, search-dir stacking, label matching |
| `test_task_tombstones.py` | 7 | Soft/hard delete, visibility, tombstone persistence, update rejection |

### Running

```bash
# Full suite
python -m pytest tests/

# Single file with verbose output
python -m pytest tests/test_slm_intent.py -v

# Exclude the two slow tests
python -m pytest tests/ -m "not slow"

# Against an isolated DB (see bin/setup_test_db.py)
python bin/setup_test_db.py --database memory/_test.db --force
M3_DATABASE=memory/_test.db python -m pytest tests/
```

---

## 2. End-to-end harnesses (`bin/test_*.py`)

These are self-contained scripts that print a `✅/❌` summary. They exist
for scenarios that need a real live DB, a running MCP server, or LM Studio
on localhost — things pytest would have to mock extensively.

All honor `M3_DATABASE` so you can point them at a scratch DB:

```bash
python bin/setup_test_db.py --database memory/_test.db --force
M3_DATABASE=memory/_test.db python bin/test_memory_bridge.py
```

| File | Invocation | Count | Requires | Covers |
|---|---|---|---|---|
| `test_memory_bridge.py` | `python bin/test_memory_bridge.py` | 193 checks across 47 scenarios | LM Studio (skips embedding tests if offline); DB via `M3_DATABASE` | All 38+ MCP tools: memory write/get/search/update/delete, conversations, tasks, handoffs, inbox, agent registry, notifications, GDPR, sync |
| `test_debug_agent.py` | `python bin/test_debug_agent.py` | ~25 checks | LM Studio (graceful skip); DB via `M3_DATABASE` | Debug agent bridge: thermal, LLM routing, debug_analyze / bisect / trace / correlate / history / report |
| `test_mcp_proxy.py` | `python bin/test_mcp_proxy.py` | 4 scenarios, ~10 checks | MCP proxy running on :9000, `ANTHROPIC_API_KEY`, DB | Proxy end-to-end: `/health`, Claude + Gemini routing, aider subprocess, activity_logs delta |
| `test_mcp_proxy_unit.py` | `python -m unittest bin.test_mcp_proxy_unit` (or pytest) | ~15 methods | none | Proxy internals: tool-list merging, destructive-tool gating, `inject_agent_id` enforcement |
| `test_bulk_parity.py` | `python bin/test_bulk_parity.py` | ~15 checks | SQLite (mocked schema) | Bulk vs single write parity: enrichment, variant isolation, contradiction detection |
| `test_embedding_logic.py` | `python bin/test_embedding_logic.py` | 2 ops | LM Studio, `LM_API_TOKEN` | Embed path smoke: token resolution, `_embed()` round-trip |
| `test_keychain.py` | `python bin/test_keychain.py` | 3 lookups | OS keychain or env vars | `auth_utils.get_api_key()` across env / keyring / platform-native stores |
| `test_unified_router.py` | `python bin/test_unified_router.py` | 3 payloads | LM Studio on :1234, `LM_API_TOKEN`, optional `ANTHROPIC_API_KEY` / `GEMINI_API_KEY` | OpenAI-compatible router: local dispatch + cloud fallback (Claude / Gemini) |
| `test_mission_control.py` | `python bin/test_mission_control.py` | ~18 checks | LM Studio (skip if offline), nvidia-smi (skip if no GPU), network (skip if unreachable) | Status dashboard: platform detect, paths, KV stats, GPU / VRAM, ping, single render pass |

### Routing test output to a scratch DB

The three harnesses that take over the live DB (`test_memory_bridge.py`,
`test_debug_agent.py`, `test_mcp_proxy.py`) resolve `DB_PATH` at import
time via `m3_sdk.resolve_db_path(None)`, so setting `M3_DATABASE` before
invocation isolates them. Typical flow:

```bash
python bin/setup_test_db.py --database memory/_test.db --force
M3_DATABASE=memory/_test.db python bin/test_memory_bridge.py
M3_DATABASE=memory/_test.db python bin/test_debug_agent.py
```

See [CLI_REFERENCE.md](CLI_REFERENCE.md#running-tests-against-an-isolated-db)
for details.

---

## 3. Benchmark harnesses (`benchmarks/`)

Retrieval-quality benchmarking (LongMemEval, LoCoMo) is maintained
internally; see the published
[LME-S report](../benchmarks/longmemeval/LME-S_Benchmarking_Report.md)
for methodology and results.

---

## 4. Running the full test ladder

When merging a large change and you want maximum coverage:

```bash
# 1. CI-equivalent (fast, no external deps)
ruff check bin/ memory/
mypy bin/ --ignore-missing-imports
python -m pytest tests/

# 2. Bridge + subsystem e2e (needs LM Studio + live DB)
python bin/setup_test_db.py --database memory/_test.db --force
M3_DATABASE=memory/_test.db python bin/test_memory_bridge.py
M3_DATABASE=memory/_test.db python bin/test_debug_agent.py
python -m unittest bin.test_mcp_proxy_unit

# 3. Benchmark smoke — internal harness, see published LME-S report
```

---

## 5. Writing new tests

**Prefer `tests/` (pytest)** for anything that:
- Can mock its dependencies
- Doesn't need a running LM Studio / Ollama
- Should gate merges via CI

**Use `bin/test_*.py`** only when:
- The scenario truly needs a live MCP server or live LM Studio
- You're debugging end-to-end behavior and need to print intermediate state
- A pytest version would be drowning in mocks

**Use `benchmarks/`** only for:
- Retrieval quality measurement against a real dataset
- Ingest-cost profiling
- Anything that produces artifacts for offline analysis

### Writing a new pytest file

Place under `tests/test_<subject>.py`. Follow the `test_slm_intent.py`
pattern:

- `conftest.py` provides a shared `_isolate_chatlog_env` fixture
- Use `tmp_path` and `monkeypatch` for all file / env isolation
- Mark slow tests with `@pytest.mark.slow`
- No hard-coded paths — everything relative to `tmp_path`

### Adding a gate-controlled module

If your feature is env-gated (like `M3_SLM_CLASSIFIER`), test both states:

```python
def test_feature_returns_none_when_gate_off(monkeypatch):
    monkeypatch.delenv("M3_FEATURE_GATE", raising=False)
    assert my_feature("input") is None

def test_feature_runs_when_gate_on(monkeypatch, tmp_path):
    monkeypatch.setenv("M3_FEATURE_GATE", "1")
    # ... positive path ...
```

The `test_slm_intent.py` module demonstrates this end-to-end.

---

## PostgreSQL-backed tests (`requires_pg`)

m3 runs on both SQLite (default) and PostgreSQL through one storage seam. A set of
`*_pg_live.py` suites verify the PG path against a **real** cluster — schema parity,
the maintenance pass, the files store, chatlog, etc. They are marked
`@pytest.mark.requires_pg`.

**Running them.** Point the suite at any throwaway PostgreSQL via the DSN env vars
and run as normal:

```bash
export M3_PRIMARY_PG_URL="postgresql://<user>:<pass>@<host>:5432/<throwaway_db>"
export M3_PG_URL="$M3_PRIMARY_PG_URL"          # fallback the tests also read
python -m pytest -m requires_pg -q             # every PG-live test
python -m pytest tests/test_schema_parity_pg_live.py -q   # the migration-parity gate
```

Use a **disposable** database — the fixtures create/drop schemas freely. The DSN is
resolved `M3_PRIMARY_PG_URL` > `M3_PG_URL`; the suite deliberately **never** reads
`PG_URL` (that names the production/warehouse store). Applying the schema to a fresh
DB is `psql -f memory/migrations/postgres/pg_primary_v1.sql` then
`migrate_pg.run_pending_pg_migrations(conn)` (pass a live connection, not a DSN
string); the files-store schema builds itself via its own migrations on first
`files_memory.db.init_db()`.

**Auto-skip is by design.** With no DSN set (or no reachable cluster), every
`requires_pg` test **skips cleanly** — that is expected on a SQLite-only checkout and
in the default CI lane, not a failure. A conftest collection hook probes reachability
once per session and skips the marked tests when the probe fails.

> ⚠️ **Silent-skip footgun.** Because that reachability result is cached for the whole
> session, a *single* failed probe skips **all** PG-live tests for the run — a false
> green that hides real PG regressions. The probe therefore retries a few times before
> giving up (`tests/conftest.py::_pg_reachable`), so a momentary blip (the cluster
> still starting, a service restart mid-run) doesn't poison coverage. If your PG tests
> **skip while a cluster is up**, the probe isn't reaching it: verify the DSN host is
> actually accepting TCP connections from where pytest runs (a common cause is a DB
> bound to loopback-only behind a flaky port forward — bind it so your test host can
> reach it directly). A quick check: `python -c "import psycopg2,os;
> psycopg2.connect(os.environ['M3_PRIMARY_PG_URL']).close(); print('reachable')"`.

Native Windows PostgreSQL is not a supported test target on some setups (its
per-connection fork model can crash under a locked-down DLL environment); a Linux
cluster — including one inside WSL — is the reliable path. Maintainers keep the
machine-specific stand-up recipe (creating the throwaway role/db, binding the
cluster so the test host can reach it) in the internal runbooks, since it carries
local credentials and paths that don't belong in the public tree.

---

## 6. Related docs

- [`docs/CLI_REFERENCE.md`](CLI_REFERENCE.md) — every DB-aware CLI, including
  `bin/setup_test_db.py` for isolated test DBs
- [`docs/CHATLOG.md`](CHATLOG.md) — architecture of the chatlog subsystem
  exercised by ~60 of the pytest cases
- [`docs/SLM_INTENT.md`](SLM_INTENT.md) — SLM classifier subsystem (tested
  by `tests/test_slm_intent.py`)
- [`CHANGELOG.md`](../CHANGELOG.md) — dated record of which test
  counts landed with which refactor
