Metadata-Version: 2.4
Name: p-layers
Version: 0.6.0
Summary: Governed memory for AI agents — P0-P6 layer governance enforced in code. SQLite + FTS5 + pluggable embeddings, PostgreSQL backend, MCP server, eval harness. (GitHub: p-layer)
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# p-layer

**Governed memory for AI agents.** A stdlib-only Python memory layer — SQLite + FTS5 + pluggable embeddings — with P0-P6 layer governance enforced in code, not prose.

[![CI](https://github.com/humanerd-drew/p-layers/actions/workflows/ci.yml/badge.svg)](https://github.com/humanerd-drew/p-layers/actions/workflows/ci.yml)
```
7 layers. 1 memory. Every write audited.
```

[한국어](./README.ko.md)

## Why this exists

The P0-P6 "brain layer" memory idea (drewgent, p-layer) is sound, but the reference implementations carry the same core defects:

| Defect | drewgent / p-layer | p-layers |
|---|---|---|
| Schema management | `CREATE TABLE IF NOT EXISTS` everywhere, no versioning | Forward-only, checksummed migrations (`schema_migrations`) |
| Search index | External-content FTS5 + triggers (fragile; p-layer's `forget` breaks it) | Standalone FTS5, no trigger coupling |
| Dual backends | Two parallel implementations that drift (TS + Python; SQLite + Pg with silent feature loss) | One implementation, one schema |
| Governance | A table in the README ("P0 overrides everything") | **Enforced in code**: layer ACLs raise `WriteDenied` |
| "Remember" tool | Hardcodes layer=P6, bypassing its own governance | Layer is a first-class write parameter, ACL-checked |

This repo is the production-grade rebuild: the governance ideas ported from p-layer, the schema discipline the originals lacked, and an eval harness that proves governance improves retrieval.

## What it does

| Feature | What |
|---|---|
| **P0-P6 layer ACLs** | Who may write to each layer is enforced in code (`P0` system-only … `P6` agent+manual). Denied writes are audited. |
| **Hybrid recall** | FTS5 + semantic (Ollama or pluggable), RRF fusion, ranked by confidence × freshness, type-diversified, superseded excluded. |
| **Supersede-not-delete** | `forget`/`update` supersede entries; history is preserved and recall stops surfacing them. |
| **Snapshots** | Freeze active entries under a version label; rollback supersedes everything after the snapshot. |
| **Audit log** | Every write *and every denied write* is recorded — the compliance evidence. |
| **Contradiction scan** | Heuristic scan (no LLM): conflicting rule priorities, cross-layer duplicates. |
| **P5 wiki compile** | Offline compile of active memory into per-layer markdown with provenance + INDEX. |
| **MCP server** | 12 tools (`remember`, `recall`, `forget`, `update`, `snapshot_*`, `memory_stats`, `memory_audit`, `assemble`, `graph_explore`, `graph_trace`, `graph_rca`) — zero-dependency stdio implementation, any client. |
| **Import tool** | `import-drewgent` migrates an existing drewgent `knowledge.db` (schema re-validated, re-embedded, sessions carried into episodes). |
| **Graph & inference** | `graph_explore` / `graph_trace` / `graph_rca` (caused/fixed_by chains) / `transitive_closure` — drewgent graph_query.py parity, cycle-safe traversal. |
| **Vault ingest** | `import-rules` (rules.md → rules) and `import-incidents` (P6 incidents → episodes) — the vault stays files, p-layers references it. |
| **p-layers compat** | This package is published on PyPI as **`p-layers`** (GitHub repo: p-layer). `p_layer/` keeps the 0.1.x `KnowledgeDB` API and `knowledge_*` MCP tools over this engine — existing p-layers integrations upgrade without code changes. |
| **Re-embed job** | `reembed` backfills embeddings after a model switch; vectors are versioned (old versions stay queryable), recall only reads the current version. Idempotent. |
| **Consolidation** | `consolidate` compresses unconsolidated episodes into `insight` digests — deterministic offline summarizer, pluggable LLM hook, idempotent, audited. |
| **PostgreSQL backend** | `PgStore` — the same interface, governance, and parity-tested behavior on Postgres (pg_trgm ILIKE for CJK, pgvector semantic). Ops jobs stay SQLite-only, loudly. |

## Proof: governance improves retrieval

Same data, two engines, one command (`p-layer eval suite.json`):

```
recall@k (same data, two engines):
  drewgent baseline : 0.667 (2/3)      ← naive FTS OR-join, insertion order
  p-layer            : 1.000 (3/3)      ← confidence/freshness-ranked
  delta             : +0.333
ACL compliance: 100.0% (30/30) enforcement cases correct
```

The baseline can't move — it has no metadata. p-layers turns governance metadata (confidence, layer, supersession) into retrieval quality, and the ACL suite proves the governance is real: every (layer, who) combination is allowed or denied exactly as specified.

## Quick start

```bash
# no dependencies — stdlib only (Python >= 3.9)
export P_LAYER_EMBED=hash   # offline fallback; ollama is the default
export P_LAYER_DB=~/.p_layer/memory.db

python3 -m p_layer init
python3 -m p_layer remember "switched to portone v2 for payments" --type decision --layer P5
python3 -m p_layer recall "portone"
python3 -m p_layer assemble --budget 12000     # rules first, then recent knowledge
```

Python API:

```python
from p_layer.store import Store, WriteDenied

db = Store()
db.add_knowledge("client prefers weekly sync", type="preference", layer="P6", who="agent")
print(db.recall("weekly sync", limit=5))
try:
    db.add_knowledge("secret", layer="P0", who="agent")   # P0 is system-only
except WriteDenied:
    pass
print(db.audit_log(denied_only=True))                     # the denial is on record
```

MCP (any client — opencode, Claude Desktop, Cursor):

```json
{
  "mcp": {
    "p-layers": {
      "type": "local",
      "command": ["python3", "-m", "p-layers", "serve"],
      "env": { "P_LAYER_DB": "~/.p_layer/memory.db" }
    }
  }
}
```

## Architecture

```
                ┌─────────────────────────────────────────────┐
  rules (P0-P1) │  knowledge (P2-P6)   episodes    entities/   │
  precedence-   │  FTS5 + embeddings   append-only  relations  │
  ordered       │  + confidence/TTL    (sessions,   (typed,    │
                │  + superseded_by     incidents)   validated) │
                └─────────────────────────────────────────────┘
                    SQLite (WAL, FK on) — one schema, migrations v1→v3
                                    │
        ┌───────────────┬───────────┼──────────────┬────────────┐
   recall (hybrid)  assemble (budget)  audit_log  contradictions  compile_wiki
   RRF+conf+fresh   rules→recent       every write   heuristic     P5 wiki
```

Tables: `knowledge` · `knowledge_fts` · `embeddings` (versioned) · `episodes` · `entities` · `relations` (constraint-validated) · `rules` · `snapshots` · `audit_log` · `schema_migrations`.

## Governance model

| Layer | Purpose | Who may write |
|---|---|---|
| P0 | Immutable rules | system only |
| P1 | Identity & persona | system only |
| P2 | Raw session archive | system, gateway, cron |
| P3 | Tool integrations | system, gateway, cron |
| P4 | Skills & growth | system, cron, agent, manual |
| P5 | Compiled knowledge | system, cron, agent, manual, tool |
| P6 | Incidents & RCA | system, cron, agent, manual, tool |

Precedence is data, not prose: lower priority/higher authority wins, and `assemble()` emits rules in precedence order under a token budget.

## Development

```bash
python3 -m unittest discover -s tests -v   # 133 tests (22 PG tests skip without a DSN)
```

## Examples

- `examples/quickstart.py` — API walkthrough
- `examples/demo_import_eval.sh` — the full migration story: drewgent fixture (knowledge + sessions + ontology + vault files) → import → vault ingest → eval before/after governance → audit → graph → contradictions → wiki
- `examples/suite.example.json` — eval suite format
- `examples/opencode-p-layer.jsonc` — ready-to-paste MCP config that replaces drewgent's remember/recall tooling

## Replace drewgent's memory with p-layers

The vault (identity, persona, skills as files) stays as files — it is a different storage class and should not be a database. p-layers replaces the *knowledge layer*:

```bash
# 1. migrate the data (knowledge + entities + relations + sessions)
python3 -m p_layer import-drewgent ~/.drewgent/.opencode/knowledge.db --embed ollama

# 2. ingest what the vault holds that belongs in the store (optional)
python3 -m p_layer import-rules ~/.drewgent/@identity/brain/rules.md
python3 -m p_layer import-incidents ~/.drewgent/P6-prefrontal/incidents

# 3. point the agent at the MCP server (examples/opencode-p-layer.jsonc),
#    and update AGENTS.md so it uses the p-layers tools
```

Then `p-layer eval suite.json` proves the swap: same data, recall@k 0.667 → 1.000 with governance metadata, ACL 30/30.

## PostgreSQL backend (multi-agent / SMB phase)

`PgStore` mirrors the SQLite `Store` interface — same methods, same governance, verified by a shared parity suite that runs every behavioral assertion against both backends.

```python
from p_layer.pgstore import PgStore

db = PgStore("dbname=memory host=localhost user=me")   # or P_LAYER_PG_DSN
db.add_knowledge("switched to portone v2", type="decision", layer="P5")
print(db.recall("portone"))
```

- **FTS**: `to_tsvector('simple')` + ts_rank, complemented by a pg_trgm ILIKE search (CJK-friendly).
- **Semantic**: pgvector (`vector(768)`), optional — without it the store is FTS-only and reports `semantic_available: false`.
- **Safety**: `statement_timeout` + `connect_timeout` — lock waits become clean errors, never hangs.
- **Ops boundary**: single-writer maintenance jobs (`reembed`, `consolidate`, `compile-wiki`) run on the SQLite store; on Pg they raise `NotImplementedError` loudly instead of silently degrading.
- **CI**: a postgres service container runs the full suite against a real database in CI.

## Credits


Built as a production-grade rebuild of ideas from:

- [opencode-drewgent](https://github.com/humanerd-drew/opencode-drewgent) — P0-P6 vault concept, provenance convention
- [p-layer](https://github.com/humanerd-drew/p-layer) — layer authority/ACL design, supersede-not-delete, confidence/TTL ranking, snapshots
- [Gajae-Code](https://github.com/Yeachan-Heo/gajae-code) — agent orchestration conventions

The critique that motivated this repo is documented in the README above; the credits are where the good ideas came from.

## License

MIT
