Metadata-Version: 2.4
Name: lorekeep
Version: 0.13.0
Summary: Temporal knowledge graph for AI agents via MCP
Project-URL: Homepage, https://github.com/manhhailua/lorekeep
Project-URL: Repository, https://github.com/manhhailua/lorekeep
Project-URL: Security, https://github.com/manhhailua/lorekeep/security/policy
Author: Manh Pham
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: litellm>=1.40
Requires-Dist: mcp>=1.0
Requires-Dist: mistune>=3.0
Requires-Dist: networkx>=3.2
Requires-Dist: platformdirs>=4.0
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Description-Content-Type: text/markdown

# Lorekeep

<p align="center"><img src="cover.jpeg" alt="Lorekeep" /></p>

**A second brain for code — a temporal knowledge graph that aggregates what you and your agents learn across devices, the services you build, and your team, over MCP. Agents read at query time and propose facts at runtime through journal-based write tools.**

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Lorekeep is a **second brain**: every coding agent you run (Claude Code, Cursor,
Codex, opencode) reads from and contributes to one knowledge graph; devices
rebuild the same graph from Git-synced raw docs, schema, and agent journals; the
software you build and operate feeds back in; and your team shares the parts
that matter. Under the hood it compiles raw docs + agent
contributions into a temporal knowledge graph (`facts.jsonl`) served over MCP.
Knowledge is processed once, not re-RAG'd per query.

---

## Why

A second brain for code has to pull from many sources — your coding agents, your
devices, the services you operate, your team — and stay coherent as it grows.
Most tools cover one slice:

| | file-based | temporal KG | compile step | team permission | agent + device + software sources | MCP |
|---|---|---|---|---|---|---|
| Obsidian + MCP | ✅ | ❌ | ❌ | ❌ | ❌ (manual) | ✅ |
| mcp-knowledge-graph | ✅ | ❌ | ❌ | ❌ (local) | ❌ | ✅ |
| mem0 / cognee | ❌ (DB) | partial | ❌ | partial (DB) | partial | ✅ |

No tool combines all six. Lorekeep does: **file-based + temporal graph +
compile-once + namespace permission + multi-source (agents/devices/software/team) + MCP**.

## Features

- **Append-and-resolve** — three write paths (raw/ compile, agent ingest,
  import sessions) converge into one resolve step. Journals are append-only;
  resolve is pure logic, zero LLM cost.
- **Agent-driven knowledge** — agents propose facts at runtime via MCP write
  tools at **zero marginal LLM cost**. Confidence-gated: high-confidence
  auto-merge, low-confidence quarantine.
- **File-sovereign** — raw docs + schema + append-only agent journals are the
  durable, Git-syncable sources. `facts.jsonl` is a deterministic derived store
  rebuilt on each device; no binary store is committed.
- **Temporal** — every fact carries `valid_from`/`valid_to` (half-open
  `[from, to)`); query "what was true at *T*", history, diffs.
- **Namespace permission** — facts are tagged `ns` from the directory tree
  (`raw/<ns>/`); agents scoped to namespaces; cross-namespace edges
  hidden unless both endpoints are visible. Deny-by-default.
- **MCP, stdio-first** — `lorekeep serve` exposes 9 read + 5 write tools;
  `lorekeep mcp add` wires Claude Code / Cursor / Codex / opencode.
- **Autonomous agent daemon** — `lorekeep agent watch` keeps the graph current:
  auto-compile on raw/ change, auto-resolve pending journals, delta import of
  agent session memory. **Auto-backup**: syncs to remote git after compile (pull --rebase + push).
  Runs in the background; MCP server lazy-reloads.
- **Session-end hooks** — `lorekeep hook` auto-imports agent memory when a
  session ends (Claude / Cursor / Codex / opencode). Wired by `mcp add`.
- **Obsidian + Tolaria wiki** — auto-generated after every compile/resolve: flat
  markdown pages with `[[wikilinks]]`, YAML frontmatter (incl. relationship
  fields), tags. The same `wiki/` folder opens in both Obsidian and Tolaria.
- **Subject-aware ontology** — work-context node types (person, role, skill,
  domain, goal, …) + cross-namespace edges (owns, contributes_to, skilled_in)
  weave your personal knowledge into the team graph. The `me` namespace is
  subject-centric (altitude rule: tokens → attributes, not nodes).
- **Lazy-reload** — graph updates visible on next query. Connect once, use forever.
- **Provider-pluggable extraction** — litellm (OpenAI / Anthropic /
  DashScope/Qwen / Ollama). Strict-privacy → Ollama, fully local.
- **Tier-1 eval** — extraction P/R/F1 vs a gold corpus, entity-resolution F1,
  graph-structure metrics, determinism property tests.
- **Tier-2 LoCoMo eval** — long-term conversations → graph → retrieval QA.
  F1 per category (single-hop, temporal, multi-hop, adversarial abstention).

## Install

```bash
# no install needed — uvx runs it directly:
uvx lorekeep init

# or install permanently:
uv tool install lorekeep
```

## Quickstart

```bash
# 1. bootstrap data home (config + schema + dirs + agent wiring)
uvx lorekeep init

# 2. add docs under the data home's raw/<namespace>/
mkdir -p ~/.local/share/lorekeep/raw/private
cp your-docs.md ~/.local/share/lorekeep/raw/private/

# 3. set a provider (edit ~/.config/lorekeep/config.yaml), then compile
uvx lorekeep compile                # raw/*.md → graph/facts.jsonl + wiki/

# 4. wire a coding agent (writes a portable .mcp.json)
uvx lorekeep mcp add --agent claude --ns private

# 5. verify
uvx lorekeep doctor
```

Restart Claude Code → 14 Lorekeep tools are available (9 read + 5 write), scoped to your namespace. Open `~/.local/share/lorekeep/wiki/` in Obsidian to browse the graph as a human — or just run `uvx lorekeep wiki --open` to generate + launch it. See [Browsing the wiki in Obsidian](docs/guides/wiki.md).

## Lifecycle

The full journey from install to continuous use — see the
[Getting started guide](docs/guides/getting-started.md) for details.

```
 INIT          CURATE              SERVE              KEEP CURRENT          SYNC
 ════          ═══════             ══════             ════════════          ════
 ┌─────┐   ┌──────────┐        ┌─────────┐       ┌───────────────┐    ┌────────┐
 │init │──►│raw/*.md  │──►     │mcp add  │──►    │ agent watch   │──► │backup  │
 │     │   │compile   │ compile│serve    │ serve │  raw/    → compile   │
 │     │   │          │────────│+hook    │       │  pending/ → resolve  │
 └─────┘   └──────────┘   wiki │         │       │  memory/  → import   └────────┘
                               └─────────┘       └───────┬───────┘         │
                                    ▲                     │ lazy-reload     │ git sync
                                    │◄────────────────────┘                 │
                                    │◄──────────────────────────────────────┘
```

| Step | Command | What it does |
|---|---|---|
| 1. Bootstrap | `lorekeep init` | Create data home (config + schema + dirs) |
| 2. Curate | `raw/<ns>/*.md` | Drop markdown docs under namespace dirs |
| 3. Compile | `lorekeep compile` | LLM-extract → `facts.jsonl` + `wiki/` (cached, deterministic) |
| 4. Wire agent | `lorekeep mcp add --agent claude --ns <ns>` | Write `.mcp.json` + session-end hook, scoped to namespace |
| 5. Verify | `lorekeep doctor` | Graph loads, schema valid, tool responds |
| 6. Serve | `lorekeep serve` | MCP server (9 read + 5 write tools, lazy-reload) |
| 7. Keep current | `lorekeep agent watch &` | Daemon: auto-compile, auto-resolve, delta-import sessions |
| 8. Back up | `lorekeep backup` | Push data home to private git repo (raw/ + schema.json) |
| 9. Persist daemon | `lorekeep agent daemon install` | Survive restart (systemd/launchd/startup) |

Personal knowledge + team sharing:

| Command | Purpose |
|---|---|
| `lorekeep profile [--open]` | Show / open your personal profile source (`raw/<ns>/about.md` + `profile.md`) — edit in Obsidian/Tolaria, then `compile`. |
| `lorekeep contribution` | Suggest team-knowledge gaps: nodes in your personal ns not yet shared with a team ns. |
| `lorekeep schema upgrade [--dry-run]` | Upgrade the stock ontology v2 schema to v3 with a backup; custom schemas require `--force`. |

Steps 1–6 are one-time setup. Step 7 runs in the background for continuous
updates. Step 8 syncs across machines.

## How it works

```
               THREE WRITE PATHS                            SYNC
               ════════════════
raw/<ns>/*.md ──► ingest ──► extract(LLM) ──┐
                                            │
agent propose ──► MCP write tools ──► ──────┤
  (ZERO LLM cost, journal append)           │
                                            ├──► resolve ──► writer ──► facts.jsonl ──► wiki/*.md
import ──► raw/ ──► compile ────────────────┘    (pure logic,    (sorted)       (Obsidian)
                                                   ZERO LLM)
                                                        ┌───────────────┘
                                                        ▼ (git / S3 sync)
               SERVE + QUERY (runtime, per device)
facts.jsonl ──load──► GraphStore ──► ScopedGraph(ns) ──► MCP (9 read + 5 write) ──► agent
                         ▲              ▲                      │
                          │              │         ◄── read queries
                          │              └────────── write proposals (journal)
                         └── lazy-reload on mtime change

               AUTONOMOUS AGENT DAEMON
               lorekeep agent watch:
                 ├── watch raw/ → auto-compile
                 ├── watch pending/ → auto-resolve
                 └── watch agent memory/ → delta import → raw/
```

**Three write paths → one resolve**: markdown is compiled by an LLM (chunked + cached); agents propose facts at runtime through MCP write tools at **zero marginal LLM cost** (the agent already ran the LLM for the conversation); agent sessions are imported into raw/. All converge at `resolve` — pure Python logic that merges, deduplicates, validates, and writes byte-stable `facts.jsonl`.

**Serve**: `GraphStore` loads `facts.jsonl` into a networkx graph with temporal
queries. `ScopedGraph` is the single permission chokepoint — every query is
filtered through strict visibility rules. The FastMCP server exposes 9 read
+ 5 write tools over `ScopedGraph`. It lazy-reloads when
`facts.jsonl` changes, so `compile` is instantly visible without reconnecting.

## Concepts

**fact** — one line of `facts.jsonl`, a `node` or `edge`:
```jsonl
{"kind":"node","id":"svc:payments","type":"service","ns":["backend"],"valid_from":"2024-01-15","valid_to":null,"props":{"lang":"go"},"src":["raw/backend/payments.md:12"]}
{"kind":"edge","id":"e_depends_on_0001","type":"depends_on","from":"svc:payments","to":"svc:auth","ns":["backend"],"valid_from":"2024-01-15","valid_to":"2025-03-01","props":{},"src":["...:20"]}
```
- `ns` — namespace set; `["public"]` is globally visible.
- `valid_to: null` ⇒ current. History = multiple edges, same endpoints, different windows.
- `src` — provenance to raw doc line (audit, incremental re-compile, agent citations).

**Permission** — effective_ns = allowed ∪ {public}. Node visible iff
`ns ∩ effective_ns ≠ ∅`. Edge visible iff **both** endpoints visible **and**
`edge.ns ∩ effective_ns ≠ ∅`. Deny-by-default; an edge never reveals a
neighbor the caller can't see.

**Temporal queries** — `at_time(T)` (snapshot of facts valid at T, half-open
`[from,to)`), `history(id)` (versions of an entity), `changes(t1,t2)` (edges
that began/ended in the window).

**Agent-driven knowledge** — agents propose facts at runtime through MCP write tools (zero LLM cost). Facts land in `pending/<ns>/journal.jsonl` with agent id, confidence score, and timestamp. Resolve merges them into the graph: high-confidence (≥0.8) auto-merge, medium (0.5-0.8) merge + flag, low (<0.5) quarantine.

**Autonomous agent daemon** — `lorekeep agent watch` keeps the graph current: watches `raw/` for changes → auto-compile; monitors `pending/` → auto-resolve; delta-imports agent session memory (Claude / Cursor / Codex / opencode) into `raw/`. Session-end hooks auto-trigger `lorekeep hook` when the agent exits. Scheduled lint and weekly suggestions are planned. See [docs/architecture/agent.md](docs/architecture/agent.md).

## MCP tools (9 read + 5 write, scoped)

**Read:** `search` · `get_node` · `neighbors` · `at_time` · `history` · `changes` · `list_namespaces` · `schema` · `meta`.

**Write** (journal-based, zero LLM cost): `propose_fact` · `link_facts` · `flag_contradiction` · `update_fact` · `suggest_improvement`.

Every result is filtered to the caller's namespace. Write tools append to `pending/` journals; facts enter the graph on the next resolve pass.

## Configuration

`config.yaml` (resolved by precedence: explicit `LOREKEEP_*` env > `LOREKEEP_HOME` >
dev marker > XDG):
```yaml
provider:
  model: dashscope/qwen-plus                           # {provider}/{model} — litellm routes by prefix
  api_key_env: DASHSCOPE_API_KEY                       # env var name (preferred)
  api_key: null                                        # or inline (gitignored config only)
ns:
  default: [me]                                      # serve-time default scope
  personal: me                                       # subject-centric extraction
install_source: pypi                                   # pypi = portable .mcp.json
```
Native providers (`openai`, `anthropic`, `deepseek`, `dashscope`, `gemini`, …)
need **no** `api_base` — litellm knows their endpoint. Set `api_base` only for a
custom OpenAI-compatible endpoint (vllm, lm_studio, a proxy/gateway, or Ollama on
a non-default host).
API keys never live in committed files — use `api_key_env` (env) or inline
`api_key` in the gitignored config only. Examples (DashScope / OpenAI / Ollama)
in [`.lorekeep/config.yaml.example`](.lorekeep/config.yaml.example).

## Data home & dev mode

Path resolution (high → low): explicit `LOREKEEP_*` env → `LOREKEEP_HOME` →
**dev mode** (`.lorekeep/` in CWD; auto-detected in a source checkout)
→ XDG (`~/.config/lorekeep`, `~/.local/share/lorekeep`).

Back up the data home to a private git repo with `lorekeep backup` — see
[`docs/guides/backup.md`](docs/guides/backup.md).

Full details, per-path overrides, and `lorekeep init`: [`docs/guides/data-home.md`](docs/guides/data-home.md).
For usage, see the [`docs/`](docs/README.md) index.

## Evaluation

**Tier-1** (CI): extraction P/R/F1 vs a gold corpus, entity-resolution pairwise F1,
graph-structure metrics, determinism. *Dev-only command: `lorekeep eval`.*

**Tier-2 LoCoMo** (per-release): long-term conversation → graph → retrieval QA.
*Dev-only command: `lorekeep eval-locomo`.*

| Category | Recall | Description |
|---|---|---|
| Temporal | 0.71 | "When did X happen?" |
| Single-hop | 0.58 | "What is X's Y?" |
| Multi-hop | 0.49 | "Who did X's sister work with?" |
| Descriptive | 0.73 | "What did X do?" |
| Adversarial | 0.85 | Abstention — plausible-but-wrong answer not found |
| **Overall** | **0.71** | 199 QA, 1 conversation, DeepSeek extraction |

*Graph-guided retrieval: keyword search → get_node → neighbors(depth=1-2) →
source markdown enrichment. No agent-LLM synthesis (programmatic baseline).*

The north star is *systematic thinking with complete information* — memory-recall
benchmarks (LoCoMo, LongMemEval) are parity checks, not the optimization target.
See [`docs/architecture/evaluation.md`](docs/architecture/evaluation.md).

## Configuration

After `init`, adjust settings without editing YAML:

```bash
lorekeep config show                                    # print current config
lorekeep config set provider.model deepseek/deepseek-chat
lorekeep config set provider.api_key_env DEEPSEEK_API_KEY
lorekeep config set ns.default backend,frontend
lorekeep config set observability.provider langfuse     # optional tracing
```

Observability (optional): set `observability.provider` to `langfuse` or
`langsmith` for LLM call tracing via litellm callbacks.

## Backup & sync

Back up `raw/` + `schema.json` to a private git repo. API keys, graph, cache,
and wiki are never committed — they're regenerated per machine.

```bash
# First time — create a private repo on GitHub, then:
lorekeep backup --init https://github.com/<you>/lorekeep-data.git

# Routine — after editing raw/ or compiling:
lorekeep backup

# Restore on a new machine:
lorekeep init                                        # bootstrap config
rm -rf .lorekeep && git clone https://github.com/<you>/lorekeep-data.git .lorekeep
lorekeep compile                                     # rebuild graph from raw/

# Multi-device sync — if push is rejected (non-fast-forward):
cd .lorekeep && git pull --rebase && cd - && lorekeep backup
```

Each machine sets its own API key — no secrets in the backup.

## Project layout

```
src/lorekeep/
  models.py            shared contract (Node/Edge/Schema/Manifest)
  facts_io.py          facts.jsonl loader (store + eval)
  paths.py             4-tier path resolution (env/home/dev/XDG)
  providers.py         litellm provider/model enumeration for init
  defaults.py          default schema + config (for `init`)
  config.py, schema_io.py
  compile/{ingest,extract,resolve,writer}.py    the compile pipeline
  compile/providers.py                          LiteLLMProvider + observability
  journal.py           append-only journal writer + loader
  agent.py             autonomous agent: ingest, lint, suggest, status, watch
  store/{graph,fts}.py                          GraphStore + FTS5 cache
  perm/ns.py                                    ScopedGraph permission chokepoint
  mcp_server.py                                 FastMCP + 9 read + 5 write tools
  wiki.py                                        Obsidian-compatible wiki generator
  importer/{claude,cursor,codex,opencode}.py    agent session → raw/ importers
  integrations/{claude_code,cursor,codex,opencode,common}.py
  pipeline.py, cli.py
  eval/{gold,construction,retrieval}.py
  eval/locomo.py                                Tier-2 LoCoMo eval
tests/                 ~410 tests
docs/                  README.md index, architecture/, guides/
```

## Status

**v1 (implemented)** — the second-brain foundation: compile pipeline + serve (store/permission/MCP 9 read+5 write/4-agent integrations) + import (Claude/Cursor/Codex/opencode) + session-end hooks + agent daemon (watch/ingest/lint/suggest/status) + journal + resolve + data-home + dev mode + lazy-reload + backup + eval + scope awareness (`meta` tool) + **subject-aware ontology v2** (personal `me` ns + team ns + cross-ns edges) + **Obsidian/Tolaria wiki** + profile/contribution commands. Published to PyPI as `lorekeep`.

**Next** — the second-brain direction is multi-faceted: multi-agent concurrency, multi-device sync, software-source connectors, a proactive agent, a team server, better retrieval. See the full **[Roadmap](docs/ROADMAP.md)**.

## Documentation

The [`docs/`](docs/README.md) index is the entry point.

**Guides**
- [Getting started](docs/guides/getting-started.md)
- [Importing agent sessions](docs/guides/import.md)
- [Compiling the knowledge graph](docs/guides/compile.md)
- [Serving the graph to coding agents](docs/guides/serve.md)
- [Browsing the wiki](docs/guides/wiki.md)
- [Data home & path resolution](docs/guides/data-home.md)
- [Backing up the data home](docs/guides/backup.md)

**Architecture**
- [Overview](docs/architecture/overview.md) · [Data model](docs/architecture/data-model.md) · [Pipeline](docs/architecture/pipeline.md) · [Journal](docs/architecture/journal.md) · [Agent](docs/architecture/agent.md) · [Permission](docs/architecture/permission.md) · [Temporal](docs/architecture/temporal.md) · [Serve & MCP](docs/architecture/serve-mcp.md) · [Testing & evaluation](docs/architecture/evaluation.md)

## License

Lorekeep is released under the **MIT License** — see [`LICENSE`](LICENSE).

Copyright © 2026 Manh Pham. You're free to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the software, provided the
copyright and permission notice are included in all copies. The software is
provided "as is", without warranty of any kind.
