Metadata-Version: 2.4
Name: poma-memory
Version: 0.6.0
Summary: Persistent context for AI coding agents. No cloud. Just markdown.
Author-email: "POMA AI GmbH, Berlin" <sdk@poma-ai.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/poma-ai/poma-memory
Project-URL: Documentation, https://github.com/poma-ai/poma-memory#readme
Keywords: poma,memory,rag,agentic,markdown,claude-code,mcp
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: poma-primecut-nano<1,>=0.1.0
Requires-Dist: tiktoken<1,>=0.11
Requires-Dist: bm25s<1,>=0.3
Requires-Dist: numpy<3,>=2.2
Provides-Extra: semantic
Requires-Dist: model2vec<1,>=0.7; extra == "semantic"
Provides-Extra: openai
Requires-Dist: openai<2,>=1.109; extra == "openai"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.26; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Dynamic: license-file

# poma-memory

**Your AI agent loses context every session. poma-memory gives it back.**

AI coding agents (Claude Code, Cursor, Copilot) accumulate valuable project knowledge — decisions, architecture notes, task history — in markdown files (`.claude/`, `.cursor/`, `.github/copilot/`). But when context windows fill up or sessions restart, that knowledge becomes invisible. Grep finds strings; it doesn't understand structure.

poma-memory indexes those markdown files and returns **complete, readable context** — not disconnected snippets. When you search for "auth middleware", you get the matching paragraph _plus_ its parent headings and surrounding context, assembled into a coherent cheatsheet with `[...]` gap markers. The result reads like a compressed version of the original document.

**No POMA account or API key required.** Free, local, open-source.

---

## The Problem

Standard search (grep, embeddings over flat chunks) breaks document structure:

- **Orphaned content** — a paragraph arrives without its heading, so the agent doesn't know what section it belongs to
- **Lost hierarchy** — a nested list item loses its parent context
- **Fragment soup** — five hits from the same file come back as five disconnected blocks instead of one readable summary

For agent memory files — which are deeply hierarchical by design — this means the agent retrieves *text* but not *understanding*.

## The Solution

poma-memory preserves the full document hierarchy during chunking. Every retrieval unit is a root-to-leaf path through the heading tree, so results always carry complete context. Multiple hits from the same file are merged into a single cheatsheet.

This is an open-source extraction of [POMA](https://poma-ai.com)'s heuristic chunking engine, optimized for the clean, consistent markdown that AI agents produce.

---

## Install

```bash
pip install poma-memory                         # BM25 keyword search (always works)
pip install 'poma-memory[semantic]'               # + model2vec local embeddings (30MB, no API key)
pip install 'poma-memory[openai]'                 # + OpenAI text-embedding-3-large
pip install 'poma-memory[mcp]'                    # + MCP server for Claude Code
pip install 'poma-memory[semantic,mcp]'           # recommended combo
```

## Quick start

```bash
poma-memory index .claude/                                    # index your context files
poma-memory search "authentication middleware" --path .claude/ # search
poma-memory search "rollback" --path .claude/ --where kind=runbook  # search one kind
```

## MCP server (Claude Code)

Add poma-memory as an MCP server so Claude Code can search your project memory automatically:

```bash
claude mcp add --transport stdio --scope user poma-memory -- poma-memory-mcp
# Exposes poma_search, poma_index, poma_forget, poma_status tools
```

Once added, Claude Code can call `poma_search` during planning and exploration to surface relevant decisions, patterns, and context from prior sessions.

## Python API

```python
from poma_memory import index, search, status, forget

index(path=".claude/")
results = search("session context", path=".claude/", top_k=5)
for r in results:
    print(f"{r['file_path']} (score: {r['score']:.4f})")
    print(r['context'])

forget("old-project/", db_path="shared.db")   # drop a directory's rows
```

### Serving one index from a long-lived process

`search()` opens the database and rebuilds the BM25 corpus and the embedding
matrix on every call. A service that answers many queries holds a `MemoryIndex`
instead: one warm index, rebuilt when another connection commits or the file is
replaced.

```python
from poma_memory import MemoryIndex, index_file

idx = MemoryIndex("/data/live/.poma-memory.db")
idx.search("rollback", top_k=5, where={"repo": "acme/billing"})

with idx.lock:                                    # re-entrant; every method takes it
    index_file("/data/live/r/acme/billing/decision/ab12.md", path="/data/live")
    idx.ensure_embeddings()                       # embeds only rows with no valid embedding
```

Hold `idx.lock` around a write and `ensure_embeddings()` so no search sees the
half-way state and the new rows are embedded under the lock instead of by
whichever search arrives next. After that, searches write nothing, as long as
every writer to this database runs in this process under `idx.lock`. A CLI, a
hook or a daemon writing from another process changes the database under the
handle, and the next search then rebuilds (and embeds) while holding the lock.
If the embedder fails during `ensure_embeddings` it raises, instead of leaving
the index BM25-only. A base install without `model2vec` is BM25-only by design
and does not raise.

---

## Filtering by metadata

One index can hold more than one kind of document. To search a subset, give
each file some metadata and pass a predicate.

Metadata comes from two places. **Path rules** name what your layout already
says, and need no change to the files themselves — put a
`.poma-metadata.json` at the root of the indexed directory:

```json
{
  "rules": [
    {"glob": "events/**/*.md", "metadata": {"kind": "event"}},
    {"glob": "DECISIONS.md",   "metadata": {"kind": "decision"}},
    {"glob": "**/*.md",        "metadata": {"kind": "note"}}
  ]
}
```

First matching rule wins, so put the catch-all last. **Front-matter** is the
second source, and wins per key over a path rule:

```markdown
---
kind: event
session: laptop-2
---
```

Front-matter is read by a **restricted** parser, not a full YAML one: `key:
value`, `key: [a, b]`, a block list, and one level of nesting flattened to
`parent.child`. Values stay strings, so `true` and `1` are the strings `"true"`
and `"1"`. A trailing `# comment` outside quotes is dropped.

Anything else — block scalars (`|`, `>`), anchors, flow mappings, duplicate
keys, an unterminated fence — makes the whole block **unparsed**: the file gets
no front-matter metadata, and `poma-memory status` names it. If a filter misses
a file you expected, check `status` first.

Then filter:

```bash
poma-memory search "disk pressure" --path .agent/ --where kind=event
poma-memory search "disk pressure" --path .agent/ --where kind=event --where kind=decision
```

```python
search("disk pressure", path=".agent/", where={"kind": ["event", "decision"]})
```

Keys are AND-ed, values within a list are OR-ed, and comparison is
case-sensitive string equality. There are no operators and no negation: the
predicate cannot express "kind is not X" or "key is absent", so emit the key
on every document and filter positively.

Filtering narrows the corpus **before** anything is ranked. That matters
because the relevance gate reads the top semantic score, and a gate that saw
documents you filtered out would answer for a corpus you did not ask about.

Run `poma-memory index` after adding metadata or editing the rules file — it
re-reads metadata in place, with no re-chunking and no re-embedding. Until
then, a filtered search **refuses** rather than returning an empty list you
could not tell apart from "nothing matches". That covers both ways the index
can be behind: a file never scanned for metadata, and a file still carrying
what an earlier version of the rules file said about it. `poma-memory status`
shows both.

Each row records which directory's rules produced it, so this holds however the
database is addressed — including `--db` pointing somewhere else, and two
directories sharing one database. Directories sharing a database must be
**disjoint**: index `/proj/.agent` and `/proj/.agent/events` into one and each
file belongs to whichever run touched it last, silently, with `status`
reporting the index complete. A run given a narrower `--glob` reaches only part
of the corpus and leaves the rest on the old rules; re-run over each directory
to clear it.

`index` also **removes** documents that are gone from disk, so a deleted or
renamed file stops appearing in results. It only ever does this for files under
the directory it was given — a run over one directory never removes another's
rows, even when they share a database and even with `--prune`.

It holds back rather than removing when the directory it was given is itself
absent, when a whole directory under it has disappeared or been left completely
empty (more than five of its indexed files gone, nothing else in it), or when
the removals would be most of the index, since all of these look more like
something that failed to mount than a deletion. Held-back documents stay in the index and keep
appearing in results until you decide: `--prune` removes them, `--no-prune`
never removes anything. A handful of individual files going missing is removed
without asking, and so is deleting some of a directory's files while others
remain. The price of the empty-directory rule: deliberately deleting every file
of a larger directory needs `--prune` once. A change is detected by mtime, size or ctime, so an edit
restored from a backup with its timestamp intact is still picked up.

`--prune` overrides both hold-backs — the proportional one and a vanished or
emptied subdirectory — but never the absent-root gate: a run whose own directory is
absent removes nothing whatever the flag says, because at that moment nothing
can tell an unmounted drive from a deleted one. (It also creates nothing, so
the directory's absence stays true for the next run.)

For a directory that is gone for good, **`poma-memory forget <dir> --db <db>`**
drops its rows — it does not need the directory to exist, and it reaches
nothing outside it. That is the way out of two states nothing else can clear:
a directory deleted while it shared a database with another, and a directory
that was renamed. In both, the old rows hold metadata resolved against rules
nobody can re-read, so every filtered search refuses — including searches that
have nothing to do with the directory that moved. Name `--db` explicitly: the
default database lives *inside* the directory that is missing. A refusal caused
by this prints the exact command.

Independently of rules, a filtered search also refuses when a file it would
return no longer exists on disk (`MetadataGhosts`) — the only refusal a
directory with no `.poma-metadata.json` can raise, since it has no rules to go
stale. A search whose answer cannot include a vanished file is unaffected. The
vanished directory may only be unmounted: remount it rather than running the
`forget` the message names.

Why the filter runs before ranking rather than after, and what the refusals are
protecting against: **[`docs/metadata-filtering.md`](docs/metadata-filtering.md)**.

---

## How it works

1. **Hierarchical chunking.** Markdown is parsed into depth-annotated chunks that preserve heading hierarchy, lists, code blocks, and tables.
2. **Chunkset assembly.** Leaf chunks are paired with their ancestors into self-contained retrieval units (root-to-leaf paths), so every result carries full context.
3. **Hybrid search.** BM25 keyword matching (always available) + optional semantic vectors, merged via Reciprocal Rank Fusion.
4. **Empty gate.** RRF scores are rank-based — something always tops the list, so a fused-score floor can't express "the corpus has no answer." The gate can: when even the *best* semantic hit's cosine similarity is below a per-embedder calibrated threshold (model2vec: 0.35 — calibrated on a real corpus where irrelevant queries topped out at 0.25 and answerable ones started at 0.44), *all* results are suppressed instead of returning the best of a bad lot. Override per call (`--empty-gate`, `empty_gate=`), via `POMA_MEMORY_EMPTY_GATE`, or disable with `0`. BM25-only mode has no cosine and is ungated.
5. **Cheatsheet merging.** Multiple hits from the same file are merged into one block with `[...]` gap markers — reads like a summary, not a list of excerpts.
6. **Incremental indexing.** Append-only files (like agent context logs) only process new content on re-index.

### Search backends

| Backend | Install | Requires | Best for |
|---------|---------|----------|----------|
| BM25 | included | nothing | exact term matching |
| model2vec | `[semantic]` | 30MB local model | general-purpose, no API key |
| OpenAI | `[openai]` | `OPENAI_API_KEY` | highest quality |

Multiple backends are automatically fused via Reciprocal Rank Fusion when available.

---

## What this is (and isn't)

poma-memory extracts the **heuristic chunking and retrieval** logic from POMA's document processing platform. It works well on clean, predictable markdown — exactly what agents produce.

It does **not** include POMA's ML-powered indentation analysis, fine-tuned embedding models, or cloud processing pipeline. For complex document processing (scanned PDFs, inconsistent formatting, enterprise scale), see [poma-ai.com](https://poma-ai.com).

## Built for

- AI coding agents that persist context in markdown (`.claude/`, `.cursor/`, `.github/copilot/`)
- [Megavibe](https://github.com/poma-ai/megavibe) multi-agent framework (ships with poma-memory integration)
- Claude Code hook pipelines (augment Grep results with semantic context)

## License

MIT
