Metadata-Version: 2.4
Name: mnemo-engine
Version: 0.13.2
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Summary: Encrypted, portable agent-memory engine — Python bindings
Home-Page: https://toarchkumar.github.io/mnemo/
License: Apache-2.0
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/toarchkumar/mnemo
Project-URL: Repository, https://github.com/toarchkumar/mnemo

# Memory Nemo (MNemo) — Python bindings

Repository overview: [root README](../README.md) ·
landing page: [index.html](../index.html).

Python bindings for **Memory Nemo (MNemo)**, the encrypted single-file
agent-memory engine. The package is a thin [PyO3](https://pyo3.rs) wrapper
over the Rust core in the
sibling `mnemo/` crate — the storage engine, AES-256-GCM encryption, the
write-ahead log, the IVF+PQ index, snapshots, and the agent-memory model all
run as compiled Rust; Python only sees a small, ergonomic surface.

> Distribution name on PyPI: `mnemo-engine` (both `mnemo` and
> `mnemo-db` were already taken by unrelated projects). The import
> name is **`mnemo_engine`** starting in v0.7.0, so the full setup is
> `pip install mnemo-engine` then `import mnemo_engine`. The
> pre-v0.7.0 `import mnemo` keeps working for the v0.7.x series via a
> pure-Python shim that emits one `DeprecationWarning` on import; the
> shim is removed in v0.8.0. Code that wants to keep its call sites
> unchanged can `import mnemo_engine as mnemo` — same objects, no
> warning. See the "Breaking: import name" section in the CHANGELOG
> for the full migration.

## For AI agents

An agent that's been handed a `.mnemo` file and its passphrase can become
productive in two calls — no external documentation required:

```python
import mnemo_engine as mnemo, os

db = mnemo.open("agent.mnemo", os.environ["MNEMO_PASSPHRASE"])

# The file introduces itself: returns memories tagged metadata.area="onboarding",
# manifest first. Each entry tells you the embedder, agent_id convention,
# project metadata, and any other context the file's author recorded.
for entry in db.about():
    print(entry["content"])
```

Creating a new database? It's self-describing from creation:

```python
db = mnemo.open("new.mnemo", "passphrase", dimensions=384)
db.insert_default_manifest()    # same scaffold that `mnemo init` adds
db.flush()
```

The scaffold tells the next agent what to do: replace it with one that
records your real embedder and conventions. See the [main README](../mnemo/README.md#self-describing-databases)
for the full pattern.

## Install

The published wheel is the normal install path — nothing to build:

```bash
pip install mnemo-engine
```

Prebuilt wheels ship for CPython 3.8+ on macOS x86_64 + arm64, Linux
x86_64 + aarch64 (manylinux), and Windows x86_64. If none fits (e.g. a
BSD, aarch64 musl, or any other platform mnemo doesn't ship a wheel
for), pip falls back to the sdist and compiles from source — that path
needs a working Rust toolchain (`rustup`) and will take a few minutes.

> ARM users building from source (sdist, or `cargo install mnemo-engine`
> directly): set `RUSTFLAGS="--cfg aes_armv8 --cfg polyval_armv8"` before
> install to enable hardware AES. The prebuilt wheels have this baked
> in; your sdist build doesn't see our repo's `.cargo/config.toml`.
> Verify afterwards with `mnemo.build_info()['aes_backend']`, which
> should read `"armv8 hardware"` (not `"armv8 software"`).

### From source (contributors)

```bash
pip install maturin
cd mnemo-python
maturin build --release          # produces a wheel in target/wheels/
pip install target/wheels/mnemo-*.whl
```

`maturin develop` installs straight into the active virtualenv during
development. The extension is built against the stable ABI (`abi3-py38`), so a
single wheel works on CPython 3.8 and newer.

## Usage

```python
import mnemo_engine as mnemo

# Open an existing database, or create one (dimensions required to create).
db = mnemo.open("agent.mnemo", "passphrase", dimensions=4)

# Store typed memories. memory_type is one of:
# "episodic", "semantic", "procedural", "working".
db.remember(
    "the user prefers concise answers",
    "procedural",
    [0.1, 0.2, 0.3, 0.4],
    importance=0.8,
    agent_id="assistant",
    metadata={"source": "onboarding"},
)

# Multi-signal recall — similarity blended with recency, importance, frequency.
for hit in db.recall([0.1, 0.2, 0.3, 0.4], top_k=5):
    print(hit["score"], hit["content"])

db.flush()
db.close()
```

`mnemo.open` returns a `Mnemo` object that is also a context manager —
`with mnemo.open(...) as db:` flushes automatically on exit.

### Sessions

A `Session` wraps the database for one conversation: it records each turn as
`working` memory and, when closed, consolidates those turns into durable
`episodic` memory.

```python
db = mnemo.open("agent.mnemo", "passphrase", dimensions=4)

with db.session("assistant") as chat:
    chat.add_turn(mnemo.Turn.user("my flight is Friday", [1.0, 0.0, 0.0, 0.0]))
    chat.add_turn(mnemo.Turn.assistant("noted", [0.9, 0.1, 0.0, 0.0]))
    context = chat.recall([1.0, 0.0, 0.0, 0.0], top_k=5)
    # leaving the block consolidates the turns into episodic memory

# or, explicitly:
chat = db.session("assistant")
chat.add_turn(mnemo.Turn("system", "be concise", [0.0, 0.0, 0.0, 1.0]))
chat.close()      # consolidate working -> episodic
# chat.discard()  # alternative: throw the turns away
```

`mnemo.Turn` has `Turn.user(...)`, `Turn.assistant(...)`, `Turn.system(...)`,
and `Turn(role, content, vector)`. A `Session`'s `recall` is always scoped to
its own agent.

## API

`mnemo.open(path, passphrase, dimensions=None) -> Mnemo`

`Mnemo` methods:

| Method | Purpose |
|---|---|
| `remember(content, memory_type, vector, *, agent_id, importance, session_id, ttl_secs, shared, metadata)` | Store a memory; returns its id |
| `recall(query, top_k=10, memory_types=None, agent_id=None, track_access=True, n_probe=None, n_rerank=None)` | Multi-signal ranked retrieval. `track_access=False` skips access-stat updates (fully read-only recall). `n_probe` / `n_rerank` (PR ANN-1): per-query ANN tuning — `None` uses the index's build-time defaults (ANN-1 auto-formulas: `(partitions+1)/2` and `500`) |
| `search(query, top_k=10)` | Exact nearest-neighbour search |
| `get(id)` / `delete(id)` | Fetch / soft-delete by id |
| `about()` | Self-describing onboarding briefing — memories tagged `metadata.area="onboarding"`, manifest first |
| `about_essence()` | The project essence render for the briefing (markdown), or `None` when the file has none |
| `insert_default_manifest()` | Insert the canonical scaffold manifest (same one `mnemo init` adds); returns its id |
| `session(agent_id)` | Begin a conversation `Session` |
| `flush()` / `close()` | Persist pending changes |
| `verify()` | Decrypt and re-validate every record |
| `build_index()` / `drop_index()` / `has_index()` | Approximate index control |
| `snapshots()` / `restore_to(txn_id)` / `restore_to_time(unix_secs)` | Point-in-time recovery |
| `set_cache_capacity(pages)` / `page_cache_stats()` | Page-cache tuning (renamed from `cache_stats` in v0.4.0 — that name now belongs to the result cache below) |
| `cache_put(namespace, key, value, content_type="text", ttl_secs=None)` | Exact-key result cache put (Phase 10.1). `value` accepts `str` or `bytes` |
| `cache_get(namespace, key)` | Exact-key cache get — returns `dict` on hit (`value` is `bytes`) or `None` |
| `cache_delete(namespace, key)` / `cache_purge(namespace=None, expired_only=False)` | Tombstone by key or in bulk |
| `cache_stats(namespace=None)` | Result-cache stats: `{entries, bytes, hits, misses, hit_rate, evictions}` |
| `cache_put_semantic(namespace, key, vector, value, model, content_type="text", ttl_secs=None)` | Semantic cache put (Phase 10.2) — vector must match db dimensions |
| `cache_get_semantic(namespace, query, model, threshold=0.97)` | Top-1 cosine over the namespace's vectored entries whose `model` matches |
| `set_cache_flush_policy(policy, max_dirty=None, max_age_secs=None)` | `policy="strict"` (default; every put durable on return) or `"batched"` (engine auto-flushes at `max_dirty` puts or `max_age_secs` seconds). Under `batched`, `cache_put` and friends no longer force an eager flush — call `db.flush()` yourself if you need it before a specific boundary |
| `set_cache_budget(namespace, max_entries, max_bytes)` | Per-namespace LRU cap (defaults 10 000 entries / 64 MiB). On the next `cache_put` the LRU is evicted until both limits are satisfied |
| `set_max_snapshots(max)` | Override the snapshot-manifest retention cap (default 256; `0` disables) |
| `set_vector_cache_budget(bytes)` | v9 files: keep the fp16 vector column in memory while it takes at most `bytes` (default 96 MiB, about 125K vectors at 384 dims); above it recall reads vectors through the page cache (less memory, slower recall). This handle only |
| `set_turn_compress_after(secs)` | v9 files: compress thread turns once they're older than `secs` (default 7 days; `0` compresses at write). This handle only |
| `memories()` | Enumerate every live (non-deleted, non-expired) memory as a list of dicts |
| `dimensions()` | Embedding dimensionality this database was created with (also on `stats()["dimensions"]`) |
| `rekey(new_passphrase, fast=False)` | Change the passphrase. Re-derives the KEK and re-wraps the DEK; encrypted pages are not rewritten (O(1) in db size). `fast=True` uses the cheap-Argon2 test params — omit for production. If it raises, the old passphrase is still in force (the one exception is an error that says the file may open with either). If it returns but couldn't scrub every copy of the old key wrap, it emits a `RuntimeWarning`: the call succeeded and the new passphrase is in force (where warnings are errors, it goes to `sys.unraisablehook` and the call still returns) |
| `set_embedder(fn, *, name, dimensions, normalize=True)` | Register a Python callable `embed(texts: list[str]) -> list[list[float]]` as this handle's text embedder (Phase 3). Dimension-checks against the database's width; identity-checks against (and on first use upserts) the scaffold manifest's `embedder_name` / `embedder_dimensions` / `embedder_normalize` keys, so swapping embedders between opens fails loudly |
| `remember_text(content, memory_type, **kwargs)` | Embed `content` via the registered embedder and `remember` the result. Accepts the same per-memory kwargs as `remember` (`agent_id`, `importance`, `session_id`, `ttl_secs`, `shared`, `metadata`) |
| `recall_text(query, top_k=10, memory_types=None, agent_id=None, track_access=True, n_probe=None, n_rerank=None, filter=None, max_bytes=None, keywords=None, keyword_weight=None)` | Embed `query` via the registered embedder and run `recall`. Same return shape and kwargs as `recall`. `keywords` (v9 files): identifiers or exact terms to match (a version, a PR number, a function name), not the question; their BM25 score is added with weight `keyword_weight` (default 0.5) |
| `cache_get_semantic_text(namespace, text, threshold=0.97)` | Text-side semantic cache lookup; uses the embedder's `name` as the semantic cache's `model` component |
| `stats()` | Summary statistics — includes `fsyncs_last_flush` (count of `sync_all` the most recent `flush()` on this handle issued; `0` on a fresh handle before any flush — Phase 6 instrumentation) and `recall_mode` (`"exact_resident"`: the live facts' vectors fit the vector cache budget and recall scans them in memory; `"exact_paged"`: over the budget, or a v8 file, so recall reads them from the file; `"indexed"`: an IVF+PQ index shortlists). Since v0.13 also `snapshots_retained`, `oldest_restorable_at` (unix seconds, or `None`) and `snapshot_pinned_bytes`: what a restore can reach now, which under churn can be far less than the retention policy |
| `export_encrypted(dest)` | Copy the (already-encrypted) file elsewhere |
| `len(db)` | Live memory count |

Plus two module-level functions:

| Function | Purpose |
|---|---|
| `mnemo.compact_file(path, passphrase)` | Rewrite the file out-of-place, dropping tombstoned and expired memories and reclaiming stale pages. Caller's own handle must be closed first. Returns `{"before": int, "after": int}` |
| `mnemo.compact_file_with_key(path, key)` | The same for a raw-key file (`key`: 32 bytes, as from `generate_key()`), under the same key; converts a v8 file to v9 (since v0.13.1) |
| `mnemo.build_info()` | Returns `{"version", "target", "aes_backend", "polyval_backend"}` for the installed wheel. Lets you verify an ARM wheel picked up the hardware AES path — `aes_backend` should read `"armv8 hardware"` on Apple Silicon / Graviton (anything else, like `"armv8 software (constant-time fallback)"`, means the compile missed the fix). PR 11.0, v0.5.1+ |

`Session` methods: `add_turn(turn)`, `recall(query, top_k=10, memory_types=None)`,
`close()`, `discard()`, `id()`, `agent()`, `turn_ids()`, `turn_count()`; also a
context manager (exiting consolidates).

Memories and results are returned as plain dicts; `metadata` round-trips as a
nested dict.

### Errors

Since v0.13.1 every engine error is an `mnemo_engine.MnemoError`, which
subclasses `RuntimeError`, so `except RuntimeError` keeps working. The
subclasses tell the cases apart (argument errors the binding catches itself,
before the engine sees them, are still `ValueError`):

| Exception | What happened | What to do |
|---|---|---|
| `FormatTooOldError` | the file predates v9: a v8 file opens only with `read_only=True`, v4–v7 not at all | run `mnemo compact <file>` once (`--key-file` for a raw-key file, or `compact_file_with_key`); it keeps every live memory |
| `NeedsWriteOpenError` | the file has pending work (a committed write to finish, a torn header to repair) | open it read-write once; on a pre-v9 file the message says to run `mnemo compact` instead |
| `EmbedderMismatchError` | the embedder given isn't the one the file's vectors came from | use the original embedder, or create a new file for the new model |
| `WrongPassphraseError` | the passphrase or key doesn't open this file | check the secret |
| `LockedError` | another handle (another process, or one not yet closed) has the file | close it, or open read-only alongside a reader |
| `ReadOnlyError` | a write on a `read_only=True` handle | reopen read-write |
| `DimensionMismatchError` | a vector of the wrong width | embed with the file's model |
| `InvalidArgumentError` | a bad argument the engine refused (an all-zero vector, say) | fix the call |
| `NotFoundError` | the id isn't in the file | |
| `UnsupportedVersionError` | a file from a newer engine | upgrade `mnemo-engine` |
| `NotAMnemoFileError` | not a `.mnemo` file | check the path |
| `CorruptError` | a page or the header failed its integrity check: corrupt or tampered | restore from a backup or an export |
| `MnemoError` (only) | I/O, cryptography, key derivation or serialization | the message says which |

```python
import mnemo_engine as mnemo

try:
    db = mnemo.open(path, passphrase)
except mnemo.FormatTooOldError:
    db = mnemo.open(path, passphrase, read_only=True)  # a v8 file; compact it later
```

Each message is the engine's own, unchanged from before, so code that
matched on a message's start keeps working too.

## Result caching

MNemo doubles as a durable result cache — memoize LLM tool-call outputs,
prompt→completion pairs, HTTP responses, anything reconstructible on miss —
in the same encrypted single file that holds your agent memory. See the
[core README's Result caching section](../mnemo/README.md) for the
concepts (namespaces, budgets, TTL, Strict vs Batched flush policy).

### Recipe: `@db.cached(...)` decorator (~90 lines, pure Python)

Copy this into your project as `mnemo_cached.py` or paste inline. It
wraps any function whose arguments serialize to JSON, keying the cache
on `f"{namespace}:{fn_name}:{json_args}"`. When `embed` is supplied it
switches to semantic mode; otherwise it's an exact-key cache.

```python
"""@db.cached — pure-Python helper on top of mnemo.Mnemo's cache API.

Usage:
    import mnemo_engine as mnemo, json, os
    from mnemo_cached import cached

    db = mnemo.open("agent.mnemo", os.environ["MNEMO_PASSPHRASE"])

    @cached(db, namespace="llm", ttl=3600)
    def call_llm(prompt: str) -> str:
        return openai.chat.completions.create(...).choices[0].message.content

    # Second call with the same prompt is a cache hit.
    answer = call_llm("summarize this doc")
"""
from __future__ import annotations

import functools
import hashlib
import json
from typing import Any, Callable, Optional


def cached(
    db,
    *,
    namespace: str,
    ttl: Optional[int] = None,
    embed: Optional[Callable[[str], list[float]]] = None,
    model: Optional[str] = None,
    threshold: float = 0.97,
    key_fn: Optional[Callable[..., str]] = None,
):
    """Memoize a function's results into a mnemo.Mnemo cache.

    - Exact-key mode (default): key is JSON(args, kwargs) plus fn name.
    - Semantic mode: pass `embed=my_embedder` and `model="..."`; the key
      is embedded and semantic recall is tried before falling back.
    - Custom `key_fn(*args, **kwargs) -> str` overrides the default key.
    """
    if embed is not None and model is None:
        raise ValueError("semantic mode requires `model=`")

    def decorator(fn: Callable[..., Any]) -> Callable[..., Any]:
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            key = key_fn(*args, **kwargs) if key_fn else _default_key(fn, args, kwargs)

            # Try semantic hit first (if configured); otherwise exact-key.
            if embed is not None:
                vec = embed(key)
                hit = db.cache_get_semantic(namespace, vec, model, threshold)
                if hit is not None:
                    return _decode(hit["value"])
                # Miss — compute the real result.
                result = fn(*args, **kwargs)
                encoded = _encode(result)
                db.cache_put_semantic(namespace, key, vec, encoded, model, ttl_secs=ttl)
                return result

            hit = db.cache_get(namespace, key)
            if hit is not None:
                return _decode(hit["value"])
            result = fn(*args, **kwargs)
            encoded = _encode(result)
            db.cache_put(namespace, key, encoded, ttl_secs=ttl)
            return result

        return wrapper

    return decorator


def _default_key(fn, args, kwargs) -> str:
    payload = json.dumps(
        {"fn": fn.__qualname__, "args": args, "kwargs": kwargs},
        default=str, sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()


def _encode(v: Any) -> bytes:
    if isinstance(v, (bytes, bytearray)):
        return bytes(v)
    if isinstance(v, str):
        return v.encode()
    return json.dumps(v, default=str).encode()


def _decode(b: bytes) -> Any:
    # Best-effort round-trip: try JSON first, fall back to str, then bytes.
    try:
        return json.loads(b)
    except (json.JSONDecodeError, UnicodeDecodeError):
        try:
            return b.decode()
        except UnicodeDecodeError:
            return b
```

### Recipe: OpenAI/Anthropic call wrapped end-to-end

```python
import os, mnemo
from openai import OpenAI
from mnemo_cached import cached

db = mnemo.open("agent.mnemo", os.environ["MNEMO_PASSPHRASE"])
client = OpenAI()

@cached(db, namespace="openai-gpt-4o-mini", ttl=86_400)
def chat(prompt: str) -> str:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content

# First call → hits the model, caches the response.
# Second identical call → hits the cache, no OpenAI request.
print(chat("Give me a haiku about SQLite."))
print(chat("Give me a haiku about SQLite."))

print("cache stats:", db.cache_stats("openai-gpt-4o-mini"))
```

Anthropic swap-in — same shape, different client:

```python
import anthropic
from mnemo_cached import cached

client = anthropic.Anthropic()

@cached(db, namespace="claude-3-5-sonnet", ttl=86_400)
def chat(prompt: str) -> str:
    r = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return r.content[0].text
```

### Recipe: semantic cache with an embedder

```python
from openai import OpenAI
from mnemo_cached import cached

client = OpenAI()
def embed(text: str) -> list[float]:
    return client.embeddings.create(
        model="text-embedding-3-small", input=text,
    ).data[0].embedding

# Semantic mode: prompts that differ in wording but mean the same
# thing hit the same cache entry.
@cached(db, namespace="llm-semantic", ttl=86_400,
        embed=embed, model="text-embedding-3-small", threshold=0.97)
def chat(prompt: str) -> str:
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content
```

**Note:** the database must have been created with `dimensions` matching
your embedder's output — `text-embedding-3-small` is 1536-dim,
`bge-large-en-v1.5` is 1024-dim, etc. Set at
`mnemo.open(path, pw, dimensions=1536)` on first creation.

**Don't use this for:** multi-node fleets where the cache must be
consistent across hosts (use Redis / DynamoDB DAX). MNemo's cache is
optimized for a single-host agent that owns its cache file.

### Recipe: `set_embedder` + text-only `remember` / `recall`

Phase 3 ships a per-handle embedder hook so callers never have to
pass vectors by hand. Register once at open time; every subsequent
`remember_text` / `recall_text` / `cache_get_semantic_text` call on
the handle uses it.

```python
import mnemo_engine as mnemo
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, normalized

def embed(texts: list[str]) -> list[list[float]]:
    return model.encode(texts, normalize_embeddings=True).tolist()

db = mnemo.open("agent.mnemo", "pw", dimensions=384)
db.set_embedder(embed, name="all-MiniLM-L6-v2",
                dimensions=384, normalize=True)

db.remember_text("the user prefers tea", "semantic",
                 importance=0.8, agent_id="assistant")
hits = db.recall_text("how does the user like their drinks?", top_k=5)
for h in hits:
    print(h["score"], h["content"])
```

Or with OpenAI's hosted embeddings:

```python
from openai import OpenAI
client = OpenAI()

def embed(texts: list[str]) -> list[list[float]]:
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [item.embedding for item in r.data]

db = mnemo.open("agent.mnemo", "pw", dimensions=1536)
db.set_embedder(embed, name="text-embedding-3-small",
                dimensions=1536, normalize=False)
```

The identity (`name`, `dimensions`, `normalize`) is recorded in the file
the first time you `set_embedder` (since v0.13 on every file, with or
without a manifest; a file with a manifest also keeps it there).
Reopening with a *different* embedder — say you swap MiniLM for BGE on
the same file — raises `RuntimeError: embedder identity mismatch: this
file's vectors come from embedder (name="all-MiniLM-L6-v2", …), but the
embedder given is (name="bge-large-en-v1.5", …). Open the file with its
original embedder, or create a new store for the new model.` A file from
before v0.13 records the first embedder attached after the upgrade.

Mnemo also now refuses any `remember` call with an all-zero vector,
so a mis-wired embedder (returning `[0.0] * dims`) fails loudly on
the first write instead of polluting recall silently.

## License

Apache-2.0.

