Metadata-Version: 2.4
Name: memvault-ai
Version: 0.1.0
Summary: Open source, local-first persistent memory system for LLMs
Project-URL: Homepage, https://github.com/memvault/memvault
Project-URL: Repository, https://github.com/memvault/memvault
Project-URL: Issues, https://github.com/memvault/memvault/issues
Author: MemVault Contributors
License-Expression: MIT
License-File: LICENSE
Keywords: knowledge-graph,llm,local-first,mcp,memory
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: chromadb<0.6.0,>=0.4.0
Requires-Dist: litellm>=1.0.0
Requires-Dist: mcp>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rank-bm25>=0.2.2
Requires-Dist: rich>=13.0.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: typer>=0.9.0
Provides-Extra: dev
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: nli
Requires-Dist: torch>=2.0.0; extra == 'nli'
Requires-Dist: transformers>=4.30.0; extra == 'nli'
Provides-Extra: reranker
Requires-Dist: sentence-transformers>=2.2.0; extra == 'reranker'
Description-Content-Type: text/markdown

# ⚡ MemVault

**Open source, local-first memory system for Claude and MCP workflows.**

> Zero data leaves your machine. Ever.

[![Tests](https://img.shields.io/badge/tests-329%20passed-brightgreen)]()
[![Python](https://img.shields.io/badge/python-3.9+-blue)]()
[![License](https://img.shields.io/badge/license-MIT-green)]()
[![R@5](https://img.shields.io/badge/R%405-100%25-brightgreen)]()

---

## The Problem

LLMs are stateless mathematical functions — `y = f(x, θ)` — that forget everything between sessions.

| Metric | Value |
|--------|-------|
| 6-month AI conversation history | ~19.5 million tokens |
| Largest context window (Claude) | ~200k tokens |
| **MemVault wake-up cost** | **~350 tokens** |
| Annual cost estimate | ~$10/yr |
| Data leaving your machine | **Zero** |

## Quick Start

```bash
# Install
pip install memvault

# Initialize
memvault init

# For manual backfill, mine a project's Claude sessions first
memvault mine-convos ~/.claude/projects/-Users-you-my-project --project my-project --track auth

# Recall project decisions from session memory
memvault recall "What database did we choose for the auth service?" --project my-project --track auth

# Index project code/docs separately when needed
memvault index-project /path/to/repo --project my-project --track auth

# Search indexed project files separately
memvault search-project "database migrations" --project my-project --track auth

# Generate wake-up context for a new AI session
memvault wake-up
```

### Recommended Beta Workflow

MemVault is most useful when we keep session memory and project indexing separate.

- Use `mine-convos` and `recall` for session recall, decisions, and prior discussions.
- Use `index-project` and `search-project` only when you explicitly want code/docs indexing.
- Scope search to a project whenever possible.
- Use `--track` to narrow within a project, for example `auth`, `infra`, or `onboarding`.
- Avoid broad mixed ingestion as the default beta workflow.
- For Claude, prefer the automatic hook flow for new sessions and use `mine-convos` for backfills.

For decision-style questions, MemVault now favors conversational memory before raw file text.

### Automatic Claude Setup

For a one-time Claude installation with automatic session-start injection, automatic session-end mining, and MCP tool wiring:

```bash
memvault init
memvault install --client claude
```

After that, Claude will:

- inject MemVault context automatically at session start
- mine the finished transcript automatically at session end
- infer `project` and `track` from the working directory and transcript when possible
- expose MemVault MCP tools during the session

Recommended Claude pattern:

1. Install once with `memvault install --client claude`
2. Work normally in Claude inside a project folder such as `/Users/you/keystone/backend/auth`
3. Let the stop hook file the session into the `keystone` project and `auth` track automatically
4. Use `memvault recall --project keystone --track auth` only when you want to inspect what got stored manually

### MCP Integration (Cursor, VS Code, etc.)

Add to your MCP settings:

```json
{
  "mcpServers": {
    "memvault": {
      "command": "memvault",
      "args": ["serve"]
    }
  }
}
```

The AI assistant automatically gets 20 MCP tools for search, context retrieval, duplicate checks, memory management, knowledge graph access, and contradiction workflows.

If your MCP client can send its current working directory, MemVault can now infer:

- `project` from the active repo path
- `track` from the current sub-area such as `auth`, `infra`, or `onboarding`
- the best retrieval lane automatically, with a softer fallback if the first lane is empty

## Architecture

### 4-Layer Memory Stack

```
┌──────────────────────────────────────────────────┐
│  L0  Identity               ~50 tokens   ALWAYS  │
├──────────────────────────────────────────────────┤
│  L1  Critical Facts         ~300 tokens  ALWAYS  │
├──────────────────────────────────────────────────┤
│  L2  Project Context        ~2,000 tokens ON-DEM │
├──────────────────────────────────────────────────┤
│  L3  Deep Search            ~5,000 tokens ON-DEM │
└──────────────────────────────────────────────────┘
```

Wake-up loads **L0+L1 only** (~350 tokens). Deeper context is fetched only when needed through MCP tool calls.

### Memory Types

| Type | Store | Examples |
|------|-------|----------|
| **Episodic** | ChromaDB | Past events, sessions, decisions |
| **Semantic** | SQLite KG | Facts about user, projects, entities |
| **Procedural** | SQLite | Preferences, coding style, guardrails |

### Retrieval Strategy

MemVault keeps two retrieval lanes:

- **Session memory** for decisions, prior discussions, and corrections
- **Project memory** for code/docs context that you explicitly index

Each lane can be narrowed with:

- **project** for the overall codebase or initiative
- **track** for a narrower thread inside that project, such as auth, infra, or onboarding

For beta, the intended flow is:

1. Let Claude auto-mine new sessions through hooks
2. Backfill older conversations with `mine-convos` when needed
3. Recall within a project and optional track
4. Index project files separately only when needed
5. Pull only the relevant memories needed for the current question

### Hybrid Search Pipeline

```
Query ─┬─► BM25 Keyword Search ──┐
       │                          ├─► RRF Fusion (k=60) ─► Scoring ─► [Reranker] ─► Results
       └─► ChromaDB Vectors ─────┘
```

**Scoring formula:** `final = relevance×0.40 + confidence×0.35 + recency×0.25`

### Intelligence Features

- **Contradiction Detection** — Rule-based + optional NLI (DeBERTa-v3-base). Every fact is checked against existing knowledge.
- **Entity Detection** — 80+ technologies, concepts, and person names auto-detected and registered in the Knowledge Graph.
- **Memory Decay** — Time-based confidence decay (soft at 180 days, hard at 365 days). Keeps memory fresh.
- **Extractive Summarization** — Decision-focused sentence scoring. No external LLM required.

## Benchmark Results

Current benchmark results from local runs:

```
Config:      Vector-only
Queries:     15
R@1:         86.7%
R@3:         100.0%
R@5:         100.0%
R@10:        100.0%
MRR:         0.9333
Avg Latency: 384.3ms
```

```text
Config:      BM25 + Vector + RRF (Hybrid)
Queries:     15
R@1:         86.7%
R@3:         93.3%
R@5:         100.0%
R@10:        100.0%
MRR:         0.9167
Avg Latency: 309.5ms
```

```text
Config:      BM25 + Vector + RRF + CrossEncoder
Queries:     15
R@1:         86.7%
R@3:         93.3%
R@5:         100.0%
R@10:        100.0%
MRR:         0.9167
Avg Latency: 739.3ms
```

Detailed report: [benchmarks/REAL_BENCHMARK_REPORT.md](/Users/koushik.reddy/llm_context/memvault/benchmarks/REAL_BENCHMARK_REPORT.md)

Run benchmarks: `python benchmarks/longmemeval_bench.py`

## Tech Stack

| Component | Technology |
|-----------|-----------|
| Language | Python 3.9+ |
| Package Manager | uv |
| Vector Store | ChromaDB ≥0.4.0 (embedded, no server) |
| Knowledge Graph | SQLite temporal entity-relationship triples |
| Embeddings | all-MiniLM-L6-v2 (local, 80MB) |
| Keyword Search | rank-bm25 with RRF hybrid fusion |
| Re-ranking | ms-marco-MiniLM-L-6-v2 (optional, local) |
| Contradiction | DeBERTa-v3-base NLI (optional, local) |
| MCP Server | mcp Python SDK |
| CLI | Typer + Rich |

## CLI Commands

```bash
# Core
memvault init [--dir DIR]           # Initialize MemVault
memvault install [--client claude]  # One-time client integration install
memvault status                     # System status
memvault mine <dir> [--mode convos] [--project P]  # Beta default: conversation-first ingestion
memvault mine-convos <dir> [--project P] [--track T]   # Session memory only
memvault index-project <dir> [--project P] [--track T] # Project files/docs only
memvault search "query" [--scope auto] [--cross]       # Hybrid search with lane control
memvault recall "query" [--project P] [--track T]      # Session-first recall
memvault search-project "query" [--project P] [--track T] # Project/document search
memvault wake-up [--raw]            # L0+L1 context

# Memory Ops
memvault health                     # Memory health dashboard
memvault explain "query" [--project P] [--track T]      # Why MemVault would answer this
memvault inspect-memory <id>        # Show stored memory text + metadata
memvault edit-memory <id> [--text]  # Correct stored memory text/metadata
memvault save-memory "text" [--kind decision] [--project P] [--track T] # Durable manual filing
memvault tracks --project P         # Review known tracks within a project
memvault dashboard [--project P]    # Pilot-friendly project/track dashboard
memvault review-stale [--project P] # Review stale memories that need cleanup
memvault contradictions [-p PROJ]   # List contradictions
memvault resolve <id> [--keep new]  # Resolve contradiction
memvault decay [--dry-run]          # Apply memory decay
memvault forget [concept|--project] # Selective deletion
memvault rebuild-layers             # Refresh derived L0/L1 context inputs

# Server
memvault install-claude             # Claude-specific installer
memvault serve                      # Start MCP server (stdio)
```

## MCP Tools

| Tool | Description |
|------|-------------|
| `memvault_status` | System status and memory counts |
| `memvault_search` | Hybrid search across all stores |
| `memvault_get_context` | Wake-up context (L0+L1, ~350 tokens) |
| `memvault_get_project_context` | Project-specific L2 context |
| `memvault_list_projects` | List tracked projects |
| `memvault_list_entities` | List KG entities |
| `memvault_check_duplicate` | Check whether a memory likely already exists |
| `memvault_explain_answer` | Explain which stored memories support an answer |
| `memvault_get_memory` | Inspect one stored memory by ID |
| `memvault_add_memory` | Add a memory directly through MCP |
| `memvault_edit_memory` | Edit a stored memory and its scope/metadata |
| `memvault_delete_memory` | Delete a memory by ID |
| `memvault_forget` | Forget by concept, project, or entity |
| `memvault_update_confidence` | Update a memory confidence score |
| `memvault_detect_entities` | Detect + register entities from text |
| `memvault_kg_query` | Query the knowledge graph |
| `memvault_kg_add` | Add a knowledge graph triple |
| `memvault_kg_invalidate` | Invalidate a knowledge graph triple |
| `memvault_kg_timeline` | Show an entity timeline |
| `memvault_kg_cross_project` | Query KG facts across projects |
| `memvault_health` | Return memory health metrics |
| `memvault_contradictions` | List detected contradictions |
| `memvault_resolve` | Resolve a contradiction |

## Optional Dependencies

```bash
# Cross-encoder re-ranking (improves precision)
pip install memvault[reranker]

# NLI contradiction detection (semantic)
pip install memvault[nli]

# Everything
pip install memvault[reranker,nli]
```

## Development

```bash
# Clone and install
git clone https://github.com/memvault/memvault.git
cd memvault
uv pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run benchmarks
python benchmarks/longmemeval_bench.py --all -v

# Lint
ruff check memvault/ tests/

# Type check
mypy memvault/
```

### Project Structure

```
memvault/
├── memvault/
│   ├── __init__.py            # Package exports
│   ├── cli.py                 # Typer CLI
│   ├── config.py              # Configuration + constants
│   ├── knowledge_graph.py     # SQLite temporal KG
│   ├── user_model.py          # User preferences + memory metadata
│   ├── vectorstore.py         # ChromaDB wrapper
│   ├── convo_parser.py        # Multi-format parser
│   ├── session_scorer.py      # Session quality scoring
│   ├── ingestor.py            # Chunking + embedding pipeline
│   ├── searcher.py            # Hybrid search (BM25+Vector+RRF)
│   ├── reranker.py            # Cross-encoder re-ranking
│   ├── layers.py              # L0-L3 memory stack
│   ├── context_builder.py     # Dynamic token budget packing
│   ├── fact_checker.py        # NLI contradiction detection
│   ├── entity_detector.py     # Entity detection + registration
│   ├── memory_pipeline.py     # End-to-end orchestrator
│   ├── summarizer.py          # Extractive summarization
│   ├── claude_integration.py  # Claude lifecycle + MCP installer
│   ├── mcp_server.py          # MCP server tools for search, save, inspect, and KG access
│   └── onboarding.py          # Guided setup flow
├── tests/                     # 329 tests
├── benchmarks/                # LongMemEval benchmark suite
├── hooks/                     # Legacy shell hook examples
└── pyproject.toml
```

## How It Works

1. **Mine Conversations** — Parse sessions first, score them, chunk them, embed them, and register entities in the KG.
2. **Index Projects** — Index code/docs separately when you want implementation lookup.
3. **Recall** — Decision queries favor session memory before file text.
4. **Inspect / Edit** — Use `explain`, `inspect-memory`, and `edit-memory` to debug or correct what MemVault stored.
5. **Search Projects** — Implementation queries can target indexed project files explicitly.
6. **Wake-Up** — Load ~350 tokens of identity + critical facts at session start. No expensive retrieval.
7. **On-Demand** — When the AI needs deeper context, MCP tools fetch scoped memory on the fly.
8. **Integrity** — New facts are checked against existing knowledge for contradictions. Stale memories decay over time.

## Claude Lifecycle

`memvault install --client claude` writes three Claude hooks plus the MemVault MCP server entry into `~/.claude/settings.json`:

- `SessionStart` runs `python -m memvault hook session-start` and injects wake-up context automatically.
- `Stop` runs `python -m memvault hook stop` and mines the Claude session transcript automatically.
- `PreCompact` runs `python -m memvault hook precompact` and applies memory decay before compaction.

The installer is safe to re-run: it updates existing MemVault hooks, preserves unrelated Claude settings, and creates a timestamped backup before modifying an existing settings file.

## Beta Notes

- Conversation-first mining is the recommended default for Claude workflows.
- Project indexing is optional and should be used intentionally.
- Dependency folders and generated content are excluded from raw file mining by default.
- The cross-encoder reranker is optional and slower; the hybrid default is usually the best latency/quality tradeoff.
- The intended workflow is `mine-convos` + `recall` first, then `index-project` + `search-project` when code/docs lookup is needed.

## Pilot Workflow

For an internal pilot, the most practical workflow is:

1. Install once for Claude:
   ```bash
   memvault init
   memvault install --client claude
   ```
2. Work normally in Claude inside a repo or sub-area like `keystone/backend/auth`
3. Let Claude auto-mine sessions on exit
4. Ask recall questions naturally in a new Claude session
5. Use the inspection tools when needed:
   ```bash
   memvault explain "What database did we choose for keystone auth?" --project keystone --track auth
   memvault inspect-memory <memory-id>
   memvault edit-memory <memory-id> --confidence 0.95
   memvault dashboard
   memvault review-stale --project keystone
   ```
## License

MIT — see [LICENSE](LICENSE).
