Metadata-Version: 2.4
Name: sieve-memory
Version: 0.1.1
Summary: A fast, local, calibrated memory-curation layer for AI agents, built on Laya.
Author-email: Ashutosh Bansal <ashutoshbansalcs@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/ashutoshb206/sieve
Project-URL: Documentation, https://github.com/ashutoshb206/sieve#readme
Project-URL: Repository, https://github.com/ashutoshb206/sieve
Project-URL: Issues, https://github.com/ashutoshb206/sieve/issues
Keywords: memory,ai-agents,garbage-collection,llm,context-window,laya,langchain,vector-db
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: laya
Requires-Dist: laya; extra == "laya"
Provides-Extra: serve
Requires-Dist: fastapi; extra == "serve"
Requires-Dist: uvicorn; extra == "serve"
Provides-Extra: integrations
Requires-Dist: chromadb; extra == "integrations"
Requires-Dist: langchain-core; extra == "integrations"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: httpx; extra == "dev"
Provides-Extra: all
Requires-Dist: laya; extra == "all"
Requires-Dist: fastapi; extra == "all"
Requires-Dist: uvicorn; extra == "all"
Requires-Dist: chromadb; extra == "all"
Requires-Dist: langchain-core; extra == "all"
Requires-Dist: pytest; extra == "all"
Requires-Dist: httpx; extra == "all"
Dynamic: license-file

<div align="center">

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://sieve-memory.vercel.app/logo-white.svg">
  <source media="(prefers-color-scheme: light)" srcset="https://sieve-memory.vercel.app/logo-black.svg">
  <img src="https://sieve-memory.vercel.app/logo.png" alt="Sieve Sweep Logo" width="96" height="96" />
</picture>

# Sieve

### Autonomous Memory Compaction & Self-Cleaning Store for AI Agents

[![PyPI version](https://img.shields.io/pypi/v/sieve-memory.svg)](https://pypi.org/project/sieve-memory/)
[![Website](https://img.shields.io/badge/website-sieve--memory.vercel.app-10B981)](https://sieve-memory.vercel.app/)
[![Docs](https://img.shields.io/badge/docs-interactive-blue)](https://sieve-memory.vercel.app/docs)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![Tests](https://img.shields.io/badge/tests-53%20passed-brightgreen.svg)](https://sieve-memory.vercel.app/)
[![Zero Dependencies Option](https://img.shields.io/badge/core%20deps-zero%20(pure%20python)-orange.svg)](https://github.com/ashutoshb206/sieve)

**Sieve acts as an automated garbage collector and cache eviction daemon for AI agent memory.**  
Instead of dumping endless logs into a vector store or burning money on an LLM call every turn to resolve contradictions, Sieve continuously scores, supersedes, and prunes memories in the background using non-autoregressive calibrated decisions (**~33ms, 100% local, $0 API cost**).

[Why Sieve?](#the-problem-sieve-solves) •
[Quickstart](#quickstart-5-minutes) •
[Interactive Web UI](#interactive-web-dashboard) •
[Integration Guide](#how-to-use-sieve-in-your-projects) •
[Benchmarks](#reproducible-benchmarks) •
[Architecture](#how-it-works) •
[REST API & CLI](#rest-api--cli)

</div>

---

## What Sieve Does (In 30 Seconds)

1. **Ingest & Score (~0.2ms Mock / ~33ms Laya)**: Categorizes incoming conversation turns into `decision`, `reference`, `task`, or `smalltalk` and scores them on calibrated scales for `importance` (0.0 to 1.0) and `actionable` (0.0 to 1.0).
2. **Background Pairwise Sweeps (FTS5 BM25)**: Evaluates whether a new fact contradicts or supersedes an older fact (e.g. *"We switched from PostgreSQL to MongoDB"* supersedes *"Our database is PostgreSQL"*). The outdated item is marked `status='archived'` with `superseded_by='<new_id>'`.
3. **Pinned Memory Retention**: High-priority facts (or manual pins) remain immune to staleness pruning and stay active indefinitely.
4. **Clean System Prompt Context**: `memory.get_context(namespace="user_1")` or `GET /context` yields clean, deduplicated, non-contradictory context ready for injection into your LLM prompt.

---

### Pattern 4: Standalone Microservice (FastAPI REST Server)

Run Sieve as an internal microservice for distributed agents across any tech stack (Python, TypeScript, Go, Rust):

```bash
pip install "sieve-memory[serve]"
sieve serve --port 8000 --db sieve.db --api-key my_secret_key
```

* `GET /context?namespace=default`: Formatted markdown bullet points ready for LLM prompt injection
* `POST /items`: Ingest single memory with immediate scoring
* `POST /items/batch`: Bulk ingest with per-item status tracking
* `GET /active?limit=50&offset=0`: Fetch active memory items (including pinned) ranked by importance
* `GET /inbox?threshold=0.6`: Fetch open action items ranked by actionability
* `POST /item/{id}/pin` & `/unpin`: Pin or unpin memories
* `POST /item/{id}/restore`: Reactivate an archived memory and clear supersession
* `GET /namespaces`: List all active namespaces
* `DELETE /namespace/{namespace}`: Permanently delete all memories in a namespace
* `GET /metrics`: System operational metrics (DB size, item counts, scorer backend, sweep durations)
* `POST /sweep/staleness`: Trigger background staleness sweep
* `POST /sweep/redundancy`: Trigger background redundancy detection
* `POST /sweep/retention`: Enforce retention bounds (`max_active_items` and `archive_ttl_seconds`)
* `GET /ui`: Interactive web dashboard (supports API key token prompt and toast notifications)

### Production Deployment & Security

```bash
# Production server with API key auth, custom checkpoint, and SQLite WAL storage
SIEVE_DB_PATH=/var/data/sieve.db \
SIEVE_API_KEY=your-secure-secret-token \
SIEVE_SCORER=laya \
SIEVE_LAYA_CHECKPOINT=convaiinnovations/laya \
SIEVE_LAYA_DEVICE=cuda \
uvicorn sieve.api:app --host 0.0.0.0 --port 8000
```

| Setting | Default | Purpose |
| :--- | :--- | :--- |
| `SIEVE_DB_PATH` | `sieve.db` | Filepath to persistent SQLite database. Runs in WAL mode with auto-migrations. |
| `SIEVE_API_KEY` | *(None)* | Bearer token / `X-API-Key` auth. When set, all read/write endpoints require authentication. |
| `SIEVE_SCORER` | `mock` | Decision backend: `mock` (offline heuristic) or `laya` (calibrated neural model). |
| `SIEVE_LAYA_CHECKPOINT` | `convaiinnovations/laya` | Hugging Face model checkpoint for LayaScorer. Cached in memory. |
| `SIEVE_LAYA_DEVICE` | `None` *(auto)* | Target device for neural inference: `cuda`, `mps`, or `cpu`. |
| `SIEVE_CORS_ORIGINS` | *(None)* | Comma-separated list of allowed origins (or `*`) for external browser clients. |

> **Dashboard Authentication**: When `SIEVE_API_KEY` is configured on the server, the `/ui` dashboard provides an inline **🔑 API Key** manager modal, storing your token in this browser's `localStorage` and attaching it to all asynchronous requests.
> **Security Note**: Never expose Sieve directly to the public internet without `SIEVE_API_KEY` configured or without an authenticated reverse-proxy (Nginx/Caddy) providing TLS and access control.

### Retention Policy & Operational Backups

Sieve supports automatic capacity bounds and live online backups:

```bash
# Atomic online hot backup (WAL-safe without locking concurrent writes)
sieve backup /var/backups/sieve_$(date +%Y%m%d).db --db /var/data/sieve.db

# Enforce active context limits and TTL purging via CLI
sieve retention --namespace user_1 --max-active 20 --ttl 2592000
```

In Python:
```python
from sieve import SieveMemory, PolicyConfig

# Configure retention limits: max 20 active facts, purge archived facts older than 30 days
policy = PolicyConfig(
    max_active_items=20,
    archive_ttl_seconds=30 * 86400,
    pinned_immune=True,  # Pinned items are never auto-pruned
)
memory = SieveMemory(policy_config=policy)
```
---

## The Problem Sieve Solves

Every modern AI agent stores conversational memory, but current approaches break at scale:

| Approach | Latency | Per-Turn Cost | Resolves Supersession & Staleness? |
| :--- | :--- | :--- | :--- |
| **Vector DBs (Chroma, Pinecone)** | ~15ms | $0 | ❌ **No.** Embeddings compute *topical similarity*, not *ground truth*. If an agent records *"User prefers PostgreSQL"* on Monday and *"Switched to MongoDB"* on Wednesday, vector search returns both with high cosine similarity. The LLM hallucinates outdated facts. |
| **Generative LLMs (Mem0, Letta)** | ~800–2500ms | $$$ | ✅ **Yes, but slow & expensive.** Asking GPT-4o on every single conversational turn to extract facts and check for contradictions adds 1–2 seconds of user latency and thousands of unnecessary tokens. |
| **Sieve (Built on Laya)** | **~33ms** | **$0** | **✅ Yes.** Calibrated non-autoregressive scoring runs locally. Background sweeps automatically supersede outdated facts and keep system prompts clean and up-to-date. |

> **Sieve is not a vector store, not a search index, and not a chat UI.**  
> It is the lightweight metadata and curation layer that sits beside your agent storage and tells you what to **keep**, **pin**, or **retire**.

---

## Quickstart (5 Minutes)

### 1. Installation

```bash
# Zero-dependency core (runs anywhere, instant setup, built-in heuristic MockScorer)
pip install sieve-memory

# Optional extras for specific deployment targets:
pip install "sieve-memory[laya]"         # Real neural decision model (ConvAI Laya checkpoint)
pip install "sieve-memory[serve]"        # FastAPI REST microservice & visual UI dashboard
pip install "sieve-memory[integrations]" # LangChain & ChromaDB connectors
pip install "sieve-memory[all]"          # Complete suite with all backends
```

### 2. Run the Interactive Demos

```bash
# Agent workflow demo (remember -> update -> sweep -> context window)
python examples/agent_quickstart.py

# End-to-end batch curation demo
python examples/run_v0.py
```

---

## Interactive Web Dashboard

Sieve ships with a built-in dark mode dashboard for visualizing memory curation in real-time.

```bash
# Start the visual dashboard (or pass --db path)
sieve ui --port 8000
```
Open **`http://localhost:8000/ui`** in your browser:
* **Active Memories**: Real-time cards showing importance levels, categories, and pinning controls.
* **Superseded & Stale**: View facts pruned by background sweeps with pointers to the superseding item.
* **Action Items Inbox**: Real-time ranking of open tasks and pending follow-ups.
* **System Prompt Live Preview**: Inspect exactly what prompt context Sieve generates for your LLM.

---

## How to Use Sieve in Your Projects

### Pattern 1: Drop-in Memory for Any AI Agent (OpenAI, Anthropic, LangChain)

Use `SieveMemory` directly inside your agent loop. It automatically partitions by namespace (`user_id` or `session_id`), supports continuous background workers, and prepares clean context strings for your system prompt:

```python
from sieve import SieveMemory

# 1. Initialize memory store (persisted in SQLite + FTS5)
memory = SieveMemory(db_path="agent_memory.db")

# Optional: Start background curation daemon (runs sweeps every 60s)
memory.start_daemon(interval_seconds=60)

# 2. Record conversation facts as your agent talks to a user
session = "user_42"
memory.remember("User's primary database is PostgreSQL.", namespace=session)
memory.remember("TODO: Send invoice to Alice by Friday.", namespace=session)
memory.remember("Deployment target is AWS us-east-1.", namespace=session)

# 3. Later, the user changes a decision:
memory.remember("Update: We switched the database from PostgreSQL to MongoDB.", namespace=session)

# 4. Trigger a sweep manually (or let the background daemon handle it)
sweep_stats = memory.sweep(namespace=session)
print(f"Archived {sweep_stats['items_archived']} superseded items.")

# 5. Inject only active, non-superseded facts into your LLM system prompt
system_context = memory.get_context(namespace=session)
print(system_context)
```

**What the LLM sees in its prompt:**
```text
- [DECISION] Update: We switched the database from PostgreSQL to MongoDB.
- [TASK] TODO: Send invoice to Alice by Friday.
- [REFERENCE] Deployment target is AWS us-east-1.
```
*(Notice: The old PostgreSQL fact was automatically identified as superseded, archived, and excluded from the active context window.)*

---

### Pattern 2: Modern LangChain Agent Integration (`LCEL`)

Use `SieveChatMessageHistory` as a drop-in replacement for standard message history in LangChain:

```python
from sieve.integrations.langchain import SieveChatMessageHistory

# Automatically prunes superseded messages every 4 turns
history = SieveChatMessageHistory(session_id="user_alice", db_path="memory.db")

history.add_user_message("We chose PostgreSQL as our database.")
history.add_ai_message("Got it, PostgreSQL configured.")
history.add_user_message("Actually, we migrated to MongoDB instead.")

# In your LangChain agent prompt:
# Only active, non-superseded messages remain in history.messages!
print(history.messages)
```

---

### Pattern 3: Sidecar alongside ChromaDB / Vector DBs

You don't need to replace your vector database. Sieve acts as an **active filter**:

```python
import chromadb
from sieve.integrations.chroma import SieveChromaSync

client = chromadb.Client()
collection = client.create_collection("agent_docs")
sync = SieveChromaSync(collection, db_path="sieve.db")

# Add memories into both Sieve and Chroma
sync.add(
    documents=["Primary DB is PostgreSQL.", "Actually, switched to MongoDB."],
    ids=["doc_1", "doc_2"],
    namespace="user_alice",
)

# Run sweep: marks doc_1 as archived in both Sieve and Chroma metadata
sync.sweep(namespace="user_alice")

# Query Chroma: automatically filters out stale/superseded records!
results = sync.query(
    query_texts=["What database are we using?"],
    namespace="user_alice",
    only_active=True,
)
```

---

### Pattern 4: Standalone Microservice (FastAPI REST Server)

Run Sieve as an internal microservice for distributed agents across any tech stack (Python, TypeScript, Go, Rust):

```bash
pip install "sieve-memory[serve]"
sieve serve --port 8000 --db sieve.db --api-key my_secret_key
```

* `GET /context?namespace=default`: Formatted markdown bullet points ready for LLM prompt injection
* `POST /items`: Ingest single memory with immediate scoring
* `POST /items/batch`: Bulk ingest with per-item status tracking
* `GET /active?limit=50&offset=0`: Fetch active memory items (including pinned) ranked by importance
* `GET /inbox?threshold=0.6`: Fetch open action items ranked by actionability
* `POST /item/{id}/pin` & `/unpin`: Pin or unpin memories
* `POST /item/{id}/restore`: Reactivate an archived memory and clear supersession
* `GET /namespaces`: List all active namespaces
* `GET /metrics`: System operational metrics (DB size, item counts, scorer backend)
* `POST /sweep/staleness`: Trigger background staleness sweep
* `POST /sweep/redundancy`: Trigger background redundancy detection
* `GET /ui`: Interactive web dashboard (supports API key token prompt)

### Production Deployment & Security

```bash
# Production server with API key auth, custom checkpoint, and SQLite WAL storage
SIEVE_DB_PATH=/var/data/sieve.db \
SIEVE_API_KEY=your-secure-secret-token \
SIEVE_SCORER=laya \
SIEVE_LAYA_CHECKPOINT=convaiinnovations/laya \
SIEVE_LAYA_DEVICE=cuda \
uvicorn sieve.api:app --host 0.0.0.0 --port 8000
```

| Setting | Default | Purpose |
| :--- | :--- | :--- |
| `SIEVE_DB_PATH` | `sieve.db` | Filepath to persistent SQLite database. Runs in WAL mode with auto-migrations. |
| `SIEVE_API_KEY` | *(None)* | Bearer token / `X-API-Key` auth. When set, all read/write endpoints require authentication. |
| `SIEVE_SCORER` | `mock` | Decision backend: `mock` (offline heuristic) or `laya` (calibrated neural model). |
| `SIEVE_LAYA_CHECKPOINT` | `convaiinnovations/laya` | Hugging Face model checkpoint for LayaScorer. Cached in memory. |
| `SIEVE_LAYA_DEVICE` | `None` *(auto)* | Target device for neural inference: `cuda`, `mps`, or `cpu`. |

> **Dashboard Authentication**: When `SIEVE_API_KEY` is configured on the server, the `/ui` dashboard provides an inline **🔑 API Key** manager modal, storing your token in this browser's `localStorage` and attaching it to all asynchronous requests.
> **Security Note**: Never expose Sieve directly to the public internet without `SIEVE_API_KEY` configured or without an authenticated reverse-proxy (Nginx/Caddy) providing TLS and access control.

---

## Reproducible Benchmarks

Run the local pipeline benchmark on your machine:
```bash
python benchmarks/benchmark.py --items 50
```

> **Benchmark Transparency**: The performance numbers below evaluate the local pipeline infrastructure (SQLite FTS5 candidate pairing, scoring throughput, and token bloat reduction) using the zero-dependency `MockScorer`. Real-world neural inference via `LayaScorer` adds ~33ms per evaluated pair.

### Typical Pipeline Baseline (Local M-Series / Linux):

```text
======================================================================
 SUMMARY COMPARISON TABLE
======================================================================
Metric                       | Sieve (Local)      | LLM per Turn (4o-mini)
------------------------------------------------------------------------
Latency per Item             | 1.2 ms             | ~850 ms             
API Cost (10,000 items)      | $0.00              | ~$1.35              
Synthetic Benchmark Acc     | ~100% (synthetic)  | ~95%                
Context Window Growth        | Bounded (Pruned)   | Bounded             
Context Bloat Reduction      | 33.5% saved        | 30–35% saved        
======================================================================
```

---

## Switching from Mock to Real Neural Inference (Laya)

By default, Sieve runs `MockScorer`—a zero-dependency heuristic engine designed for rapid development and CI tests.

To use the real **Laya** calibrated neural checkpoint:

```bash
pip install laya
```

```python
from sieve import SieveMemory

# Loads the calibrated Laya checkpoint from Hugging Face (~33ms per inference)
memory = SieveMemory(use_real_laya=True)
```

Every other layer of your stack (SQLite store, FTS5 sweeps, policy rules, REST API) remains completely unchanged.

---

## How It Works

```
Raw Text Memory
      │
      ▼
1. INGESTION  ───► SQLite Store (Partitioned by Namespace)
      │
      ▼
2. CALIBRATED ───► Evaluates:
   SCORING         • Importance: Will this matter beyond the moment? (0.0 - 1.0)
                   • Actionable: Is there an open task or follow-up? (P(true))
                   • Category: decision | reference | task | smalltalk
      │
      ▼
3. POLICY     ───► • Pin if importance >= 0.85 (immune to auto-archiving)
   ENGINE          • Route to Action Inbox if actionable >= 0.60
                   • Queue for batched review if confidence is low
      │
      ▼
4. PAIRWISE   ───► • Uses SQLite FTS5 BM25 to find topically related candidates
   SWEEPS          • Scores staleness: Does the newer item invalidate the older one?
                   • Archives outdated facts with pointer to `superseded_by`
```

### What Sieve Scores

| Metric | Type | Question Evaluated |
|---|---|---|
| **`importance`** | `score` | *Will this matter beyond the moment it was written?* |
| **`actionable`** | `noul` | *Is there an open task, pending decision, or required follow-up?* |
| **`category`** | `choice` | `decision` · `reference` · `task` · `smalltalk` |
| **`staleness`** | `noul` (pairwise) | *Does the newer item make the older item outdated or invalid?* |
| **`redundancy`** | `noul` (pairwise) | *Do two items convey substantially identical information?* |

---

## CLI Reference

```bash
# Launch Sieve REST API server & web dashboard
sieve serve --port 8000 --db sieve.db --api-key my_secret

# Launch visual dashboard in browser
sieve ui --port 8000

# Remember a fact directly from terminal
sieve remember "Decided to adopt PostgreSQL" --namespace project_x

# Print active memory context formatted for direct LLM system prompt injection
sieve context --namespace project_x

# Pin critical items so they are permanently protected from pruning
sieve pin cli_179166_4ff0
sieve unpin cli_179166_4ff0

# Archive or delete memories
sieve archive cli_179166_4ff0
sieve delete cli_179166_4ff0

# Run curation sweeps (supersede outdated facts)
sieve sweep --namespace project_x

# Check open task inbox
sieve inbox --namespace project_x

# View store statistics
sieve stats --namespace project_x
```

---

## Testing

Sieve has 100% passing tests covering CRUD, thread-safety, namespaces, FTS5 BM25 candidate ranking, policy staleness, LangChain & ChromaDB integrations, background daemons, and the REST API:

```bash
python -m pytest tests/ -v
```

---

## License

MIT © [Ashutosh Bansal](https://github.com/ashutoshb206)
