Metadata-Version: 2.5
Name: outhad_contextkitai
Version: 0.1.4
Summary: Long-term memory for AI Agents
Project-URL: Homepage, https://contextkit.outhad.com
Project-URL: Documentation, https://contextkit.outhad.com/docs/
Project-URL: Changelog, https://contextkit.outhad.com/changelog/
Project-URL: Issues, https://github.com/tanzil7890/contextkit-site/issues
Author-email: Outhad_Contextkit <idrisitanzil@gmail.com>
License-File: LICENSE
Requires-Python: <4.0,>=3.9
Requires-Dist: cryptography>=41.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: openai>=1.33.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: posthog>=3.5.0
Requires-Dist: pydantic>=2.7.3
Requires-Dist: pynacl>=1.5.0
Requires-Dist: pytz>=2024.1
Requires-Dist: qdrant-client>=1.9.1
Requires-Dist: sqlalchemy>=2.0.31
Provides-Extra: cloud
Requires-Dist: aiosqlite>=0.19.0; extra == 'cloud'
Requires-Dist: alembic>=1.13.0; extra == 'cloud'
Requires-Dist: asyncpg>=0.29.0; extra == 'cloud'
Requires-Dist: httpx>=0.27.0; extra == 'cloud'
Requires-Dist: pydantic-settings>=2.0.0; extra == 'cloud'
Requires-Dist: python-jose[cryptography]>=3.3.0; extra == 'cloud'
Requires-Dist: sqlalchemy[asyncio]>=2.0.31; extra == 'cloud'
Requires-Dist: svix>=1.20.0; extra == 'cloud'
Provides-Extra: context-graph-kafka
Requires-Dist: confluent-kafka>=2.3; extra == 'context-graph-kafka'
Provides-Extra: dev
Requires-Dist: isort>=5.13.2; extra == 'dev'
Requires-Dist: pytest>=8.2.2; extra == 'dev'
Requires-Dist: ruff>=0.6.5; extra == 'dev'
Provides-Extra: extras
Requires-Dist: boto3>=1.34.0; extra == 'extras'
Requires-Dist: elasticsearch>=8.0.0; extra == 'extras'
Requires-Dist: langchain-community>=0.0.0; extra == 'extras'
Requires-Dist: langchain-memgraph>=0.1.0; extra == 'extras'
Requires-Dist: opensearch-py>=2.0.0; extra == 'extras'
Requires-Dist: sentence-transformers>=5.0.0; extra == 'extras'
Provides-Extra: graph
Requires-Dist: langchain-aws>=0.2.23; extra == 'graph'
Requires-Dist: langchain-neo4j>=0.4.0; extra == 'graph'
Requires-Dist: neo4j>=5.23.1; extra == 'graph'
Requires-Dist: rank-bm25>=0.2.2; extra == 'graph'
Provides-Extra: llms
Requires-Dist: google-genai>=1.0.0; extra == 'llms'
Requires-Dist: google-generativeai>=0.3.0; extra == 'llms'
Requires-Dist: groq>=0.3.0; extra == 'llms'
Requires-Dist: litellm>=0.1.0; extra == 'llms'
Requires-Dist: ollama>=0.1.0; extra == 'llms'
Requires-Dist: together>=0.2.10; extra == 'llms'
Requires-Dist: vertexai>=0.1.0; extra == 'llms'
Provides-Extra: multimodal
Requires-Dist: laion-clap>=1.1.0; extra == 'multimodal'
Requires-Dist: torch>=2.0.0; extra == 'multimodal'
Requires-Dist: torchvision>=0.15.0; extra == 'multimodal'
Requires-Dist: transformers>=4.35.0; extra == 'multimodal'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.7; extra == 'test'
Requires-Dist: pytest-mock>=3.14.0; extra == 'test'
Requires-Dist: pytest>=8.2.2; extra == 'test'
Provides-Extra: vector-stores
Requires-Dist: azure-search-documents>=11.4.0b8; extra == 'vector-stores'
Requires-Dist: chromadb>=0.4.24; extra == 'vector-stores'
Requires-Dist: faiss-cpu>=1.7.4; extra == 'vector-stores'
Requires-Dist: pinecone-text>=0.10.0; extra == 'vector-stores'
Requires-Dist: pinecone<=7.3.0; extra == 'vector-stores'
Requires-Dist: pymochow>=2.2.9; extra == 'vector-stores'
Requires-Dist: pymongo>=4.13.2; extra == 'vector-stores'
Requires-Dist: upstash-vector>=0.1.0; extra == 'vector-stores'
Requires-Dist: vecs>=0.4.0; extra == 'vector-stores'
Requires-Dist: weaviate-client>=4.4.0; extra == 'vector-stores'
Description-Content-Type: text/markdown

# Outhad_ContextKit: Intelligent Memory Layer for AI Applications

<div align="center">

**Long-term memory for AI agents: graph, timeline and personalised recall in one library**

[![PyPI version](https://badge.fury.io/py/outhad_contextkitai.svg)](https://pypi.org/project/outhad-contextkitai/)
[![Cloud SDK](https://img.shields.io/pypi/v/outhad-contextkit-sdk?label=cloud%20sdk)](https://pypi.org/project/outhad-contextkit-sdk/)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)

[Quick Start](#quick-start) • [Features](#core-features) • [Cloud](#hosted-cloud) • [API Reference](#memory-api-reference) • [Examples](#examples) • [Docs](#documentation)

</div>

---

## Why Outhad_ContextKit?

Traditional AI applications send the **entire conversation history** to the LLM on every request. The result:

- **Growing cost**: every turn re-sends every previous turn
- **Slow responses**: the model re-reads the whole history each time
- **Privacy risk**: sensitive details are repeatedly sent to the LLM provider

**Outhad_ContextKit solves this** by storing facts once and retrieving only what matters:

```python
# ❌ Traditional approach: sends ALL messages, every time
response = openai.chat.completions.create(
    model="gpt-4o",
    messages=conversation_history,  # grows with every turn
)

# ✅ Outhad_ContextKit: sends ONLY relevant memories
relevant_memories = memory.search(query=message, user_id="alice", limit=3)
response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[system_prompt_with_memories, user_message],  # a handful of facts
)
```

### What's inside

| Feature | What it does | Default |
|---|---|---|
| [**TCMGM**](#1-tcmgm-temporal-causal-multimodal-graph-memory) | Episodic memory: dated events and cause → effect links ("when?" and "why?") | On when a graph store is set (`tcmgm`) |
| [**Context-Graph Layer**](#2-context-graph-layer-cgl) | Memories linked as a graph; graph-first retrieval | Off |
| [**MSPR**](#3-multi-stage-personalized-retrieval-mspr) | Ranking that adapts to intent, usage and feedback | Off |
| [**Tenants**](#4-tenant--sub-tenant-architecture) | Isolated memory per customer, department or project | Off |
| [**Decay & Versioning**](#5-time-aware-decay--versioning) | Memories age, version, roll back and archive | Off |
| [**Adaptive Chunking**](#6-adaptive-chunking) | Chunks long documents, keeps short ones whole | Off |
| [**PPMF**](#7-ppmf-privacy-preserving-memory-firewall) | Detect, redact and encrypt sensitive data | Off |
| [**Multimodal**](#8-multimodal-memory) | Images, audio and text in one memory | Off |

Every opt-in layer is a **guarded no-op when disabled**: turning a feature off leaves existing behaviour exactly as it was.

---

## Quick Start

**New to Outhad_ContextKit?** Choose your path:

- 🚀 **[5-Minute Quick Start](./docs/setup/QUICK_START_NO_INFRASTRUCTURE.md)**: no infrastructure needed
- 🔧 **[Full Setup Guide](./docs/setup/)**: production-ready with all features
- 📚 **[Troubleshooting](./docs/TROUBLESHOOTING.md)**: common issues and solutions

### Installation

```bash
pip install outhad_contextkitai
```

**Optional dependencies:**
```bash
pip install "outhad_contextkitai[graph]"          # Neo4j / Neptune graph memory (TCMGM)
pip install "outhad_contextkitai[vector_stores]"  # Pinecone, Weaviate, Chroma, FAISS, ...
pip install "outhad_contextkitai[llms]"           # Groq, Together, Ollama, Gemini, ...
pip install "outhad_contextkitai[multimodal]"     # Image + audio embeddings (CLIP, CLAP)
pip install "outhad_contextkitai[extras]"         # Bedrock, S3, Elasticsearch, OpenSearch, Memgraph
pip install networkx scipy                        # Context-Graph Layer (scipy powers PageRank retrieval)
pip install tiktoken                              # Token-based adaptive chunking
```

### Prerequisites

- **Python 3.9+**
- **An LLM API key**: OpenAI by default (`export OPENAI_API_KEY="sk-..."`); Anthropic, Gemini, Groq and [more](#9-multi-provider-support) work via config
- **Neo4j** *(optional)*: only for TCMGM and the CGL `neo4j` backend

```bash
docker run --name neo4j -p 7474:7474 -p 7687:7687 -e NEO4J_AUTH=neo4j/password neo4j:latest
```

**Without Neo4j:** ✅ vector memory · ✅ Context-Graph Layer · ✅ MSPR · ✅ Tenants · ✅ Decay & Versioning · ✅ Chunking · ✅ PPMF

**With Neo4j:** all of the above, ✅ plus temporal reasoning · ✅ causal analysis · ✅ entity graphs · ✅ fused retrieval

### 60-Second Setup

```python
from openai import OpenAI
from outhad_contextkit import Memory

openai_client = OpenAI()
memory = Memory()

# Add memories from conversations
memory.add([
    {"role": "user", "content": "I'm allergic to peanuts"},
    {"role": "assistant", "content": "I'll remember that!"},
], user_id="alice")

# Search relevant memories
relevant_memories = memory.search("What are my dietary restrictions?", user_id="alice", limit=3)

# Use memories in LLM context
response = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": f"User memories: {relevant_memories['results']}"},
        {"role": "user", "content": "What should I avoid eating?"},
    ],
)
print(response.choices[0].message.content)
# "Based on your allergy, you should avoid peanuts and foods containing them."
```

---

## Core Features

### 1. TCMGM (Temporal-Causal-Multimodal Graph Memory)

**Episodic memory**: ask *when* and *why*, not just *what*.

With a Neo4j graph store, conversations become dated events linked by cause → effect. List them in order, trace an event back to its causes, or search them together with vectors and entities:

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "graph_store": {
        "provider": "neo4j",
        "config": {"url": "bolt://localhost:7687", "username": "neo4j", "password": "password"},
    },
    "tcmgm": {"build_timeline_on_add": True},  # also store events on every add()
})

memory.add([
    {"role": "user", "content": "I deployed the new API"},
    {"role": "user", "content": "Response times doubled because of the deploy"},
], user_id="dev_team")

events = memory.get_timeline(user_id="dev_team")  # oldest first
causes = memory.get_causal_chain(events[-1]["event_id"], user_id="dev_team")  # "why?"

results = memory.search(
    "When did the performance issue start?",
    user_id="dev_team",
    use_tcmgm=True,  # one ranking over vector + entity graph + timeline
)
```

**What TCMGM adds:**
- **Timelines**: `build_timeline()` (or `tcmgm.build_timeline_on_add`) stores each event with its time; `get_timeline()` lists them
- **Causes**: links typed `caused_by`, `leads_to`, `enables`, `prevents`; `get_causal_chain()` walks back ("why?") or forward ("what next?")
- **Fused Retrieval**: `search(use_tcmgm=True, time_window=...)` returns `fused_ranking` instead of `results`
- **Cost Control**: `tcmgm.causal_extraction_mode` is `"llm"` (default), `"cascade"`, or `"rules"` (no LLM call)

Timelines and causal links need the `neo4j` graph store. Event times start at `session_start` (default: now); dates written in the text are not parsed. Cross-modal results (`multimodal_results`) are not connected yet and come back empty.

**[Complete TCMGM Documentation →](./docs/core-concepts/TCMGM/)**

### 2. Context-Graph Layer (CGL)

**Every memory becomes a node**: memories link to each other, not just to entities.

Connects memories with typed edges and retrieves *through* the graph, so related context arrives with the best match:

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "context_graph": {"enabled": True, "backend": "networkx"},  # or "neo4j" for > 100k memories
})

memory.add("I love pizza", user_id="u1", run_id="r1")
memory.add("Italian food is my favourite", user_id="u1", run_id="r1")

# Graph-first retrieval: seed → expand along edges → re-rank
results = memory.search("what should I eat?", user_id="u1", use_context_graph=True)
print(results["results"], results["subgraph"])

# Already have memories? Build the graph in one idempotent call
memory.backfill_context_graph(user_id="u1")  # {"nodes": 42, "edges": 117}
```

**What makes CGL powerful:**
- **Typed Edges**: replies, timeline order, topic similarity, document links and updates are linked automatically; opt-in LLM edges add `SUPPORTS` / `CONTRADICTS` / `REFINES` / `ELABORATES`
- **Graph-First Retrieval**: BFS or Personalised PageRank, re-ranked by dense + keyword + graph scores
- **Natural Forgetting**: edges and nodes decay over time; steer them with `mark_relevant()` / `mark_irrelevant()`
- **Time Travel & Audit**: every change is logged. Snapshot it, replay to any point in time, or stream it to webhooks, SSE or Kafka

**[Complete CGL Documentation →](./docs/core-concepts/context-graph-layer.mdx)**

### 3. Multi-Stage Personalized Retrieval (MSPR)

**Search that learns from each user**: ranking adapts to intent, usage and feedback.

Layers personal signals on top of CGL, each behind its own switch so you can roll them out one at a time:

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "context_graph": {"enabled": True, "retrieval": {"delta_personal": 0.3, "zeta_success": 0.2}},
    "mspr": {"enabled": True, "feedback": {"enabled": True}, "success": {"enabled": True}},
})

memory.add("Q4 revenue forecast: $4.2M", user_id="u1")
results = memory.search("revenue forecast", user_id="u1", use_context_graph=True)

# Tell it what helped - future searches for this query rank it higher
memory.record_feedback(results["results"][0]["id"], helpful=True,
                       user_id="u1", query="revenue forecast")
```

**What makes MSPR powerful:**
- **Intent-Aware**: classifies each query and re-weights the ranking for that intent
- **Feedback & Frequency**: helpful / unhelpful verdicts and real usage feed a personal boost
- **Learns What Worked**: memories that answered the same query before get lifted
- **Role-Aware**: a `RolePolicy` filters what each caller may see (tenant isolation, role scopes)

New ranking weights default to `0.0`, so nothing changes until you opt in.

**[Complete MSPR Documentation →](./docs/core-concepts/multi-stage-personalized-retrieval.mdx)**

### 4. Tenant + Sub-Tenant Architecture

**Multi-tenant memory in one config line**: isolated per customer, department or project.

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "tenant": {"enabled": True, "isolation": {"mode": "collection"}},  # one collection per tenant
})

# Admin API
memory.tenant.create_tenant(id="acme", name="Acme Corp")
memory.tenant.create_sub_tenant(tenant_id="acme", name="finance", kind="department")
memory.tenant.assign_role(tenant_id="acme", principal="alice@acme.com", role="member")

# Tenant-scoped memory
memory.add("Q4 forecast: $4.2M", user_id="alice",
           metadata={"tenant_id": "acme", "sub_tenant_id": "finance"})
results = memory.search("revenue forecast", user_id="alice", filters={"tenant_id": "acme"})
```

**What makes Tenants powerful:**
- **Three Isolation Modes**: shared collection + filter, one collection per tenant, or one database per tenant *(reserved)*
- **Sub-Tenants & RBAC**: departments, projects or users under each tenant, with role bindings
- **Lifecycle Ops**: `export_tenant`, `import_tenant`, `migrate_tenant`, and GDPR-style hard delete
- **Zero New Infrastructure**: SQLite registry + your existing vector store; `backfill_tenant()` upgrades old data

**[Complete Tenant Documentation →](./docs/core-concepts/tenant-architecture.mdx)**

### 5. Time-Aware Decay & Versioning

**Memories that age, version and archive themselves**: built for long-lived production stores.

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "context_graph": {"enabled": True},
    "decay_v2": {
        "enabled": True,
        "versioning": {"mode": "immutable"},                      # enables versions() / rollback()
        "schedule": {"enabled": True, "interval_seconds": 1800},  # decay every 30 min
    },
})

memory.decay_score(memory_id)             # 0.0 (stale) → 1.0 (fresh)
memory.archive_low(threshold=0.05)        # archive what has gone stale

memory.versions(memory_id)                # version chain
memory.rollback(memory_id, to_version=2)  # restore an earlier version
```

**What makes Decay & Versioning powerful:**
- **Decay Score**: blends recency, access frequency and feedback into one number
- **Background Scheduler**: decays, archives and prunes old versions on an interval
- **Version Chain**: diff any two versions; a rollback is written as a new version
- **Cold Storage**: move stale memories to local disk or S3 and bring them back on demand

> **Note:** `Memory.update()` still edits a memory in place, so versions are currently added by `rollback()` only.

**[Complete Decay & Versioning Documentation →](./docs/core-concepts/time-aware-decay-and-versioning.mdx)**

### 6. Adaptive Chunking

**Smart document processing**: only chunks when needed.

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "chunking": {
        "enabled": True,
        "strategy": "token",        # or "character", "semantic", "hierarchical"
        "chunk_size": 512,
        "chunk_overlap": 50,
        "min_document_size": 1500,  # don't chunk short docs
        "merge_chunks_on_retrieval": True,
    }
})

# Short documents (< 1500 chars) → stored as a single memory
memory.add("Alice loves Italian food", user_id="alice")

# Long documents (≥ 1500 chars) → chunked into 512-token segments
memory.add(research_paper_text, user_id="researcher")

# Retrieval merges related chunks back together
results = memory.search("methodology section", user_id="researcher")
```

**Adaptive Chunking Benefits:**
- **Prevents Over-Fragmentation**: short texts keep their meaning intact
- **Context Preservation**: configurable overlap between chunks
- **Coherent Retrieval**: neighbouring chunks are merged back at search time
- **Four Strategies**: character, token, semantic, or hierarchical (structure-aware)

**[Complete Chunking Documentation →](./docs/CHUNKING_AND_RETRIEVAL/CHUNKING_USER_GUIDE.md)**

### 7. PPMF (Privacy-Preserving Memory Firewall)

**Privacy building blocks**: detect, redact and encrypt sensitive data.

```python
from outhad_contextkit.memory.privacy import (
    EncryptionAlgorithm, EncryptionManager, MemorySanitizer, PPMFConfig, RedactionMode, redact_text,
)

config = PPMFConfig(enabled=True, encryption_algorithm=EncryptionAlgorithm.AES_256_GCM)

# Detect + redact
text = "Email me at john@example.com or call 555-123-4567"
result = MemorySanitizer(config).classify(text)
print(redact_text(text, result.sensitive_spans, RedactionMode.REPLACE))
# "Email me at [PII_EMAIL] or call [PII_PHONE]"

# Encrypt at rest
encryption = EncryptionManager(config=config)
ciphertext, meta = encryption.encrypt("john@example.com")
encryption.decrypt(ciphertext, meta)  # "john@example.com"
```

**PPMF Features:**
- ✅ PII / PHI / financial / secret detection (fast rules + optional LLM classifier)
- ✅ Five redaction modes: mask, replace, encrypt, remove, hash
- ✅ Fernet / AES-256-GCM / NaCl encryption
- ✅ Telemetry sanitising + MEXTRA-style adversarial testing

> **Note:** `ppmf.enabled=True` does not yet run these automatically inside `Memory.add()` / `search()`. Call them directly as shown above.

**[Complete PPMF Documentation →](./docs/ppmf/README.md)**

### 8. Multimodal Memory

**Images alongside text**: a vision LLM turns pictures into searchable facts.

```python
from outhad_contextkit import Memory

memory = Memory.from_config({
    "llm": {"provider": "openai", "config": {"model": "gpt-4o", "enable_vision": True}},
})

memory.add([
    {"role": "user", "content": "This is our new product design"},
    {"role": "user", "content": {"type": "image_url", "image_url": {"url": "https://example.com/product.jpg"}}},
], user_id="design_team")

results = memory.search("product design with blue accents", user_id="design_team")
```

**Multimodal Features:**
- **Vision Ingest**: images are described by the LLM and stored as regular memories
- **Embedders**: CLIP / SigLIP (images) and CLAP (audio) embedders ship in `[multimodal]`; cross-modal search inside TCMGM is not connected yet
- **Opt-In Weight**: install `[multimodal]` only when you need it; nothing heavy loads otherwise

**[Complete Multimodal Documentation →](./docs/core-concepts/TCMGM/MULTIMODAL_SUPPORT_USAGE_GUIDE.md)**

### 9. Multi-Provider Support

**Flexible infrastructure**: use your preferred tools.

| Component | Supported Providers |
|-----------|---------------------|
| **LLMs** | OpenAI, Anthropic, Gemini, Azure OpenAI, AWS Bedrock, Groq, Together, Ollama, LiteLLM, DeepSeek, xAI, Sarvam, LM Studio, vLLM, LangChain |
| **Embedders** (10) | OpenAI, Azure OpenAI, Gemini, Vertex AI, HuggingFace, Ollama, Together, LM Studio, AWS Bedrock, LangChain |
| **Vector Stores** (16) | Qdrant (default), Chroma, pgvector, Milvus, Pinecone, Weaviate, FAISS, MongoDB, Redis, Elasticsearch, OpenSearch, Supabase, Upstash, Azure AI Search, Vertex AI, LangChain |
| **Graph Stores** (3) | Neo4j, Memgraph, AWS Neptune |

```python
# Example: Anthropic Claude + Qdrant + Neo4j
memory = Memory.from_config({
    "llm": {"provider": "anthropic", "config": {"model": "claude-haiku-4-5-20251001"}},
    "embedder": {"provider": "openai", "config": {"model": "text-embedding-3-small"}},
    "vector_store": {"provider": "qdrant", "config": {"host": "localhost", "port": 6333}},
    "graph_store": {"provider": "neo4j", "config": {
        "url": "bolt://localhost:7687", "username": "neo4j", "password": "password"}},
})
```

---

## Hosted Cloud

**Managed memory over HTTP**: the same engine with no infrastructure to run.

```bash
pip install outhad-contextkit-sdk
```

```python
from outhad_contextkit_sdk import MemoryClient  # AsyncMemoryClient has the same API

client = MemoryClient(api_key="ock_live_...")   # or set OUTHAD_CONTEXTKIT_API_KEY

client.add(messages=[{"role": "user", "content": "I love sushi."}], user_id="alice")
hits = client.search(query="What food do I like?", user_id="alice")

# Switch the LLM for a single call
client.search(query="What food do I like?", user_id="alice",
              llm={"provider": "anthropic", "model": "claude-haiku-4-5-20251001"})
```

**What the cloud adds:**
- **Every Engine Feature over HTTP**: TCMGM, CGL, MSPR, versioning, decay, feedback, pin / unpin
- **Projects & Teams**: projects, members, webhooks, environments, per-project model settings
- **Security**: scoped API keys, key rotation, IP allowlists, audit log
- **Observability**: usage, request log, top users, graph snapshot

**Self-host it:** the FastAPI control plane lives in [`server/cloud`](./server/cloud) and the Next.js dashboard in [`web/`](./web):

```bash
cd server && cp .env.example .env   # fill in DATABASE_URL, CLERK_*, OPENAI_API_KEY
make cloud-migrate && make cloud-up # API on http://localhost:8000
cd ../web && npm install && npm run dev
```

For production hosting (API image, Vercel, Clerk, DNS), see **[DEPLOYMENT.md](./DEPLOYMENT.md)**.

**[Complete SDK Documentation →](./sdk/python/outhad-contextkit-sdk/api_doc/)**

---

## Architecture

```text
                  Your application / agent framework
                    │                            │
          in-process│                            │ HTTPS + API key
                    ▼                            ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│ Memory / AsyncMemory          │◄──│ Cloud control plane           │
│ add · search · update · ...   │   │ server/cloud (FastAPI)        │
└───────────────┬───────────────┘   │ keys · projects · usage ·     │
                │                   │ audit log · webhooks          │
                │                   └───────────────────────────────┘
                ▼
┌───────────────────────────────────────────────────────────────────┐
│ Opt-in layers (all off by default)                                │
│   Retrieval   TCMGM · CGL (BFS / PPR) · MSPR                      │
│   Governance  Tenants · Decay & versioning · PPMF                 │
│   Ingest      Adaptive chunking · Multimodal                      │
└───────────────────────────────┬───────────────────────────────────┘
                                ▼
┌───────────────────────────────────────────────────────────────────┐
│ Storage                                                           │
│   Vector store (16 providers)   Graph store (Neo4j / Memgraph /   │
│   History (SQLite / Postgres)   Neptune)                          │
│   CGL graph (networkx / Neo4j)  Cold storage (local disk / S3)    │
│   SQLite sidecars: CGL changelog · feedback · query success ·     │
│   tenant registry                                                 │
└───────────────────────────────────────────────────────────────────┘
```

**[Complete Architecture Documentation →](./ARCHITECTURE.md)**

---

## Memory API Reference

All methods are available on both `Memory` and `AsyncMemory`. Click a group to expand it.

<details>
<summary><b>Core</b>: add, search, get, update, delete, history</summary>

| Method | Purpose |
|---|---|
| `add(messages, *, user_id, agent_id, run_id, metadata, infer=True)` | Extract facts with the LLM (or store raw with `infer=False`) |
| `search(query, *, user_id, limit, filters, threshold, ...)` | Semantic search; flags: `use_tcmgm`, `time_window`, `include_causal`, `use_context_graph`, `merge_chunks`, `include_archived` |
| `get(memory_id)` / `get_all(*, user_id, agent_id, run_id, filters, limit)` | Retrieve memories |
| `update(memory_id, data)` / `delete(memory_id)` / `delete_all(...)` | Modify with automatic history tracking |
| `history(memory_id, *, tenant_id, sub_tenant_id)` | Change history for a memory |
| `reset()` | Wipe the memory store |

</details>

<details>
<summary><b>TCMGM</b>: timelines and cause → effect links (neo4j graph store)</summary>

| Method | Purpose |
|---|---|
| `build_timeline(messages, *, user_id, agent_id, run_id, session_start)` | Store a conversation as dated events and causal links |
| `get_timeline(*, user_id, agent_id, run_id, time_window, limit)` | Events, oldest first |
| `get_causal_chain(event_id, *, user_id, direction, max_depth)` | Causes (`"backward"`, default) or effects (`"forward"`) of one event |

</details>

<details>
<summary><b>Context-Graph Layer</b>: backfill, decay, snapshot, replay, change feed</summary>

| Method | Purpose |
|---|---|
| `backfill_context_graph(*, batch_size, user_id, agent_id, run_id)` | Idempotent graph build from existing memories |
| `tick_decay()` / `reset_context_graph()` | One decay round / drop all graph nodes and edges |
| `mark_relevant(id, delta)` / `mark_irrelevant(id, delta)` | Manually adjust relevance |
| `snapshot_context_graph(path)` / `replay_context_graph(*, until)` | Snapshot and point-in-time replay |
| `subscribe_context_graph_changes(callback)` / `unsubscribe_context_graph_changes(token)` | Live change feed |

</details>

<details>
<summary><b>MSPR</b>: feedback, role policies, backfill</summary>

| Method | Purpose |
|---|---|
| `record_feedback(memory_id, *, helpful, user_id, query)` / `reset_feedback(*, user_id)` | Capture or wipe user verdicts |
| `set_role_policy(policy)` / `get_role_policy()` | Install or read the active `RolePolicy` |
| `backfill_mspr()` | Rebuild the success store from existing feedback |

</details>

<details>
<summary><b>Tenants</b>: admin, RBAC, export / import / migrate</summary>

| Method | Purpose |
|---|---|
| `tenant.create_tenant / list_tenants / get_tenant / update_tenant / delete_tenant(hard=)` | Tenant admin |
| `tenant.create_sub_tenant / list_sub_tenants / delete_sub_tenant` | Sub-tenant admin |
| `tenant.assign_role / revoke_role / list_roles_for / has_role` | RBAC |
| `backfill_tenant()` / `export_tenant()` / `import_tenant()` / `migrate_tenant()` | Upgrade and lifecycle |

</details>

<details>
<summary><b>Decay & Versioning</b>: scoring, scheduler, versions, cold storage</summary>

| Method | Purpose |
|---|---|
| `decay_score(id)` / `archive_low(*, threshold, dry_run)` | Score and archive stale memories |
| `start_decay_scheduler()` / `stop_decay_scheduler()` / `close()` | Background scheduler control |
| `versions(id)` / `latest_version(id)` / `diff_versions(old, new)` / `rollback(id, *, to_version)` | Version chain |
| `record_reference(id, *, strength)` | Mark a memory as actually used by the agent |
| `demote_to_cold(id)` / `promote_from_cold(id)` | Cold storage |
| `backfill_decay_v2()` | Recompute decay scores for existing data |

</details>

---

## Examples

### Chatbot with Persistent Memory

```python
from openai import OpenAI
from outhad_contextkit import Memory

openai_client = OpenAI()
memory = Memory()

def chat_with_memory(message: str, user_id: str) -> str:
    # Retrieve relevant memories
    memories = memory.search(query=message, user_id=user_id, limit=3)
    context = "\n".join(m["memory"] for m in memories["results"])

    # Generate response with memory context
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"User context: {context}"},
            {"role": "user", "content": message},
        ],
    )
    answer = response.choices[0].message.content

    # Store new conversation
    memory.add([
        {"role": "user", "content": message},
        {"role": "assistant", "content": answer},
    ], user_id=user_id)
    return answer

print(chat_with_memory("I love pizza", user_id="alice"))
print(chat_with_memory("What food do I like?", user_id="alice"))
# "You mentioned you love pizza!"
```

### Multi-Agent Shared Memory

```python
from outhad_contextkit import Memory

memory = Memory()

# Research agent stores findings
memory.add("Found 3 security vulnerabilities in the authentication module",
           agent_id="research_agent", run_id="audit_2026")

# Development agent retrieves findings
findings = memory.search("security issues", agent_id="research_agent", run_id="audit_2026")

# Both agents share knowledge
memory.add("Fixed SQL injection in login endpoint", agent_id="dev_agent", run_id="audit_2026")
```

**More runnable examples:**
- **[Context-Graph Layer](./examples/context_graph/)**: basics, backfill, decay, LLM edges, snapshot / replay, webhook audit
- **[MSPR](./examples/mspr/basic_usage.py)**: end-to-end personalised retrieval
- **[Versioning & decay](./examples/phase15/local_engine_surfaces.py)**: versions, cold storage, references
- **[Cloud SDK](./examples/cloud/)**: quickstart, provider switching, TCMGM over HTTP
- **[Multi-agent](./examples/multiagents/)** • **[Multimodal](./examples/multimodal-demo/)** • **[Graph DB](./examples/graph-db-demo/)** • **[Chunking](./examples/chunking_basic_usage.py)**

---

## Benchmarks & Evaluation

Benchmark harnesses live in [`evaluation/`](./evaluation/). Run them against your own data and models; the JSON results there are the source of truth.

```bash
python evaluation/run_locomo_eval.py                          # long-term conversational QA (LoCoMo)
python evaluation/run_coqa_eval.py                            # CoQA, with / without adaptive chunking
python evaluation/run_context_graph_eval.py --algorithm both  # graph-first vs vector; BFS vs PPR
python tools/cost_calculator.py --users 1000 --messages 50 --model gpt-4  # estimate your savings
```

---

## Documentation

| Topic | Guides |
|---|---|
| **Getting started** | [Quick start](./docs/setup/QUICK_START_NO_INFRASTRUCTURE.md) • [Setup](./docs/setup/README.md) • [Neo4j](./docs/setup/NEO4J_SETUP_GUIDE.md) • [Troubleshooting](./docs/TROUBLESHOOTING.md) |
| **Core operations** | [add](./docs/core-concepts/memory-operations/add.md) • [search](./docs/core-concepts/memory-operations/search.md) • [update](./docs/core-concepts/memory-operations/update.md) • [delete](./docs/core-concepts/memory-operations/delete.md) • [Memory types](./docs/core-concepts/Memory_TYPES.md) |
| **TCMGM** | [Retrieval](./docs/core-concepts/TCMGM/RETRIEVAL_ORCHESTRATOR_USER_GUIDE.md) • [Causal](./docs/core-concepts/TCMGM/CASUAL_RELATIONSHIP_USAGE_GUIDE.md) • [Temporal](./docs/core-concepts/TCMGM/TEMPORAL_ATTRIBUTES_USAGE_GUIDE.md) • [Timeline](./docs/core-concepts/TCMGM/TIMELINE_HELPER_USER_GUIDE.md) • [Multimodal](./docs/core-concepts/TCMGM/MULTIMODAL_SUPPORT_USAGE_GUIDE.md) |
| **Advanced layers** | [CGL](./docs/core-concepts/context-graph-layer.mdx) • [MSPR](./docs/core-concepts/multi-stage-personalized-retrieval.mdx) • [Tenants](./docs/core-concepts/tenant-architecture.mdx) • [Decay & versioning](./docs/core-concepts/time-aware-decay-and-versioning.mdx) |
| **Ingest & privacy** | [Chunking](./docs/CHUNKING_AND_RETRIEVAL/CHUNKING_USER_GUIDE.md) • [PPMF](./docs/ppmf/README.md) • [Flexible embedder](./docs/flexible_embedder/FLEXIBLE_EMBEDDER_GUIDE.md) • [Multimodal install](./docs/INSTALL_MULTIMODAL.md) |
| **Cloud** | [SDK reference](./sdk/python/outhad-contextkit-sdk/api_doc/) • [Deploying the cloud](./DEPLOYMENT.md) |
| **Project** | [Architecture](./ARCHITECTURE.md) • [Changelog](./CHANGELOG.md) • [Contributing](./CONTRIBUTING.md) |

---

## Contributing

**Quick Development Setup:**

```bash
git clone https://github.com/tanzil7890/Outhad_ContextKit.git
cd Outhad_ContextKit

hatch env create && make install  # install development dependencies
make test                         # run tests
make format && make lint          # format + lint with ruff
```

See [CONTRIBUTING.md](./CONTRIBUTING.md) for details.

## Citation

If you use Outhad_ContextKit in your research, please cite:

```bibtex
@software{outhad_contextkit2025,
  title = {Outhad_ContextKit: Intelligent Memory Layer for AI Applications},
  author = {Outhad Team},
  year = {2025},
  url = {https://github.com/tanzil7890/Outhad_ContextKit}
}
```

## License

Apache 2.0. See [LICENSE](LICENSE).
