Metadata-Version: 2.5
Name: longcache
Version: 0.1.0
Summary: Semantic similarity cache for LLM and RAG pipelines — cut repeat query costs by 40%+
Project-URL: Repository, https://github.com/ENDEVSOLS/LongCache
Project-URL: Documentation, https://endevsols.github.io/LongCache
Project-URL: Changelog, https://github.com/ENDEVSOLS/LongCache/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/ENDEVSOLS/LongCache/issues
Author-email: ENDEVSOLS <technology@endevsols.com>
License: MIT
License-File: LICENSE
Keywords: cache,cost-reduction,embeddings,faiss,langchain,langgraph,latency-optimization,llamaindex,llm,openai,rag,semantic-cache,sentence-transformers,vector-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: click>=8.1.0
Requires-Dist: faiss-cpu>=1.7.4
Requires-Dist: numpy<2.0.0,>=1.24.0
Requires-Dist: sentence-transformers>=2.3.0
Requires-Dist: torch
Provides-Extra: all
Requires-Dist: fastapi>=0.109.0; extra == 'all'
Requires-Dist: langchain-core>=0.3.0; extra == 'all'
Requires-Dist: langgraph>=0.2.0; extra == 'all'
Requires-Dist: llama-index-core>=0.10.0; extra == 'all'
Requires-Dist: openai>=1.0.0; extra == 'all'
Requires-Dist: redis<6.0.0,>=5.0.0; extra == 'all'
Requires-Dist: starlette>=0.36.0; extra == 'all'
Requires-Dist: uvicorn>=0.27.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: build>=1.1.0; extra == 'dev'
Requires-Dist: httpx>=0.27.0; extra == 'dev'
Requires-Dist: mypy>=1.9.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: twine>=5.0.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == 'docs'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3.0; extra == 'langchain'
Requires-Dist: langgraph>=0.2.0; extra == 'langchain'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.10.0; extra == 'llamaindex'
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == 'openai'
Provides-Extra: redis
Requires-Dist: redis<6.0.0,>=5.0.0; extra == 'redis'
Provides-Extra: server
Requires-Dist: fastapi>=0.109.0; extra == 'server'
Requires-Dist: starlette>=0.36.0; extra == 'server'
Requires-Dist: uvicorn>=0.27.0; extra == 'server'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/ENDEVSOLS/LongCache/main/assets/logo.png" alt="LongCache Logo" width="320"/>
</p>

<p align="center"><strong>Production-grade semantic similarity cache for LLM and RAG pipelines — cut repeat query costs by 40%+.</strong></p>

<p align="center">
<a href="https://pypi.org/project/longcache/"><img src="https://img.shields.io/pypi/v/longcache" alt="PyPI Version"></a>
<a href="https://pepy.tech/project/longcache"><img src="https://static.pepy.tech/badge/longcache" alt="Total Downloads"></a>
<a href="https://pepy.tech/project/longcache"><img src="https://static.pepy.tech/badge/longcache/month" alt="Monthly Downloads"></a>
<a href="https://github.com/ENDEVSOLS/LongCache/stargazers"><img src="https://img.shields.io/github/stars/ENDEVSOLS/LongCache?style=flat" alt="GitHub Stars"></a>
<a href="https://github.com/ENDEVSOLS/LongCache/actions/workflows/ci.yml"><img src="https://github.com/ENDEVSOLS/LongCache/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<img src="https://img.shields.io/pypi/pyversions/longcache" alt="Python Versions">
<a href="https://github.com/ENDEVSOLS/LongCache/blob/main/LICENSE"><img src="https://img.shields.io/github/license/ENDEVSOLS/LongCache" alt="License"></a>
</p>

<p align="center">
<a href="https://endevsols.github.io/LongCache/"><strong>📖 Documentation</strong></a> &nbsp;·&nbsp;
<a href="https://endevsols.github.io/LongCache/getting-started/quickstart/"><strong>Quick Start</strong></a> &nbsp;·&nbsp;
<a href="https://endevsols.github.io/LongCache/api-reference/"><strong>API Reference</strong></a> &nbsp;·&nbsp;
<a href="https://github.com/ENDEVSOLS/LongCache/blob/main/CHANGELOG.md"><strong>Changelog</strong></a>
</p>

> *"What is your refund policy?" costs $0.003 the first time.*
> *It should cost $0.0000 every time after that.*

**LongCache** is a production-grade semantic similarity cache for LLM and RAG pipelines. Unlike string-match caches, it understands that *"How do I return a product?"* and *"What's your return process?"* deserve the same cached answer — and serves it in milliseconds.

---

## Why LongCache?

| Problem | Solution |
|---------|----------|
| Duplicate queries burn 30–60% of your LLM budget | Semantic cache intercepts similar queries before they reach the model |
| GPTCache is unmaintained with 50+ dependencies | LongCache has 4 core deps and is actively maintained |
| String-match caches miss paraphrased queries | FAISS + BGE-small understands semantic meaning, not just exact tokens |
| Most caches are coupled to one LLM provider | Works with LangChain, LangGraph, LlamaIndex, OpenAI, or raw functions |
| Cross-model cache poisoning is a silent bug | Composite key scoping isolates entries per model, prompt, and temperature |

**Result: Cut repeat query costs by 40%+ in production RAG applications.**

---

## Quick Start

### Installation

```bash
# Core — SQLite backend + FAISS + BGE-small (no external services needed)
uv pip install longcache

# With LangChain + LangGraph integration
uv pip install longcache[langchain]

# With Redis backend for multi-instance production
uv pip install longcache[redis]

# Everything
uv pip install longcache[all]
```

> **Offline-friendly**: LongCache embeds queries locally. After the BGE-small model weights are downloaded on first run, no internet connection is required.

### 30-Second Example

```python
from longcache import LongCache

cache = LongCache()

@cache.cached
def my_rag_chain(query: str) -> str:
    return expensive_llm_call(query)   # Your retrieval + LLM pipeline

# First call: MISS → runs full pipeline (~2 000 ms)
result1 = my_rag_chain("What is your refund policy?")

# Second call — exact query: HIT → instant (~5 ms)
result2 = my_rag_chain("What is your refund policy?")

# Third call — paraphrase: HIT → instant
result3 = my_rag_chain("How do I get a refund?")
```

Async functions are detected automatically — no additional setup needed:

```python
@cache.cached(namespace="faq", ttl=7200)
async def my_async_chain(query: str) -> str:
    return await async_llm_call(query)
```

---

## How It Works

```
┌──────────────────────────────────────────────────────────────┐
│                        Query Lifecycle                       │
│                                                              │
│  User Query                                                  │
│       │                                                      │
│       ▼                                                      │
│  ┌───────────────────┐                                       │
│  │   LocalEmbedder   │  BAAI/bge-small-en-v1.5 — no API     │
│  │   (~5 ms, local)  │  call, runs fully offline            │
│  └────────┬──────────┘                                       │
│           │ 384-dim vector                                   │
│           ▼                                                  │
│  ┌───────────────────┐    HIT (similarity ≥ threshold)       │
│  │  SimilarityIndex  │ ─────────────────────────────────▶   │
│  │  FAISS + Backend  │         Return cached response        │
│  └────────┬──────────┘                                       │
│           │    MISS                                          │
│           ▼                                                  │
│  ┌───────────────────┐                                       │
│  │  Full RAG Chain   │  Retrieval + LLM (~1–3 s)            │
│  └────────┬──────────┘                                       │
│           │                                                  │
│           ▼                                                  │
│  ┌───────────────────┐                                       │
│  │   CacheWriter     │  Stores vector + response + TTL       │
│  │                   │  + namespace + tags + context         │
│  └───────────────────┘                                       │
└──────────────────────────────────────────────────────────────┘
```

1. **Intercept** — queries are checked before they reach the LLM
2. **Embed** — the query is embedded locally in ~5 ms (BGE-small, no API call)
3. **Search** — FAISS searches for cosine similarity against cached vectors
4. **HIT** (similarity ≥ threshold): return cached response instantly
5. **MISS**: run the full pipeline, cache the result, and return it

---

## Configuration

### CacheConfig

All settings are controlled via the `CacheConfig` dataclass:

```python
from longcache import LongCache, CacheConfig

config = CacheConfig(
    similarity_threshold=0.92,           # 0.0–1.0; higher = stricter matching
    ttl_seconds=3600,                    # Entry lifetime in seconds (0 = never expire)
    max_entries=10_000,                  # Max entries per namespace (LRU eviction)
    namespace="default",                 # Logical isolation group
    embedder_model="BAAI/bge-small-en-v1.5",  # Local embedding model
    backend="sqlite",                    # "sqlite" | "redis" | "async_redis" | "memory"
    db_path=".longcache/cache.db",       # SQLite path (auto-creates parent dirs)
    redis_url="redis://localhost:6379/0",# Redis connection URL
    track_stats=True,                    # Track hits, misses, and cost savings
    faiss_index_type="flat",             # "flat" (exact) | "hnsw" (approximate, faster at scale)
    hnsw_m=32,                           # HNSW connections per node
    hnsw_ef_search=64,                   # HNSW search depth
    # Composite key context (isolates entries per model/persona)
    model="",                            # LLM model identifier (e.g. "gpt-4o")
    system_prompt="",                    # System prompt string
    temperature=-1.0,                    # Sampling temperature (-1.0 = unset)
)

cache = LongCache(config)
```

### From a Dictionary

```python
config = CacheConfig.from_dict({
    "similarity_threshold": 0.90,
    "backend": "redis",
    "ttl_seconds": 7200,
})
```

### From Environment Variables

```bash
export LONGCACHE_SIMILARITY_THRESHOLD=0.90
export LONGCACHE_BACKEND=redis
export LONGCACHE_TTL_SECONDS=7200
export LONGCACHE_REDIS_URL=redis://prod-redis:6379/0
export LONGCACHE_MODEL=gpt-4o
```

```python
config = CacheConfig.from_env()   # Reads all LONGCACHE_* env vars automatically
cache = LongCache(config)
```

### Similarity Threshold Guide

| Threshold | Behavior | Recommended Use Case |
|-----------|----------|----------------------|
| 0.99 | Near-exact match only | Precise FAQ lookup |
| 0.95 | Very similar phrasing | Customer support |
| **0.92** | **Default — balanced** | **General RAG pipelines** |
| 0.85 | Looser matching | Exploratory search |
| 0.75 | Topic-level clustering | Broad domain matching |

**Tip:** Start at 0.92 and adjust based on your observed hit rate vs. false-positive rate.

---

## Features ✨

### Core

- ✅ **Semantic Similarity Cache:** Understands paraphrased queries — “How do I return a product?” hits the same cache entry as “What’s your return process?”
- ✅ **Local Embedding:** BAAI/bge-small-en-v1.5 runs entirely on-device (~5 ms per query, no API call, fully offline after first download)
- ✅ **`@cache.cached` Decorator:** One-line caching for any sync or async function — auto-detects coroutines
- ✅ **`@cache.cached_stream` Decorator:** Caches streaming generator responses (sync and async); yields live on MISS, replays from cache on HIT
- ✅ **Composite Cache Keys:** Scopes entries per `model`, `system_prompt`, and `temperature` via SHA-256 fingerprint — prevents cross-model cache poisoning
- ✅ **Tag-Based Invalidation:** Attach tags to entries at write time; selectively invalidate by tag without clearing entire namespaces
- ✅ **Per-Entry TTL:** Override TTL per `set()` call or decorator — supports never-expire (`ttl=0`) and short-lived entries
- ✅ **LRU Eviction:** Configurable `max_entries` per namespace; oldest entries evicted automatically when the limit is reached

### Reliability

- ✅ **Thread-Safe Statistics:** Hit/miss counts protected by a dedicated lock; safe under 8+ concurrent threads
- ✅ **Persistent Stats:** Hit/miss/savings data written to SQLite and survives process restarts
- ✅ **Rebuild-Safe FAISS Index:** Index is fully rebuilt after TTL sweeps or evictions — prevents stale vector accumulation
- ✅ **Background TTL Sweeper:** Daemon thread runs every 5 minutes to delete expired entries; shuts down cleanly on GC
- ✅ **Correct Miss Counting:** `get()`, `aget()`, and all decorators record misses accurately in all code paths
- ✅ **Config Validation:** `CacheConfig` validates all fields on construction with clear error messages

### Integrations & Backends

- ✅ **SQLite Backend:** Zero external dependencies, WAL-mode thread safety, persistent stats, default for all deployments
- ✅ **Redis Backend:** Shared cache across multiple instances/pods, atomic TTL via `EXPIRE` (sync)
- ✅ **Async Redis Backend:** Non-blocking Redis — safe inside `asyncio` event loops (FastAPI, etc.)
- ✅ **In-Memory Backend:** Zero I/O, ideal for unit tests and CI/CD pipelines
- ✅ **LangChain Integration:** `CachedLLM` and `CachedChain` — drop-in wrappers with sync, async, and streaming support
- ✅ **LangGraph Integration:** `CachedState` node for graph-based agents; async variants included
- ✅ **LlamaIndex Integration:** `CachedQueryEngine` wraps any LlamaIndex query engine
- ✅ **OpenAI Integration:** `CachedOpenAI` wraps the OpenAI client directly
- ✅ **FastAPI Middleware:** `LongCacheMiddleware` caches entire HTTP routes; `X-LongCache: HIT` header on cached responses
- ✅ **CLI:** `longcache stats`, `clear`, `inspect`, `export` — manage the cache without writing code

---

## Backends

### SQLite (Default — Zero External Dependencies)

```python
cache = LongCache(CacheConfig(
    backend="sqlite",
    db_path=".longcache/cache.db",   # Auto-creates parent directories
))
```

| | |
|---|---|
| **Pros** | Zero external deps, persistent, WAL-mode thread safety, stats survive restarts |
| **Cons** | Single-machine only |
| **Best for** | Development, single-server production, CLI tools |

### Redis (Production Multi-Instance)

```bash
uv pip install longcache[redis]
```

```python
cache = LongCache(CacheConfig(
    backend="redis",
    redis_url="redis://localhost:6379/0",
    max_entries=100_000,
))
```

| | |
|---|---|
| **Pros** | Shared across instances, atomic TTL via `EXPIRE`, highly scalable |
| **Cons** | Requires Redis; blocking I/O (not suited for `async` event loops) |
| **Best for** | Multi-instance Kubernetes deployments, microservices (sync) |

### Async Redis (Production Async Apps)

```python
cache = LongCache(CacheConfig(
    backend="async_redis",
    redis_url="redis://localhost:6379/0",
))
```

| | |
|---|---|
| **Pros** | Non-blocking — safe inside `async` event loops (FastAPI, etc.) |
| **Best for** | Fully async production deployments |

### In-Memory (Testing & CI)

```python
cache = LongCache(CacheConfig(backend="memory"))
```

| | |
|---|---|
| **Pros** | Fastest possible, zero I/O, no files created |
| **Cons** | Lost on process restart, single-process only |
| **Best for** | Unit tests, CI/CD pipelines, benchmarks |

---

## Integrations

### LangChain

```bash
uv pip install longcache[langchain]
```

#### `CachedLLM` — Wrap Any LLM

```python
from langchain_openai import ChatOpenAI
from longcache.integrations.langchain import CachedLLM

llm = CachedLLM(ChatOpenAI(model="gpt-4o"), cache=LongCache())

response = llm.invoke("What is the refund policy?")       # sync
response = await llm.ainvoke("What is the refund policy?") # async

for chunk in llm.stream("What is RAG?"):                  # sync streaming
    print(chunk, end="")

async for chunk in llm.astream("What is RAG?"):           # async streaming
    print(chunk, end="")
```

#### `CachedChain` — Wrap Any LCEL Chain

```python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from longcache.integrations.langchain import CachedChain

chain = ChatPromptTemplate.from_template("Answer: {question}") | ChatOpenAI() | StrOutputParser()
cached_chain = CachedChain(chain, cache=LongCache(), query_key="question")

result = cached_chain.invoke({"question": "What is RAG?"})
result = await cached_chain.ainvoke({"question": "What is RAG?"})
```

### LangGraph

```python
from langgraph.graph import StateGraph
from longcache import LongCache
from longcache.integrations.langgraph import CachedState

cache = LongCache()

def cache_node(state: dict) -> dict:
    return CachedState.check(state, cache, query_key="messages")

def llm_node(state: dict) -> dict:
    response = my_llm_call(state["messages"])
    CachedState.save(state, response, cache)
    return {"response": response}

def should_use_llm(state: dict) -> str:
    return "llm" if not state.get("__cache_hit") else "end"

graph = StateGraph(dict)
graph.add_node("cache", cache_node)
graph.add_node("llm", llm_node)
graph.add_conditional_edges("cache", should_use_llm)
graph.set_entry_point("cache")
```

Async variants: `CachedState.acheck()` and `CachedState.asave()`.

### LlamaIndex

```bash
uv pip install longcache[llamaindex]
```

```python
from llama_index.core import VectorStoreIndex
from longcache.integrations.llamaindex import CachedQueryEngine

engine = VectorStoreIndex.from_documents(documents).as_query_engine()
cached_engine = CachedQueryEngine(engine, cache=LongCache())
response = cached_engine.query("What is the refund policy?")
```

### OpenAI (Direct)

```bash
uv pip install longcache[openai]
```

```python
from openai import OpenAI
from longcache.integrations.openai import CachedOpenAI

client = CachedOpenAI(OpenAI(api_key="sk-..."), cache=LongCache())
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is RAG?"}],
)
```

---

## FastAPI Middleware

```bash
uv pip install longcache[server]
```

```python
from fastapi import FastAPI
from longcache import LongCache
from longcache.middleware import LongCacheMiddleware

app = FastAPI()
app.add_middleware(
    LongCacheMiddleware,
    cache=LongCache(),
    cached_routes={"/v1/chat/completions"},
)

# All POST requests to /v1/chat/completions are now semantically cached.
# X-LongCache: HIT  →  cached response served
# X-LongCache: (absent) →  fresh LLM response
```

### Streaming Cache Adapter

```python
from longcache.middleware import StreamingCacheAdapter

def my_stream(query: str):
    for chunk in llm.stream(query):
        yield chunk

adapter = StreamingCacheAdapter(cache, my_stream)

# MISS — streams from LLM, saves full response in cache
for token in adapter.stream("What is RAG?"):
    print(token, end="")

# HIT — replays cached response with streaming pacing
for token in adapter.stream("What is RAG?"):
    print(token, end="")
```

---

## CLI

LongCache ships a command-line tool for cache management and inspection:

```bash
# View cache statistics (hits, misses, hit rate, estimated savings)
longcache stats
longcache stats --namespace faq

# Clear a namespace
longcache clear --namespace faq --yes

# Clear the entire cache
longcache clear --all --yes

# Inspect the nearest cached entry for a query
longcache inspect --query "refund policy"

# Export cache entries as JSON
longcache export --format json
longcache export --namespace faq -o faq_cache.json
```

---

## API Reference

### `LongCache(config=None)`

Main facade class. Accepts an optional `CacheConfig`; uses production-safe defaults otherwise.

#### Core Methods

| Method | Description |
|--------|-------------|
| `get(query, namespace, model, system_prompt, temperature)` | Check cache. Returns cached response or `None`. Increments miss count on MISS. |
| `set(query, response, namespace, ttl, tags, model, system_prompt, temperature)` | Store a query-response pair with optional tags and context scope. |
| `aget(...)` | Async version of `get()`. Non-blocking. |
| `aset(...)` | Async version of `set()`. Non-blocking. |
| `cached(func, *, namespace, ttl, tags, model, system_prompt, temperature)` | Decorator for sync **and** async functions. |
| `cached_stream(func, *, namespace, ttl, tags, model, system_prompt, temperature)` | Decorator for sync/async **generator** functions (streaming LLMs). |
| `invalidate(namespace, model, system_prompt, temperature)` | Delete all entries in a namespace. Returns count deleted. |
| `invalidate_by_tag(tag, namespace, ...)` | Delete all entries carrying a given tag. Returns count deleted. |
| `clear()` | Clear the entire cache across all namespaces. Returns count. |
| `stats()` | Returns hit rate, miss count, estimated tokens saved, estimated cost saved, and total entries. |
| `inspect(query, namespace, ...)` | Find nearest cached entry without returning it. Useful for debugging. |
| `record_hit()` | Manually record a cache hit (thread-safe). For custom integrations. |
| `record_miss(elapsed_ms)` | Manually record a cache miss with elapsed generation time (thread-safe). |

#### `@cache.cached` Decorator Options

```python
@cache.cached(
    namespace="faq",         # Logical isolation group
    ttl=7200,                # Entry lifetime override (seconds)
    tags=["billing"],        # Tags for selective invalidation
    model="gpt-4o",          # Scopes entries to this model
    system_prompt="...",     # Scopes entries to this system prompt
    temperature=0.0,         # Scopes entries to this temperature
)
def my_chain(query: str) -> str:
    ...
```

Works identically for `@cache.cached_stream`.

#### `stats()` Return Value

```python
{
    "hits":                    420,
    "misses":                  180,
    "hit_rate":                0.70,
    "total_saved_ms":          840_000.0,
    "estimated_tokens_saved":  630_000,
    "estimated_cost_saved_usd": 18.90,
    "total_entries":           95,
}
```

Statistics are protected by a reentrant threading lock and persisted to the SQLite backend — they survive process restarts.

---

### `CacheConfig` Field Reference

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `similarity_threshold` | `float` | `0.92` | Cosine similarity threshold for cache hits (0.0–1.0) |
| `ttl_seconds` | `int` | `3600` | Entry lifetime in seconds. `0` = never expire |
| `max_entries` | `int` | `10_000` | Max entries per namespace (LRU eviction) |
| `namespace` | `str` | `"default"` | Default namespace |
| `embedder_model` | `str` | `"BAAI/bge-small-en-v1.5"` | Local sentence-transformers model |
| `backend` | `str` | `"sqlite"` | `"sqlite"`, `"redis"`, `"async_redis"`, `"memory"` |
| `db_path` | `str` | `".longcache/cache.db"` | SQLite database path |
| `redis_url` | `str` | `"redis://localhost:6379/0"` | Redis connection URL |
| `track_stats` | `bool` | `True` | Track hit/miss stats (thread-safe, persisted in SQLite) |
| `faiss_index_type` | `str` | `"flat"` | `"flat"` (exact) or `"hnsw"` (approximate) |
| `hnsw_m` | `int` | `32` | HNSW connections per node (higher = more accurate, more memory) |
| `hnsw_ef_search` | `int` | `64` | HNSW search depth (higher = more accurate, slower) |
| `model` | `str` | `""` | Global model identifier for composite key scoping |
| `system_prompt` | `str` | `""` | Global system prompt for composite key scoping |
| `temperature` | `float` | `-1.0` | Global temperature for composite key scoping (`-1.0` = unset) |

---

### `CacheEntry` Field Reference

Returned by `inspect()` and used internally by all backends.

| Field | Type | Description |
|-------|------|-------------|
| `id` | `int \| None` | Backend-assigned unique identifier |
| `namespace` | `str` | Effective namespace (includes composite key suffix if set) |
| `query` | `str` | Original query text |
| `vector` | `np.ndarray` | 384-dim L2-normalized float32 embedding |
| `response` | `Any` | Cached LLM response (must be JSON-serializable) |
| `created_at` | `float` | Unix timestamp of entry creation |
| `expires_at` | `float \| None` | Unix timestamp of expiry (`None` = no expiry) |
| `hit_count` | `int` | Number of times this entry has been served |
| `model` | `str` | Model identifier at write time |
| `system_prompt` | `str` | System prompt at write time |
| `temperature` | `float` | Temperature at write time (`-1.0` = unset) |
| `tags` | `list[str]` | Tags for selective invalidation |

---

## Advanced Usage

### Per-Namespace Threshold Tuning

```python
faq_cache = LongCache(CacheConfig(namespace="faq",     similarity_threshold=0.95))
chat_cache = LongCache(CacheConfig(namespace="general", similarity_threshold=0.85))
```

### Per-Entry TTL Override

```python
cache.set("volatile_query", response, ttl=300)      # 5-minute TTL
cache.set("stable_query",   response, ttl=86400)    # 24-hour TTL
cache.set("permanent",      response, ttl=0)        # Never expires
```

### Cache Invalidation Patterns

```python
# Clear a specific namespace
cache.invalidate("faq")

# Clear entries scoped to a specific model
cache.invalidate("faq", model="gpt-4o")

# Invalidate by tag (e.g., after a document update)
cache.invalidate_by_tag("doc:refund_v2")

# Nuclear option — clear everything
cache.clear()
```

### Async Manual Control

```python
cached = await cache.aget("What is your refund policy?")
if cached is None:
    result = await my_pipeline("What is your refund policy?")
    await cache.aset("What is your refund policy?", result, tags=["policy"])
```

### HNSW Index for Large Caches (100K+ Entries)

```python
cache = LongCache(CacheConfig(
    faiss_index_type="hnsw",
    hnsw_m=32,           # More connections → more accurate, more memory
    hnsw_ef_search=64,   # Higher → more accurate, slower
    max_entries=500_000,
))
```

### Inspecting the Cache

```python
entry = cache.inspect("refund policy")
# {
#   'id': 42,
#   'query': 'What is your refund policy?',
#   'namespace': 'default',
#   'hit_count': 7,
#   'tags': ['billing', 'policy'],
#   'model': 'gpt-4o',
#   'created_at': 1700000000.0,
#   'expires_at': None,
#   'is_expired': False,
# }
```

### Custom Integration: Manual Hit/Miss Recording

For custom wrappers that call `get()`/`set()` manually:

```python
result = cache.get(query, track_miss=False)
if result is not None:
    cache.record_hit()
    return result

t0 = time.time()
result = my_pipeline(query)
cache.record_miss(elapsed_ms=(time.time() - t0) * 1000)
cache.set(query, result)
return result
```

---

## Architecture

### Component Overview

```
src/longcache/
├── __init__.py              # Public API: LongCache, CacheConfig
├── config.py                # CacheConfig dataclass + from_dict/from_env
├── cli.py                   # CLI: stats, clear, inspect, export
├── middleware.py             # FastAPI/Starlette ASGI middleware + StreamingCacheAdapter
├── core/
│   ├── entry.py             # CacheEntry dataclass (query, vector, response, tags, context)
│   ├── embedder.py          # LocalEmbedder — BGE-small singleton, lazy-loaded
│   ├── index.py             # SimilarityIndex — FAISS Flat/HNSW, rebuild-safe
│   └── cache.py             # LongCache — main orchestrator, thread-safe stats
├── backends/
│   ├── base.py              # AbstractCacheBackend ABC
│   ├── sqlite.py            # SQLiteCacheBackend — WAL mode, tag table, stats persistence
│   ├── redis.py             # RedisCacheBackend — sync, tag support, atomic TTL
│   ├── async_redis.py       # AsyncRedisCacheBackend — async, non-blocking
│   └── memory.py            # InMemoryCacheBackend — testing, zero I/O
└── integrations/
    ├── langchain.py         # CachedLLM, CachedChain
    ├── langgraph.py         # CachedState
    ├── llamaindex.py        # CachedQueryEngine
    └── openai.py            # CachedOpenAI
```

### Embedding Model

LongCache uses **BAAI/bge-small-en-v1.5** by default:

- **384-dimensional** vectors — compact and fast
- **Sub-5ms** embedding on modern CPU hardware
- **Fully offline** after initial weight download — no API calls ever
- **L2-normalized** — inner product equals cosine similarity in FAISS
- **Singleton pattern** — the model is loaded once and reused across all instances

### FAISS Index

| Index | Config Value | Characteristics |
|-------|-------------|-----------------|
| `IndexFlatIP` | `"flat"` | Exact search, deterministic, best for ≤ 100K entries |
| `IndexHNSWFlat` | `"hnsw"` | Approximate search, sub-ms at any scale, best for 100K+ entries |

The FAISS index is **rebuild-safe**: after TTL sweeps, namespace invalidations, or evictions, the index is fully rebuilt from the backend rather than incrementally patched — preventing stale vector accumulation.

### Thread Safety

- **Stats** (`hits`, `misses`, `total_saved_ms`) are protected by a dedicated `threading.Lock()`.
- **SQLite backend** uses WAL mode and a per-connection `threading.RLock()`.
- **The TTL sweeper** runs as a daemon thread and signals shutdown cleanly via `threading.Event`.
- **`get()`/`aget()`** correctly record misses in a thread-safe manner.

### Background TTL Sweeper

A lightweight daemon thread runs every 5 minutes to delete expired entries and rebuild the FAISS index. Starts automatically when `ttl_seconds > 0`. Shuts down gracefully when the `LongCache` instance is garbage-collected.

### Persistent Statistics

Hit/miss statistics are persisted to the SQLite backend and survive process restarts. All stat mutations go through a reentrant lock to prevent race conditions under concurrent load.

---

## Performance

### Expected Cache Hit Rates

| Scenario | Hit Rate | Cost Savings |
|----------|----------|--------------|
| FAQ chatbot | 60–80% | 40–60% |
| Customer support | 40–60% | 25–40% |
| Internal knowledge base | 50–70% | 30–50% |
| General RAG pipeline | 30–50% | 20–35% |

### Latency Comparison

| Operation | Typical Latency |
|-----------|----------------|
| Full RAG pipeline (retrieval + LLM) | 1,000–3,000 ms |
| LongCache HIT (embed + FAISS search) | 5–15 ms |
| LongCache MISS overhead | 5–10 ms |

---

## Project Structure

```
longcache/
├── pyproject.toml
├── README.md
├── src/
│   └── longcache/
│       ├── __init__.py
│       ├── config.py
│       ├── cli.py
│       ├── middleware.py
│       ├── core/
│       │   ├── __init__.py
│       │   ├── entry.py
│       │   ├── embedder.py
│       │   ├── index.py
│       │   └── cache.py
│       ├── backends/
│       │   ├── __init__.py
│       │   ├── base.py
│       │   ├── sqlite.py
│       │   ├── redis.py
│       │   ├── async_redis.py
│       │   └── memory.py
│       └── integrations/
│           ├── __init__.py
│           ├── langchain.py
│           ├── langgraph.py
│           ├── llamaindex.py
│           └── openai.py
├── tests/
│   ├── conftest.py
│   ├── test_embedder.py
│   ├── test_index.py
│   ├── test_cache.py
│   ├── test_backends.py
│   ├── test_langchain.py
│   ├── test_middleware.py
│   └── test_cli.py
└── examples/
    ├── basic_langchain.py
    ├── fastapi_middleware.py
    └── redis_production.py
```

---

## Dependencies

### Core (Required)

| Package | Version | Purpose |
|---------|---------|---------|
| `sentence-transformers` | `>=2.3.0` | Local embedding — BGE-small |
| `faiss-cpu` | `>=1.7.4` | Similarity search (Flat & HNSW) |
| `numpy` | `>=1.24.0,<2.0.0` | Vector math |
| `torch` | CPU (via uv) | sentence-transformers runtime |
| `click` | `>=8.1.0` | CLI framework |

> **Note**: `numpy<2.0.0` is required — numpy 2.0 breaks ABI compatibility with `faiss-cpu` and `torch`.

### Optional Extras

| Extra | Package(s) | Purpose |
|-------|-----------|---------|
| `[redis]` | `redis>=5.0.0,<6.0.0` | Redis & async Redis backends |
| `[langchain]` | `langchain-core>=1.0.0`, `langgraph>=1.0.0` | LangChain + LangGraph |
| `[llamaindex]` | `llama-index-core>=0.10.0` | LlamaIndex integration |
| `[openai]` | `openai>=1.0.0,<2.0.0` | Direct OpenAI integration |
| `[server]` | `fastapi>=0.109.0`, `starlette>=0.36.0`, `uvicorn>=0.27.0` | ASGI middleware |
| `[all]` | All of the above | Everything |

### Development

| Package | Version | Purpose |
|---------|---------|---------|
| `pytest` | `>=8.0.0` | Test runner |
| `pytest-asyncio` | `>=0.23.0` | Async test support |
| `pytest-cov` | `>=5.0.0` | Coverage reporting |
| `httpx` | `>=0.27.0` | ASGI test client |
| `ruff` | `>=0.4.0` | Linter / formatter |

---

## Running Tests

```bash
# Install in editable mode with dev dependencies
uv pip install -e ".[dev]"

# Run all tests
uv run pytest

# Run with coverage report
uv run pytest --cov=longcache --cov-report=term-missing

# Run a specific test file
uv run pytest tests/test_cache.py

# Verbose output
uv run pytest -v
```

---

## Contributing & Community

We welcome contributions! Please see:
- [CONTRIBUTING.md](CONTRIBUTING.md) — Setup, testing, and pull request guidelines.
- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — Contributor Covenant v2.1 standards.
- [SECURITY.md](SECURITY.md) — Vulnerability reporting and coordinated disclosure.

---

## License

This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.

Copyright (c) 2026 ENDEVSOLS
