Metadata-Version: 2.5
Name: memory-reuse
Version: 0.2.0
Summary: Execution cache layer for AI agents — cut LLM and tool call costs by reusing previous results.
Project-URL: Homepage, https://github.com/pranit-p/memory-reuse
Project-URL: Documentation, https://pranit-p.github.io/memory-reuse/
Project-URL: Repository, https://github.com/pranit-p/memory-reuse
Project-URL: Issues, https://github.com/pranit-p/memory-reuse/issues
Project-URL: Changelog, https://github.com/pranit-p/memory-reuse/blob/main/CHANGELOG.md
Author-email: Pranit <your-email@example.com>
Maintainer-email: Pranit <your-email@example.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,ai,cache,caching,cost-reduction,langgraph,litellm,llm,redis,semantic-cache
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: litellm>=1.40.0; extra == 'all'
Requires-Dist: numpy>=1.26; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: redis>=5.0.0; extra == 'all'
Requires-Dist: sentence-transformers>=3.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: black>=24.0; extra == 'dev'
Requires-Dist: hypothesis>=6.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.25; extra == 'docs'
Provides-Extra: litellm
Requires-Dist: litellm>=1.40.0; extra == 'litellm'
Provides-Extra: redis
Requires-Dist: redis>=5.0.0; extra == 'redis'
Provides-Extra: semantic
Requires-Dist: litellm>=1.40.0; extra == 'semantic'
Requires-Dist: numpy>=1.26; extra == 'semantic'
Requires-Dist: openai>=1.0; extra == 'semantic'
Provides-Extra: semantic-local
Requires-Dist: litellm>=1.40.0; extra == 'semantic-local'
Requires-Dist: numpy>=1.26; extra == 'semantic-local'
Requires-Dist: openai>=1.0; extra == 'semantic-local'
Requires-Dist: sentence-transformers>=3.0; extra == 'semantic-local'
Description-Content-Type: text/markdown

# memory-reuse

[![CI](https://github.com/pranit-p/memory-reuse/actions/workflows/test.yml/badge.svg)](https://github.com/pranit-p/memory-reuse/actions/workflows/test.yml)
[![PyPI](https://img.shields.io/pypi/v/memory-reuse.svg)](https://pypi.org/project/memory-reuse/)
[![Python](https://img.shields.io/pypi/pyversions/memory-reuse.svg)](https://pypi.org/project/memory-reuse/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

An execution cache layer for AI agents that cuts LLM and tool call costs by avoiding redundant computation. Drop it into any Python agent or LangGraph workflow with a single decorator.

- **Framework-agnostic** — LangGraph, LiteLLM, or any plain Python function.
- **Zero required dependencies** — the core runs on the standard library alone.
- **Safe by default** — per-user / per-session scoping prevents cross-user cache leaks.
- **Typed** — ships with `py.typed`, fully type-hinted.

---

## How it works

When your agent calls an LLM or a tool, `memory-reuse` hashes the inputs and
checks the cache first. On a hit it returns the stored result instantly — no
tokens spent, no API call made. On a miss it runs the real call and stores the
result for next time.

```
request ──► hash inputs ──► cache lookup
                              ├── HIT  ──► return cached result (0 cost)
                              └── MISS ──► run LLM/tool ──► store ──► return
```

> **Exact vs semantic:** by default `memory-reuse` does exact-match caching —
> identical inputs hit the cache. Enabling the optional
> [semantic cache](#semantic-cache) also serves cached results for
> *similar-but-not-identical* inputs (reworded questions) using embedding
> similarity.

---

## Install

Works with both **pip** and **uv** — pick whichever you use.

**pip**
```bash
pip install memory-reuse
```

**uv**
```bash
uv add memory-reuse
```

### Optional extras

| Extra | What it adds | pip | uv |
|---|---|---|---|
| `redis` | Redis backend support | `pip install memory-reuse[redis]` | `uv add memory-reuse[redis]` |
| `litellm` | LiteLLM cached wrappers | `pip install memory-reuse[litellm]` | `uv add memory-reuse[litellm]` |
| `semantic` | Semantic cache with **API** embeddings (OpenAI / LiteLLM) — no torch | `pip install memory-reuse[semantic]` | `uv add memory-reuse[semantic]` |
| `semantic-local` | Semantic cache with **local** embeddings (sentence-transformers, pulls in torch) | `pip install memory-reuse[semantic-local]` | `uv add memory-reuse[semantic-local]` |
| `all` | Everything above | `pip install memory-reuse[all]` | `uv add memory-reuse[all]` |

> **Note:** `uv` is a fast Python package manager. If you don't have it yet:
> `pip install uv` or see [docs.astral.sh/uv](https://docs.astral.sh/uv/getting-started/installation/)

---

## Quick start

```python
from memory_reuse import MemoryCache, CacheConfig
from memory_reuse.integrations import cached_tool

cache = MemoryCache()                          # in-memory backend, 1-hour TTL

@cached_tool(cache, scope="global", ttl=300)   # cache for 5 minutes
async def search_web(query: str) -> list[str]:
    return await my_search_api(query)          # only called on cache miss
```

---

## Usage patterns

### 1 — Basic exact cache (LLM responses)

```python
from memory_reuse import MemoryCache

cache = MemoryCache()

# Manual get/set
result = await cache.exact.get(["gpt-4", prompt], scope="global", scope_id=None)
if result is None:
    result = await llm.ainvoke(prompt)
    await cache.exact.set(["gpt-4", prompt], result, scope="global",
                          scope_id=None, ttl=3600)
```

### 2 — LangGraph node caching

```python
from memory_reuse.integrations import cached_node

@cached_node(cache, scope="user", key_fields=["messages"])
async def summarise(state: dict) -> dict:
    summary = await llm.ainvoke(state["messages"])
    return {"summary": summary}
```

The decorator reads `user_id` from the state dict automatically, or from
`cache.set_context(user_id=...)`.

### 3 — LangGraph tool caching

```python
from memory_reuse.integrations import cached_tool

@cached_tool(cache, scope="session", ttl=120)
async def fetch_user_profile(user_id: str) -> dict:
    return await db.get_user(user_id)
```

### 4 — LiteLLM (works with OpenAI, Claude, Bedrock, Groq, Ollama, and 100+ more)

```python
from memory_reuse.integrations import cached_litellm_completion, cached_litellm_embedding

# Completion — same prompt + model = cache hit, 0 tokens used
response = await cached_litellm_completion(
    cache,
    model="gpt-4o-mini",                  # swap for any LiteLLM model string
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    ttl=3600,
    scope="global",
)

# Embeddings — deterministic, safe to cache for 24 hours
embeddings = await cached_litellm_embedding(
    cache,
    model="text-embedding-3-small",
    input=["What is machine learning?"],
)
```

---

## Backend options

| Backend | Extra required | Persistence | Notes |
|---------|---------------|-------------|-------|
| `memory` | none | in-process only | LRU eviction, TTL support |
| `redis` | `[redis]` | yes | connection pool, lazy connect |

Configure via code or environment variables:

```bash
export MEMORY_REUSE_BACKEND=redis
export MEMORY_REUSE_REDIS_URL=redis://localhost:6379/0
export MEMORY_REUSE_DEFAULT_TTL=600
export MEMORY_REUSE_DEFAULT_SCOPE=user
```

```python
cache = MemoryCache.from_env()
```

---

## Multi-scope support

```python
cache.set_context(user_id="alice", session_id="sess-001")

# User-scoped: alice cannot see bob's cache
await cache.exact.get(["key"], scope="user", scope_id="alice")

# Session-scoped: isolated per conversation
await cache.tool.get("search", args, scope="session", scope_id="sess-001")

# Global: shared across all users — safe for public, stateless data
await cache.exact.get(["key"], scope="global", scope_id=None)
```

Using `scope="user"` without a `user_id` raises `ScopeViolationError` to
prevent accidental cross-user data leaks.

---

## Semantic cache

Exact caching only hits when inputs are *identical*. The semantic cache also
serves a cached result when a new query is **meaningfully similar** to a stored
one — so "What is 128 multiplied by 47?" can reuse the answer to "What is 128
times 47?". This lifts hit rates for natural-language workloads (chatbots, FAQ
agents, docs Q&A) where the same intent is phrased many ways.

### Choosing an embedding provider

The semantic cache turns text into vectors using one of three interchangeable
providers, selected by `embedding_provider`:

| Provider | `embedding_provider` | Runs | Install | Notes |
|---|---|---|---|---|
| **OpenAI** | `"openai"` | OpenAI API | `pip install memory-reuse[semantic]` | Hosted, paid per call. No torch. |
| **LiteLLM** | `"litellm"` | 100+ backends (Bedrock, Cohere, …) | `pip install memory-reuse[semantic]` | Model string picks the backend. No torch. |
| **Local** | `"local"` | Your machine | see below | sentence-transformers; private, no per-call cost; pulls in torch. |

There are just two install commands to remember. The `semantic` extra covers
**both API providers** (it bundles the small `openai` and `litellm` clients,
plus `numpy` as a cosine-similarity speedup) and installs **no torch**:

```bash
pip install "memory-reuse[semantic]"     # OpenAI + LiteLLM embeddings, lightweight
```

**Local embeddings** need `sentence-transformers`, which depends on **PyTorch**.
On a CPU-only machine (no NVIDIA GPU) install the **CPU torch wheel first** to
avoid a ~2 GB GPU/CUDA download — the CPU build is ~200 MB:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install "memory-reuse[semantic-local]"
```

With a GPU you can skip the first line and just `pip install
"memory-reuse[semantic-local]"`. Either way, the model weights themselves
(e.g. `all-MiniLM-L6-v2`, ~90 MB) download from Hugging Face on first use and
are then cached on disk for offline reuse.

> **Quieter / fully offline runs.** memory-reuse already silences the Hugging
> Face log chatter and the per-embedding progress bar. Once the model is
> cached, you can additionally skip Hugging Face's cache-validation HTTP checks
> by exporting `HF_HUB_OFFLINE=1` (and `TRANSFORMERS_OFFLINE=1`) in your own
> process. Do this only in your application — set after the first (downloading)
> run, never inside a shared library.

### Enabling it

Semantic caching is **off by default** — existing exact-only code is unchanged.
Turn it on via `CacheConfig` by setting `semantic_enabled=True` and choosing an
embedding provider (`local`, `openai`, or `litellm`):

```python
from memory_reuse import MemoryCache, CacheConfig

cache = MemoryCache(CacheConfig(
    backend="memory",
    semantic_enabled=True,
    embedding_provider="local",              # local sentence-transformers model
    embedding_model="all-MiniLM-L6-v2",
    similarity_threshold=0.95,               # how close is "close enough"
))

# Exact-first, then semantic. An exact hit never computes an embedding.
result = await cache.lookup(
    ["qa", "What is 128 multiplied by 47?"],
    query_text="What is 128 multiplied by 47?",
    scope="global", scope_id=None,
)
if result is None:
    result = await run_llm(...)              # only on a miss
    await cache.store(
        ["qa", "What is 128 times 47?"],
        query_text="What is 128 times 47?",
        value=result, scope="global", scope_id=None,
    )
```

Use `cache.lookup(...)` / `cache.store(...)` for the combined exact-then-semantic
flow. The exact cache is always tried first, so a semantic embedding is only
computed on an exact miss (no extra cost when an exact hit is available).

### Three ways to set the threshold

The `similarity_threshold` is a float in `[0.0, 1.0]`; a higher value demands a
closer match. You can set it three ways, from lowest to highest precedence:

1. **Config field** — the instance-wide default:

   ```python
   CacheConfig(semantic_enabled=True, embedding_provider="local",
               similarity_threshold=0.92)
   ```

2. **Environment variable** — read by `MemoryCache.from_env()`:

   ```bash
   export MEMORY_REUSE_SEMANTIC_ENABLED=true
   export MEMORY_REUSE_EMBEDDING_PROVIDER=local
   export MEMORY_REUSE_SIMILARITY_THRESHOLD=0.90
   ```

   ```python
   cache = MemoryCache.from_env()
   ```

3. **Per-call override** — passed to a single `lookup`, taking precedence over
   the config/env value for that call only:

   ```python
   await cache.lookup(key_parts, query_text="...", scope="global",
                      scope_id=None, threshold=0.98)
   ```

### Returning just the relevant answer (`extract_answer`)

A semantic hit returns the **whole** stored answer by default. If you asked
"Tell me about Python" and later ask "Who created Python?", the second query
matches the first and returns the entire paragraph — even though only one
sentence answers it.

Set `extract_answer=True` to have the cache return **only the sentence(s) that
best match the new question**:

```python
cache = MemoryCache(CacheConfig(
    semantic_enabled=True,
    embedding_provider="local",
    extract_answer=True,          # narrow the stored answer to the best sentence
    extract_min_similarity=0.5,   # confidence a sentence needs to be picked
))
```

Now "Who created Python?" returns just *"Python is a high-level, interpreted
programming language created by Guido van Rossum and first released in 1991."*
instead of the full paragraph.

How it works and its limits:

- **Purely extractive, no LLM.** It splits the stored answer into sentences,
  embeds each with the same model, and returns the sentence closest to the
  query. It never calls an LLM and never fabricates — it can only return text
  already present in the stored answer.
- **Falls back to the full answer** when no sentence clears
  `extract_min_similarity`, so you never get an empty result.
- **String answers only.** Non-string values (dicts, numbers) and
  single-sentence answers are returned unchanged.
- **Best-effort, not QA.** It returns a whole real sentence, so it can't reshape
  text into a crisp answer the way a model would. It is off by default.

### Choosing where to use semantic matching

Semantic matching compares *meaning*, so it shines for natural-language
questions where the same intent is phrased many ways — chatbots, FAQ agents,
docs Q&A, search. That is exactly where it saves the most.

It's a similarity match, though, so keep it to reads and questions rather than
correctness-critical commands. Two prompts can look close yet mean opposite
things — "cancel order 123" vs "confirm order 123" — so use the exact cache for
anything whose result depends on precise wording, especially actions with side
effects. memory-reuse gives you two simple levers to stay on the safe side:

- **Tune the threshold.** The default (`0.95`) favours precision — matches only
  fire when queries are very close. Raise it if you ever see a wrong match;
  lower it to trade some precision for a higher hit rate.
- **Opt a call out with `exact_only=True`.** For a sensitive call site, skip the
  semantic cache entirely regardless of the global config:

  ```python
  await cache.lookup(key_parts, query_text="...", scope="global",
                     scope_id=None, exact_only=True)   # exact match only
  ```

Used this way — similarity for questions, exact for commands — semantic caching
is both safe and a big hit-rate win.

### Latency and cost tradeoff

Enabling semantic caching adds an **embedding computation on every exact miss**.
That embedding costs time (local model inference or an API round-trip) and, for
hosted providers, money. The win is fewer full LLM calls when reworded queries
match; the cost is the embedding overhead on misses. Enable it when your
workload has many differently-worded but equivalent requests, so the saved LLM
calls outweigh the embedding cost. An exact hit short-circuits before any
embedding, so identical repeats stay as cheap as Phase 1.

---

## Configuration reference

All options live on `CacheConfig`. Every field can also be set from an
environment variable (read by `MemoryCache.from_env()`) where noted.

| Field | Type / accepted values | Default | Env var | Description |
|---|---|---|---|---|
| `backend` | `"memory"` \| `"redis"` | `"memory"` | `MEMORY_REUSE_BACKEND` | Storage backend. `redis` needs the `[redis]` extra. |
| `redis_url` | `str` \| `None` | `None` | `MEMORY_REUSE_REDIS_URL` | Redis connection URL. Required when `backend="redis"`. |
| `default_ttl` | `int > 0` \| `None` | `3600` | `MEMORY_REUSE_DEFAULT_TTL` (int or `"none"`) | Default entry TTL in seconds. `None` never expires. |
| `default_scope` | `"global"` \| `"user"` \| `"session"` | `"global"` | `MEMORY_REUSE_DEFAULT_SCOPE` | Scope used when none is passed explicitly. |
| `key_prefix` | `str` | `"memreuse"` | `MEMORY_REUSE_KEY_PREFIX` | Prefix prepended to every cache key. |
| `max_key_size` | `int > 0` | `512` | — | Max cache-key length in bytes. |
| `enable_stats` | `bool` | `True` | `MEMORY_REUSE_ENABLE_STATS` (`true`/`false`) | Track hit/miss/error counters. |
| `semantic_enabled` | `bool` | `False` | `MEMORY_REUSE_SEMANTIC_ENABLED` (`true`/`false`) | Turn on the semantic cache. Requires `embedding_provider`. |
| `similarity_threshold` | `float` in `[0.0, 1.0]` | `0.95` | `MEMORY_REUSE_SIMILARITY_THRESHOLD` | Minimum similarity to count as a match. Higher = stricter. |
| `embedding_provider` | `"openai"` \| `"local"` \| `"litellm"` \| `None` | `None` | `MEMORY_REUSE_EMBEDDING_PROVIDER` | Which embedding backend to use. Required when `semantic_enabled=True`. |
| `embedding_model` | `str` \| `None` | `None` (provider default) | `MEMORY_REUSE_EMBEDDING_MODEL` | Model name passed to the provider. |
| `max_vectors_per_namespace` | `int > 0` | `10000` | — | Per-scope vector cap before LRU eviction. |
| `store_exact_on_semantic_hit` | `bool` | `True` | — | On a semantic hit, also write an exact entry so the next identical request takes the faster exact path. |
| `extract_answer` | `bool` | `False` | — | Return only the best-matching sentence(s) of a string answer on a semantic hit (extractive, no LLM). |
| `extract_min_similarity` | `float` in `[0.0, 1.0]` | `0.5` | — | Confidence a sentence needs before `extract_answer` returns it instead of the full answer. |

Invalid values raise at construction time: an out-of-range `similarity_threshold`
or `extract_min_similarity` raises `ConfigurationError`; a non-positive
`default_ttl` raises `InvalidTTLError`; enabling `semantic_enabled` without an
`embedding_provider` raises `ConfigurationError`.

---

## Cache statistics

```python
stats = cache.stats
print(f"Hit rate: {stats.hit_rate:.1%}")
print(f"Hits: {stats.hits}  Misses: {stats.misses}")
print(f"Exact hits: {stats.exact_hits}  Semantic hits: {stats.semantic_hits}")
print(stats.to_dict())
```

`hits` always equals `exact_hits + semantic_hits`, so you can see how many of
your hits came from the faster exact path versus semantic matching.

---

## Examples

Runnable examples live in [`examples/`](examples/):

- `basic_exact_cache.py` — the cache primitives with no framework.
- `langgraph_agent_example.py` — cached nodes and tools in a LangGraph-style flow.
- `langgraph_math_agent.py` — a real ReAct agent with a **calculator** and a
  **web-search** tool, calling an LLM via LiteLLM.
- `semantic_cache_demo.py` — a reworded query hitting the **semantic cache**
  via the combined `lookup`/`store` flow (offline, no model download).
- `semantic_agent.py` — a real ReAct agent (**calculator** + **web search**)
  whose LLM calls run through the **semantic cache** with a local embedding
  model, so reworded questions reuse cached answers.

```bash
export API_KEY="your-groq-key"          # example uses Groq via LiteLLM
python examples/langgraph_math_agent.py
```

---

## Roadmap

| Phase | Feature | Status |
|---|---|---|
| 1 | Exact cache (LLM + tool), Redis backend, LangGraph + LiteLLM | ✅ Shipped in v0.1 |
| 2 | Semantic cache (embedding similarity, threshold control, answer extraction) | ✅ Shipped in v0.2 |
| 3 | Graph-level and node-level execution reuse | Planned |
| 4 | Analytics dashboard, more framework integrations | Planned |

---

## Documentation

Full documentation — guides plus an auto-generated API reference — is published
at **[pranit-p.github.io/memory-reuse](https://pranit-p.github.io/memory-reuse/)**.

Build and preview it locally with the `docs` extra:

```bash
pip install -e ".[docs]"
mkdocs serve            # live preview at http://127.0.0.1:8000
mkdocs build --strict   # produce the static site in ./site
```

---

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup,
tests, and code-style guidelines, and [CONTRIBUTORS.md](CONTRIBUTORS.md) for the
list of people who have helped build this project.

---

## License

[MIT](LICENSE) © Pranit Pawar
