Metadata-Version: 2.4
Name: intellicache
Version: 0.1.0
Summary: An intelligent LLM caching library that combines semantic similarity, configurable judgment, and persistent vector storage to safely reuse responses across equivalent queries.
Author: Gavin McCarthy-Bui
License-Expression: MIT
Project-URL: Homepage, https://github.com/Cytadell/IntelliCache
Project-URL: Repository, https://github.com/Cytadell/IntelliCache
Project-URL: Documentation, https://github.com/Cytadell/IntelliCache#readme
Project-URL: Issues, https://github.com/Cytadell/IntelliCache/issues
Keywords: llm,cache,semantic-cache,semantic-search,vector-search,faiss,embeddings,retrieval,artificial-intelligence
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: faiss-cpu<2,>=1.8
Requires-Dist: numpy<3,>=1.26
Provides-Extra: openai
Requires-Dist: openai<3,>=2; extra == "openai"
Requires-Dist: aiolimiter<2,>=1.2; extra == "openai"
Requires-Dist: python-dotenv<2,>=1; extra == "openai"
Requires-Dist: tqdm<5,>=4.66; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic<1,>=0.117; extra == "anthropic"
Requires-Dist: aiolimiter<2,>=1.2; extra == "anthropic"
Requires-Dist: tqdm<5,>=4.66; extra == "anthropic"
Provides-Extra: ollama
Requires-Dist: ollama<1,>=0.6; extra == "ollama"
Requires-Dist: tqdm<5,>=4.66; extra == "ollama"
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers<6,>=3; extra == "sentence-transformers"
Provides-Extra: benchmark
Requires-Dist: gptcache==0.1.44; extra == "benchmark"
Requires-Dist: pandas<4,>=2; extra == "benchmark"
Requires-Dist: sentence-transformers<6,>=3; extra == "benchmark"
Requires-Dist: sqlalchemy<3,>=2; extra == "benchmark"
Provides-Extra: test
Requires-Dist: pytest<10,>=8.2; extra == "test"
Requires-Dist: pytest-asyncio<2,>=1; extra == "test"
Requires-Dist: pytest-cov<8,>=5; extra == "test"
Requires-Dist: pandas<4,>=2; extra == "test"
Requires-Dist: python-dotenv<2,>=1; extra == "test"
Dynamic: license-file

# IntelliCache : A Library for Creating Matching Cache for LLM Queries

## What is IntelliCache
Various large language models (LLMs) boast incredible versatility, enabling the development of a wide range of applications.
However, as your application grows in popularity and encounters higher traffic levels, the expenses related to LLM API calls can become substantial.
While semantic caching systems (like gptcache) exist to build a semantic cache for storing LLM responses, Intellicache goes beyond
semantic caching by handling cases semantic matching may mislabel through a small LLM judge, boosting both caching accuracy and precision
in a cost-effective way.

## Why IntelliCache?

Exact caches miss differently worded versions of the same request. Pure
similarity caches (like gptcache) can make the opposite mistake 
and reuse an answer for a query that only looks related.

IntelliCache combines:

- Exact query detection
- Configurable semantic matching
- Configurable LLM judgment for ambiguous candidates
- Custom embedding and judge provider interfaces


> [!WARNING]
> IntelliCache is alpha software. Its API and on-disk format may change before
> a stable release. Review the [current limitations](#current-limitations)
> before using it with production data

## Installation

IntelliCache requires Python 3.11 or newer.

The recommended setup uses local Sentence Transformers embeddings and an
OpenAI judge for ambiguous matches:

```bash
pip install "intellicache[sentence-transformers,openai]"
```

Other installation options:

```bash
# OpenAI embeddings and judge
pip install "intellicache[openai]"

# Local embeddings without an LLM judge
pip install "intellicache[sentence-transformers]"

# Anthropic Claude judge
pip install "intellicache[anthropic]"

# Local Ollama embeddings and judge
pip install "intellicache[ollama]"

# Install multiple integrations together
pip install "intellicache[anthropic,ollama]"

# Custom providers only
pip install intellicache
```

Optional integrations are loaded only when used. Importing `IntelliCache`
does not load any provider SDK, and requesting a missing integration produces
an error showing which extra to install.

## Built-in providers

Embedding providers:

- `OpenAIEmbedding`
- `SentenceTransformerEmbedding`
- `OllamaEmbedding`

Judge providers:

- `OpenAIJudge`
- `AnthropicJudge`
- `OllamaJudge`

Providers can be mixed. For example, use `OpenAIEmbedding()` with
`AnthropicJudge()`. Hosted providers read their conventional environment
variables: `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`. Ollama defaults to
`http://localhost:11434`; its models must already be available on that server.

## Quick start

Set your OpenAI API key before running the example:

```bash
export OPENAI_API_KEY="your-api-key"
```

In PowerShell, use `$env:OPENAI_API_KEY="your-api-key"` instead.

You also can make a .env file containing OPENAI_API_KEY="your-api-key"
and import it using dotenv() from python-dotenv

OpenAI API usage:

```python
import asyncio

from intellicache import IntelliCache
from intellicache.embedding import OpenAIEmbedding
from intellicache.judge_evaluator import OpenAIJudge


async def generate_response(query: str) -> str:
    # Replace this with your LLM, RAG pipeline, or API call.
    # Any configuration works so long as it accepts a query
    # str and returns a response str
    return f"Generated response for: {query}"


async def main() -> None:
    cache = IntelliCache(
        embedding_provider=OpenAIEmbedding(model_name="text-embedding-3-small"),
        judge_provider=OpenAIJudge(
            model_name="gpt-5-nano",
            reasoning_effort="low",
            max_output_tokens=1024,
        ),
        cache_dir=".intellicache_test",
        use_judge=True,
        base_threshold=0.97,
        judge_threshold=0.92,
        judge_k=10,
    )

    try:
        # Searches for query in cache, if none exists
        # it generates new response and caches it
        result = await cache.get_or_create(
            query="How many seasons are there in a year?",
            response_factory=generate_response,
        )

        print("Response:", result.response)
        print("Cache hit:", result.hit)
        print("Decision:", result.decision_path)

        # Second search of similar query will likely result in a
        # 'hit' and return the cached response rather than generating
        # a new response
        result = await cache.get_or_create(
            query="How many seasons in a year?",
            response_factory=generate_response,
        )

        print("Response:", result.response)
        print("Cache hit:", result.hit)
        print("Decision:", result.decision_path)

    finally:
        #Needed step to save cache between runs
        cache.close()


asyncio.run(main())
```

The first query is not yet in the empty cache, so IntelliCache calls
`generate_response` and stores its result. The second query expresses the same
request differently. IntelliCache embeds it, compares it with the cached
query, and can reuse the first response instead of calling
`generate_response` again. The printed `hit` and `decision_path` values show
whether reuse occurred and which matching stage made the decision.

`get_or_create()` is the main cache-aside interface: it searches first, calls
`response_factory(query)` only on a miss, stores the generated response, and
returns both the response and information about the cache decision. Because
the example uses the persistent `.intellicache_test` directory, later runs can
reuse entries created by earlier runs. Delete that directory when you want to
start again with an empty cache.

A similarity at or above `base_threshold` is accepted directly. A candidate
between `judge_threshold` and `base_threshold` is sent to the OpenAI judge,
which decides whether its cached response is safe to reuse. A lower-scoring
query is treated as a miss without a judge call. OpenAI embedding requests and
judge requests can both add provider cost and latency, so validate the
thresholds against representative data from your application.

## Core API

### `get_or_create()`

Use `get_or_create()` for the normal cache-aside workflow:

```python
async def generate_response(query: str) -> str:
    #Example client, change this for your LLM/RAG/Response system
    response = await my_client.generate(query)
    return response


result = await cache.get_or_create(
    query="How do I reset my password?",
    response_factory=generate_response,
)
```

The callable runs only on a cache miss. If it raises an exception, no new
response is stored.

When a semantic hit occurs, IntelliCache returns the existing response and
stores the submitted query as another variant of that response set. This can
improve recognition of future phrasings.

### `search()`

Search without calling a generator or inserting a query:

```python
result = await cache.search("Where can I change my password?")

if result.hit:
    print(result.response)
else:
    print("Cache miss")
```

### `put_response()`

Store a query and a response you already have:

```python
result = await cache.put_response(
    query="How do I reset my password?",
    response="Open account settings and select Reset Password.",
)
```

If the identical query already exists, the response is reused or updated. A
merely similar query does not overwrite an existing response set.

### `put_variant()`

Associate a new phrasing with an existing response:

```python
result = await cache.put_variant(
    query="Where can I update my password?",
)
```

This method requires a reusable match. It raises `KeyError` if no suitable
response set is found.

### Lifecycle and statistics

```python
print(cache.stats())
# {"responses": 1, "queries": 2, "vectors": 2}

cache.save()
cache.close()
```

`close()` saves the FAISS index before closing SQLite. Use `try/finally` when
the cache does not live for the entire process.

## Cache results

Public operations return a `CacheResult`:

| Field | Meaning |
| --- | --- |
| `hit` | Whether an existing response was reused |
| `query` | The submitted query |
| `response` | The cached or newly created response |
| `set_id` | Identifier of the associated response set |
| `matched_query` | Existing query responsible for the match |
| `score` | Similarity score, when applicable |
| `decision_path` | How the result was selected |

Common decision paths include:

- `miss`
- `new_cluster`
- `exact_duplicate`
- `updated_response`
- `base_threshold`
- `judge`

## Matching policy

With the default vector configuration, FAISS normalizes vectors and performs
inner-product search. For normalized vectors, inner product is equivalent to
cosine similarity, so higher scores represent more similar queries.

The matching policy is:

1. Retrieve the nearest stored queries from FAISS.
2. Reuse an identical best query as an exact duplicate.
3. Reuse the best candidate when its score meets `base_threshold`.
4. If enabled, call the judge when the best score is below
   `base_threshold` but at least `judge_threshold`.
5. Return a miss if no candidate is accepted.

Thresholds are model- and dataset-dependent. Benchmark them against your own
queries, with particular attention to false-positive cache hits.

## Configuration

| Option | Default | Purpose |
| --- | ---: | --- |
| `embedding_provider` | required | Produces query embeddings |
| `judge_provider` | `None` | Optionally evaluates ambiguous candidates |
| `use_judge` | `False` | Enables judge evaluation |
| `use_adv_embedding` | `False` | Enables the optional secondary embedding check |
| `adv_embedding_provider` | `None` | Provider used for that secondary check |
| `cache_dir` | `"cache_storage"` | Persistent cache directory; use `None` for an in-memory cache |
| `base_threshold` | `0.97` | Minimum score for direct semantic reuse |
| `judge_threshold` | `0.92` | Minimum score before consulting the judge |
| `judge_k` | `10` | Number of candidates supplied to the judge |
| `normalize_vectors` | `True` | L2-normalizes vectors before FAISS search |
| `full_storage_validation` | `True` | Validates SQLite and FAISS IDs at startup |

When `use_judge=True`, a judge provider is required.

Pass `cache_dir=None` for an explicitly non-persistent cache. In-memory caches
do not create SQLite, FAISS, or configuration files and are discarded by
`close()` or when the process exits.

## Persistence and compatibility

Each cache directory contains:

```text
.intellicache/
├── cache.sqlite
├── faiss.index
└── cache_config.json
```

`cache_config.json` records the embedding identity, dimension, vector
normalization, search metric, and storage schema version. IntelliCache rejects
an incompatible configuration rather than comparing vectors produced by
different embedding spaces.

Embedding providers expose a JSON-compatible `cache_identity`. Identity fields
should include settings that affect generated vectors, but never secrets such
as API keys.

If SQLite and FAISS contain different vector IDs, IntelliCache rebuilds FAISS
from the embeddings stored in SQLite. A populated legacy cache without
`cache_config.json` is rejected because its embedding configuration cannot be
verified.

## Custom providers

### Custom embedding provider

An embedding provider supplies a dimension, stable cache identity, and
embedding methods:

```python
from collections.abc import Sequence
from typing import Any

import numpy as np


class MyEmbeddingProvider:
    @property
    def dimension(self) -> int:
        return 384

    @property
    def cache_identity(self) -> dict[str, Any]:
        return {
            "provider": "my-company",
            "model": "support-embeddings-v1",
            "normalize_embeddings": True,
        }

    def embed_one(self, text: str) -> np.ndarray:
        ...

    def embed_many(
        self,
        texts: Sequence[str],
        show_progress_bar: bool = False,
    ) -> np.ndarray:
        ...
```

The identity must contain only JSON-compatible values and must change whenever
the provider begins producing an incompatible vector space.

Pass the custom provider directly to `IntelliCache`:

```python
cache = IntelliCache(
    embedding_provider=MyEmbeddingProvider(),
)
```

#### Adding a Custom LLM-backed judge

Authors of LLM-backed integrations can subclass `BaseStructuredJudge` instead
of rebuilding the orchestration layer. It provides prompt construction,
candidate-index parsing, retries, rate limiting, concurrent `judge_many()`
execution, default fallback behavior, and token-usage accounting. A provider
adapter only needs to initialize its client and implement one request method:

```python
from typing import Any

from intellicache.judge_evaluator import (
    BaseStructuredJudge,
    JudgeModelResponse,
)


class MyLLMJudge(BaseStructuredJudge):
    def __init__(
        self,
        client: Any,
        model_name: str = "my-fast-classifier",
        *,
        match_criteria: str | None = None,
    ) -> None:
        self.client = client
        self.model_name = model_name
        super().__init__(
            match_criteria=match_criteria,
            max_retries=3,
            default_judgement=None,
            # Add the provider's transient API exception classes here.
            retryable_exceptions=(),
        )

    async def _request_judgement(
        self,
        prompt: str,
        *,
        num_candidates: int,
    ) -> JudgeModelResponse:
        response = await self.client.generate(
            model=self.model_name,
            prompt=prompt,
            # Configure the provider to return JSON when supported.
        )
        return JudgeModelResponse(
            output_text=response.text,
            usage=getattr(response, "usage", None),
        )
```

The model output must be a JSON object with exactly one decision field:

```json
{"match_index": 0}
```

Use `null` when no candidate matches:

```json
{"match_index": null}
```

The base class validates the result against `num_candidates`. Provider usage
objects with `input_tokens`, `output_tokens`, and `total_tokens` attributes are
recorded automatically. Override `_record_usage()` when an SDK uses different
field names. Pass a provider rate limiter to `super().__init__()` when the
service has request limits. Local providers can omit `rate_limiter` to use the
built-in no-op limiter.

```python
cache = IntelliCache(
    embedding_provider=MyEmbeddingProvider(),
    judge_provider=MyJudgeProvider(),
    use_judge=True,
    cache_dir=".intellicache",
)
```

## Custom match criteria

Every built-in LLM judge accepts `match_criteria`. The text defines what the
judge considers safe to reuse:

> [!CAUTION]
> Custom LLM judging is experimental and does not include a complete prompt-
> injection defense. The judge receives query and candidate-query text, not
> cached response bodies, which limits direct data exposure; however, hostile
> query text could still influence a match decision and cause an unintended
> cached response to be returned. Do not treat the judge as an authorization
> boundary, and validate it against the kinds of untrusted input your
> application accepts.

```python
from intellicache.judge_evaluator import OpenAIJudge

support_criteria = """
Match only when one response can answer both queries completely and without
modification.

For account-support requests, the product, requested operation, account state,
platform, and error condition must all be the same.

Reject queries when either one adds or removes a requested step, changes the
product or platform, refers to a different error, or requires an additional
assumption. Shared topic or vocabulary alone is not sufficient.
""".strip()

judge = OpenAIJudge(
    model_name="gpt-5-nano",
    match_criteria=support_criteria,
)
```

`AnthropicJudge` and `OllamaJudge` accept the same argument. Passing `None`
uses IntelliCache's strict default criteria. A custom value **replaces** the
default criteria, so include every rule your application depends on rather
than supplying only a short amendment.

Criteria should describe observable equivalence rather than tell the model to
be generally "careful." State what details must match, list common reasons to
reject, and explicitly require `None` when uncertain. Validate changes with
representative labeled queries: stricter criteria generally reduce false
positives but can reduce recall, while looser criteria do the reverse.

The criteria apply only to candidates in the judge band: from
`judge_threshold` up to, but not including, `base_threshold`. Similarities at
or above `base_threshold` are accepted without a judge, and similarities below
`judge_threshold` miss without one. Tune the thresholds and criteria together.

## Benchmark results

This run used local `sentence-transformers/all-MiniLM-L6-v2` embeddings and a
`gpt-5-nano` judge with **low reasoning**, a `0.98` base threshold, and a
`0.92` judge threshold.

IntelliCache and GPTCache processed the same 20,000 labeled queries from Yahoo
Answers in the same order, including 908 repeat-query opportunities:
(used data is accessible at: https://huggingface.co/datasets/Cytadell/yahoo_answers_matches)

| System | Correct reuse | Incorrect reuse | Hit precision |
| --- | ---: | ---: | ---: |
| IntelliCache judge (`0.98/0.92`) | **467** | **15** | **96.89%** |
| GPTCache (`0.92`) | 476 | 67 | 87.66% |
| GPTCache (`0.97`) | 283 | 7 | 97.59% |

IntelliCache remained within one percentage point of GPTCache `0.92` correct-
repeat coverage while reducing incorrect reuse by **77.6%**. Against high-
precision GPTCache `0.97`, it recovered **65.0% more correct hits** at similar
precision. The judge was invoked for only **1.60% of requests**.

### Ordered-workload judge cost

The judge used 163,446 input tokens and 99,327 output tokens across 320 calls.
At GPT-5 Nano's listed price of $0.05 per million input tokens and $0.40 per
million output tokens, the complete 20,000-query run cost approximately
**$0.048**, or **$0.0000024 per workload request**. The judge directly supplied
225 correct reuses, making its measured cost approximately **$0.00021 per
direct judge-path correct reuse**. Local embedding compute and answer-generation
costs are not included. See [GPT-5 Nano pricing](https://developers.openai.com/api/docs/models/gpt-5-nano).
For a price-only comparison, applying GPT-5 Mini's $0.25/M input and $2/M
output rates to the same recorded token counts would cost approximately
**$0.24**, or 5x as much. An actual Mini rerun could use a different number of
tokens and produce different decisions. See [GPT-5 Mini pricing](https://developers.openai.com/api/docs/models/gpt-5-mini).

RAG does not have a universal token cost because retrieved context, system
instructions, and answer length vary. For scale, a hypothetical RAG answer
using 2,000 input tokens and 300 output tokens would cost:

| Answer model | Approximate cost per generated answer | Compared with $0.00021 judge cost per directly recovered hit |
| --- | ---: | ---: |
| GPT-5 Nano | $0.00022 | 1.0x |
| GPT-5 Mini | $0.0011 | 5.2x |
| GPT-5.6 Luna | $0.0038 | 18x |
| GPT-5.6 Terra | $0.0095 | 45x |
| GPT-5.6 Sol | $0.0190 | 89x |

This example uses [OpenAI's listed model prices](https://developers.openai.com/api/docs/models)
as of August 2026. The assumed 300 output tokens are **total billed output
tokens**, including internal reasoning tokens. If a request produces 300
visible tokens plus additional reasoning tokens, its real cost will be higher
than this table. It is a transparent break-even illustration, not a claim that
every RAG request contains exactly 2,300 billed tokens. On very small Nano
generations, judge savings may not cover the judge by themselves; the
economics become substantially stronger as the avoided answer model, reasoning
effort, or RAG context becomes more expensive.

These results demonstrate the precision/coverage tradeoff on this labeled
workload; they are not a claim that every application will produce the same
numbers.

### Adversarial PAWS pair test

A separate 677-pair PAWS test from Quora Question Pairs measured resistance to highly similar queries
whose meaning may change when words or entities are swapped. Both systems used
`text-embedding-3-small` and the same `0.95` candidate threshold. IntelliCache
used a `0.9999` direct-hit threshold and sent ambiguous candidates to a
low-reasoning `gpt-5-nano` judge.

| System | Accuracy | Hit precision | Recall | False-positive rate | False positives |
| --- | ---: | ---: | ---: | ---: | ---: |
| IntelliCache judge | **82.57%** | **67.30%** | 74.35% | **14.20%** | **69** |
| GPTCache | 41.21% | 31.94% | **95.81%** | 80.25% | 390 |

IntelliCache reduced false positives by **82.3%** and improved accuracy by
**41.36 percentage points**. In the paired comparison, IntelliCache alone was
correct on 321 examples while GPTCache alone was correct on 41. The tradeoff
was lower recall and higher latency. Because PAWS is intentionally adversarial,
the judge ran on 572 of 677 pairs (84.5%); that call rate should not be treated
as representative production traffic.

## Current limitations

- A cache directory represents one trusted application context. Do not share
  it across applications, system prompts, knowledge bases, or users that
  require data isolation.
- Queries, responses, metadata, and embeddings are stored locally without
  application-level encryption.
- Entries do not expire automatically, and there is not yet a public deletion
  or eviction API.
- Concurrent misses for the same query are not deduplicated and may call the
  generator more than once.
- Cross-process and multithreaded access guarantees are not yet defined.
- Corrupted FAISS files may require removing `faiss.index` so it can be rebuilt
  from SQLite.
- The advanced embedding path is experimental.
- The public API and storage schema may change during the alpha period.

## Development and testing

for local development:

```bash
git clone https://github.com/Cytadell/intellicache.git
cd IntelliCache
pip install -e ".[openai,sentence-transformers,test]"
```

Run the deterministic offline suite:

```bash
python -m pytest -q
```

The default tests use fake providers and do not make external API calls.

Slow provider tests are skipped unless explicitly enabled:

```bash
python -m pytest --runslow
```

Slow tests may download models, make network requests, use API credentials, and
incur provider charges.

Build and validate distribution artifacts:

```bash
python -m build
python -m twine check dist/*
```

## License

IntelliCache is available under the [MIT License](LICENSE).
