Metadata-Version: 2.5
Name: cacheverifier
Version: 0.2.0
Summary: Python client for the hosted CacheVerifier semantic-cache verification API
Project-URL: Homepage, https://www.cacheverifier.com
Project-URL: Documentation, https://www.cacheverifier.com/docs
Project-URL: Source, https://github.com/imxinchengyou/cacheverifier-python
Project-URL: Research, https://github.com/imxinchengyou/CacheVerifier
Author: Chengyou Xin
License: MIT
License-File: LICENSE
Keywords: cache-verification,cross-encoder,gptcache,llm,llm-caching,semantic-cache,semantic-cache-verification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx>=0.24
Provides-Extra: dev
Requires-Dist: mypy>=1.5; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: gptcache
Requires-Dist: gptcache>=0.1.30; extra == 'gptcache'
Provides-Extra: healthcheck
Requires-Dist: numpy<2,>=1.26; extra == 'healthcheck'
Requires-Dist: sentence-transformers<4,>=3.0; extra == 'healthcheck'
Requires-Dist: torch>=2.2; extra == 'healthcheck'
Description-Content-Type: text/markdown

# cacheverifier

[![CI](https://github.com/imxinchengyou/cacheverifier-python/actions/workflows/ci.yml/badge.svg)](https://github.com/imxinchengyou/cacheverifier-python/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/cacheverifier)](https://pypi.org/project/cacheverifier/)
[![Python](https://img.shields.io/pypi/pyversions/cacheverifier)](https://pypi.org/project/cacheverifier/)

Python client for **[CacheVerifier](https://www.cacheverifier.com)** — a hosted API that
verifies semantic-cache hits. Given a query and a candidate cached answer, it approves or
rejects serving that answer from cache, so a similarity match that is *close but wrong*
doesn't become a silent error in your app.

CacheVerifier does **not** run your cache or do similarity search. Your cache backend does
its own lookup first; you call `verify()` only on the candidates in the similarity "gray
zone", where a plain threshold match might be wrong.

- Docs / API reference: <https://www.cacheverifier.com/docs>
- Why similarity ≠ correctness: <https://www.cacheverifier.com/why-similarity-fails>
- The research behind it (paper + benchmarks): <https://github.com/imxinchengyou/CacheVerifier>

## Install

```bash
pip install cacheverifier
# with the GPTCache adapter:
pip install "cacheverifier[gptcache]"
# with the offline Health Check (adds torch + sentence-transformers):
pip install "cacheverifier[healthcheck]"
```

Requires Python 3.9+. The only runtime dependency is `httpx` — the extras above
are opt-in.

## Quickstart

Get a free API key at <https://www.cacheverifier.com> (self-serve verify and fine-tuning
are free forever, no card).

```python
from cacheverifier import CacheVerifier

cv = CacheVerifier(api_key="cv_...")

query = "how do I cancel my subscription"
candidate = "Go to Settings > Billing > Pause subscription for a month."  # from your cache

result = cv.verify(query, candidate)
if result.approved:
    answer = candidate                       # verified hit — skip the LLM call
else:
    answer = call_your_llm(query)            # not trustworthy — fall through

# Later, once you know if it was actually right (thumbs-down, reopened ticket, ...):
cv.feedback(query, answer, was_correct=True, similarity_score=0.86)
```

`verify()` returns a `VerifyResult`:

| field | meaning |
|---|---|
| `approved` | serve the cached answer (`True`) or fall through (`False`) |
| `score` / `threshold` | `approved` is `score >= threshold` |
| `model_version` | `"stock"`, `"v<id>"` (fine-tuned), or `"cold_start_fail_closed"` |
| `latency_ms` | server-side inference time |

## GPTCache

Drop the verifier into a GPTCache pipeline as its similarity evaluator — no fork required:

```python
from gptcache import cache
from cacheverifier.integrations.gptcache import CacheVerifierEvaluation

evaluator = CacheVerifierEvaluation(api_key="cv_...")
cache.init(similarity_evaluation=evaluator, ...)

# when you learn a served hit's real outcome:
evaluator.report_feedback(query, answer, was_correct=False, similarity_score=0.9)
```

See [`examples/gptcache_example.py`](examples/gptcache_example.py).

## Fine-tuning

Once you have ~20+ feedback rows (the service found fine-tuning is often a net negative
below ~1,000 on the hardest data — see the [research](https://github.com/imxinchengyou/CacheVerifier)),
train a verifier on your own gray-zone labels:

```python
job = cv.finetune()                       # or cv.finetune(target_risk=0.01, cost_ratio=5.0)
job = cv.get_finetune_job(job["id"])      # poll until status == "done"
print(job["auc_baseline"], job["auc_tuned"])

# a model can finish as "held_for_review" — promote it explicitly:
if job.get("result_model_version"):
    cv.activate_model_version(job["result_model_version"])
```

`cv.dry_run([...])` reports the same baseline-vs-tuned AUC on examples you pass directly,
without writing anything or deploying a model.

## Local Health Check (offline)

`cv.dry_run()` still uploads your examples to the API. If that's a blocker — a
compliance review, or just not wanting production traffic to leave your network —
run the identical stock-vs-fine-tuned evaluation entirely on your own machine:

```bash
pip install "cacheverifier[healthcheck]"

cacheverifier healthcheck traffic.jsonl
cacheverifier healthcheck traffic.jsonl --emit-summary summary.json
```

`traffic.jsonl` is a JSON array or JSONL of `{"query", "candidate_answer", "was_correct"}`
rows **in arrival order** (the train/calibrate/test split is chronological, matching the
hosted service so the numbers are comparable). Optional per row: `"stale": true`.

Nothing is sent anywhere — the base model downloads once from Hugging Face, then it's
fully offline. `--emit-summary` writes an aggregate-only JSON file (AUCs, counts, rates —
no query or answer text) that's safe to share for a human read.

```
results
------------------------------------------------------------------
  train / calibrate / test:         3349 / 419 / 419
  stock verifier   held-out AUC:    0.6120
  fine-tuned       held-out AUC:    0.7080   (delta +0.0960)
  label-noise proxy (disagreement): 11.4%
  ceiling status:                   still_improvable

verdict
------------------------------------------------------------------
  IMPROVED   -- fine-tuning on your own data helps this traffic
```

## API surface

| method | endpoint |
|---|---|
| `verify(query, candidate_answer)` | `POST /v1/verify` |
| `verify_batch(pairs)` | `POST /v1/verify/batch` |
| `feedback(...)` / `feedback_batch(items)` | `POST /v1/feedback` / `/batch` |
| `finetune(...)` / `dry_run(examples, ...)` | `POST /v1/finetune/jobs` / `/dry-run` |
| `get_finetune_job(id)` / `list_finetune_jobs()` | `GET /v1/finetune/jobs[/id]` |
| `activate_model_version(id)` | `POST /v1/finetune/model-versions/{id}/activate` |
| `drift_status()` | `GET /v1/monitor/drift-status` |
| `gray_zone_threshold()` | `GET /v1/monitor/gray-zone-threshold` |
| `usage()` / `savings()` | `GET /v1/usage/status` / `/savings` |

Non-2xx responses raise `CacheVerifierError` (`.status_code`, `.detail`).

## License

MIT — see [`LICENSE`](LICENSE). (The [research repository](https://github.com/imxinchengyou/CacheVerifier)
is separately licensed; this client is not.)
