Metadata-Version: 2.4
Name: simlens
Version: 0.2.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Requires-Dist: numpy>=1.21
Requires-Dist: qdrant-client ; extra == 'all'
Requires-Dist: openai>=1 ; extra == 'all'
Requires-Dist: pytest>=7 ; extra == 'dev'
Requires-Dist: maturin>=1.7 ; extra == 'dev'
Requires-Dist: hypothesis>=6 ; extra == 'dev'
Requires-Dist: faiss-cpu ; extra == 'faiss'
Requires-Dist: google-generativeai ; extra == 'gemini'
Requires-Dist: hnswlib>=0.7 ; extra == 'kg'
Requires-Dist: openai>=1 ; extra == 'llm'
Requires-Dist: openai>=1 ; extra == 'openai'
Requires-Dist: psycopg[binary]>=3 ; extra == 'pgvector'
Requires-Dist: qdrant-client ; extra == 'qdrant'
Requires-Dist: torch>=2 ; extra == 'train'
Requires-Dist: safetensors>=0.4 ; extra == 'train'
Requires-Dist: weaviate-client ; extra == 'weaviate'
Provides-Extra: all
Provides-Extra: dev
Provides-Extra: faiss
Provides-Extra: gemini
Provides-Extra: kg
Provides-Extra: llm
Provides-Extra: openai
Provides-Extra: pgvector
Provides-Extra: qdrant
Provides-Extra: serve
Provides-Extra: train
Provides-Extra: weaviate
License-File: LICENSE
Summary: Faithful, vector-only similarity & ranking attribution — the 'why' vector search never returns.
Keywords: embeddings,interpretability,vector-search,explainability,attribution,sae
Author: ghassenov
License: Apache-2.0
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://github.com/ghassenov/simlens

<p align="center">
  <img src="assets/logo.svg" alt="SimLens" width="140">
</p>

<h1 align="center">SimLens</h1>

<p align="center">
  <strong>See <em>why</em> your vectors match.</strong><br>
  Faithful, vector-only similarity &amp; ranking attribution — for any embedder, any vector store.
</p>

<p align="center">
  <a href="LICENSE"><img alt="License: Apache 2.0" src="https://img.shields.io/badge/license-Apache%202.0-4f46e5"></a>
  <img alt="Core: Rust" src="https://img.shields.io/badge/core-Rust-4f46e5">
  <img alt="Python 3.8+" src="https://img.shields.io/badge/python-3.8%2B-4f46e5">
  <img alt="Status: v0.2" src="https://img.shields.io/badge/status-v0.2-22c55e">
</p>

---

Vector search powers your RAG pipeline, your recommendations, your semantic search, your
anomaly detection. And every one of them answers a question with a **black box**: *this
matches — score `0.83`.* Which concepts drove the match? Why did the wrong result rank
first? Why was the right one buried at #7? The score won't say.

**SimLens turns that score into an answer** — the specific, named concepts that produced
it, in plain language, and it *proves* the breakdown adds up. It reads only the vectors,
so it drops into **any** stack without touching your model or your database.

> Think of it as **SHAP for vector search**: the missing explanation layer for
> similarity and ranking.

```python
import simlens

# Zero setup — an exact explanation on the very first call, no training required.
ex = simlens.Explainer(metric="cosine")
attr = ex.explain(query_vec, candidate_vec)

print(attr.as_sentence())
# → "Matched mainly on 'financial-regulation' (61%), '2024-filings' (22%)."
```

…or skip the manual setup entirely — point `autofit` at your vector store and it samples
vectors, trains the dictionary, and **names features from your payload fields + an
optional LLM**, all in one line:

```python
bundle = simlens.autofit(store)                       # zero manual labeling
# or: simlens.autofit(store, namer=simlens.naming.from_provider("openai"))
```

New here? **[docs/how-simlens-works.md](./docs/how-simlens-works.md)** is a friendly,
plain-language tour of everything below.

## Why teams reach for SimLens

- **Answers the real questions.** Not just *why does A match?* but **why did A outrank
  B?** — the margin, decomposed. The question search and recsys teams actually debug.
- **Faithful by construction.** For dot &amp; cosine, the explanation *is* the arithmetic:
  contributions provably sum to the score. Every result ships a **completeness residual**
  so you can see exactly how exact it is — no hand-wavy storytelling.
- **Three levels of zoom.** Raw dimensions (exact) → learned **monosemantic features** →
  your own **named concepts** — one consistent, additive contract across all three.
- **Steerable.** "More like this, but less of *that*" — edit a query in concept space and
  search again.
- **Drop-in &amp; fast.** Embedder-agnostic, database-agnostic, and backed by a compact
  Rust core for the per-query hot path.
- **Auditable.** Every explanation is stamped with a content hash of the artifacts that
  produced it — reproducible rationale for decisions that have to hold up.

## What you can do

| Capability | Call |
|-----------|------|
| Explain a match (dims / features / concepts / aspects) | `ex.explain(q, c, level=...)` |
| Explain a **ranking** — why A beat B | `ex.explain_margin(q, better, worse)` |
| **Centered "why"** — discriminative, past the anisotropy baseline | `ex.explain(q, c, center=True)` |
| **Any learned metric** (cross-encoder, reranker) via Integrated Gradients | `simlens.LearnedMetricExplainer(scorer).explain(q, c)` |
| Why they're **not** more similar | `ex.explain_dissimilarity(q, c)` |
| Minimal reason — what breaks the match | `ex.ablate(q, c, threshold=...)` |
| Steer a query in concept space | `ex.steer(q, {"topic": -1.0})` |
| Contrast against a background set | `ex.explain_vs_corpus(q, c, foil)` |
| **Certify** a bundle's faithfulness (signed scorecard) | `bundle.certify(vectors)` |
| Summarize a whole result page | `ex.summarize(q, hits)` |
| Late-interaction (multi-vector) attribution | `simlens.MultiVectorExplainer().explain(Q, C)` |
| **Auto-build a bundle from your store** (no manual labeling) | `simlens.autofit(store)` |
| Train &amp; package a concept bundle manually | `simlens.train.build_bundle(...)` |
| Calibrate / audit naming confidence | `simlens.eval.reliability(bundle, X, labelers)` |
| Fetch candidates from your store | `simlens.adapters.Qdrant / Pgvector / Faiss / Weaviate` |
| Inspect, verify, evaluate from the shell | `simlens info \| verify \| eval \| serve` |
| Serve over HTTP | `python -m simlens.serve --bundle b.simlens` |

## System extensions

Thin, customizable wrappers that adapt SimLens to a system type — you bring the vectors,
they bring the explanation. Business-logic-agnostic and configurable.

```python
from simlens.integrations.rag   import RagExplainer          # why retrieved / why ranked
from simlens.integrations.recsys import RecsysExplainer       # "because you liked …", steer
from simlens.integrations.kg    import KnowledgeGraphExplainer # explain / propose / type edges
from simlens.integrations.audit import AuditLog               # signed, hashed decision records
```

Optional extras (same package): `pip install "simlens[qdrant,openai,train]"` — stores
(`qdrant`, `pgvector`, `faiss`, `weaviate`), LLM naming providers (`openai`, `gemini`),
research-scale GPU training + safetensors import (`train`), and sublinear kNN edge
proposal (`kg`).

## Install / build from source

```bash
python -m venv .venv && . .venv/bin/activate
pip install maturin numpy pytest
maturin develop --release      # builds the Rust core into your environment
cargo test -p simlens-core     # Rust unit tests
pytest -q                      # Python tests
python examples/quickstart.py  # full end-to-end demo
```

## Project layout

| Path | What |
|------|------|
| `crates/simlens-core` | Rust attribution kernels — the math and the hot path |
| `crates/simlens-py` | PyO3 bindings → `simlens._native` |
| `python/simlens` | Python API: `Explainer`, `Bundle`, `train`, `eval`, `adapters`, `viz`, `serve` |
| `examples/` | Runnable, self-contained demo |
| `docs/` | [How SimLens works](./docs/how-simlens-works.md) |

## Status

**v0.2 — trustworthy defaults, measured not asserted.** Everything in v0.1 plus:

- **Real SAE trainers** — TopK (default), BatchTopK, JumpReLU with AuxK dead-feature revival,
  a zero-dep numpy backend and an optional `torch`/GPU backend; the training-time sparsity
  gate is reproduced at inference (train == inference), and safetensors/SAELens dictionaries
  drop in via `import_safetensors_sae`.
- **Detection-scored naming** — names carry a *measured* balanced-accuracy (name-as-classifier),
  and low-scoring names are dropped instead of shown as confident nonsense.
- **Centered "why"** — anisotropy correction (mean / ABTT / whitening) surfaces discriminative
  concepts past the global-mean baseline; the default for the integrations.
- **Integrated Gradients** — explain any learned/non-linear scorer with a completeness axiom.
- **Faithfulness certification** — a signed quality scorecard (FVU, L0, dead %, deletion/
  insertion AUC, detection accuracies) baked into the bundle and covered by its content hash.
- **Zero-copy numpy FFI**, **criterion + Python benchmarks** ([docs/benchmarks.md](./docs/benchmarks.md)),
  **CI + abi3 wheels + PyPI/crates.io publishing**, property-based & adversarial tests, and
  per-extension **[real-system validation](./docs/validation.md)**.

Frontier (post-v0.2): a native Rust `simlens-serve`, cross-modal and hierarchical concepts.

## Documentation

- **[How SimLens works](./docs/how-simlens-works.md)** — the concepts, end to end.
- **[Benchmarks](./docs/benchmarks.md)** · **[Real-system validation](./docs/validation.md)**.
- **[LICENSE](./LICENSE)** — Apache-2.0.

## License

SimLens is released under the [Apache License 2.0](./LICENSE).

