Metadata-Version: 2.4
Name: isotrieve
Version: 0.2.1
Summary: Migrate a vector database to a new embedding model without re-embedding the corpus. Fits a linear map from ~2K calibration texts; 87-91% retrieval retention measured on BEIR.
Project-URL: Homepage, https://github.com/krish1925/isotrieve
Project-URL: Documentation, https://github.com/krish1925/isotrieve#readme
Project-URL: Repository, https://github.com/krish1925/isotrieve
Project-URL: Issues, https://github.com/krish1925/isotrieve/issues
Author: Isotrieve Contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ada-002,beir,chroma,dimension-mismatch,embedding-alignment,embedding-migration,embedding-model,model-deprecation,nDCG,openai-embeddings,pgvector,qdrant,re-embedding,retrieval,ridge-regression,sentence-transformers,simsearch,text-embedding-3,vector-database,vector-search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: rich>=13.0
Requires-Dist: scikit-learn>=1.3
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: chromadb>=0.4; extra == 'all'
Requires-Dist: cohere>=5.0; extra == 'all'
Requires-Dist: google-generativeai>=0.5; extra == 'all'
Requires-Dist: langchain-core>=0.3; extra == 'all'
Requires-Dist: mypy>=1.8; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: psycopg[binary]>=3.1; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Requires-Dist: pytest-cov>=4.1; extra == 'all'
Requires-Dist: pytest>=7.4; extra == 'all'
Requires-Dist: qdrant-client>=1.7; extra == 'all'
Requires-Dist: ruff>=0.4; extra == 'all'
Requires-Dist: sentence-transformers>=2.2; extra == 'all'
Requires-Dist: torch>=2.0; extra == 'all'
Requires-Dist: voyageai>=0.2; extra == 'all'
Provides-Extra: benchmarks
Requires-Dist: datasets>=2.14; extra == 'benchmarks'
Requires-Dist: ir-datasets>=0.5; extra == 'benchmarks'
Requires-Dist: sentence-transformers>=2.2; extra == 'benchmarks'
Provides-Extra: chroma
Requires-Dist: chromadb>=0.4; extra == 'chroma'
Provides-Extra: cohere
Requires-Dist: cohere>=5.0; extra == 'cohere'
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: sentence-transformers>=2.2; extra == 'dev'
Provides-Extra: gemini
Requires-Dist: google-generativeai>=0.5; extra == 'gemini'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.10; extra == 'llamaindex'
Provides-Extra: mlp
Requires-Dist: torch>=2.0; extra == 'mlp'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0; extra == 'parquet'
Provides-Extra: pgvector
Requires-Dist: psycopg[binary]>=3.1; extra == 'pgvector'
Provides-Extra: pinecone
Requires-Dist: pinecone-client>=2.0; extra == 'pinecone'
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.7; extra == 'qdrant'
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers>=2.2; extra == 'sentence-transformers'
Provides-Extra: voyage
Requires-Dist: voyageai>=0.2; extra == 'voyage'
Description-Content-Type: text/markdown

# isotrieve

Embedding providers deprecate models constantly — ada-002 is gone, text-embedding-3 is next. When that happens, you either re-embed your entire corpus (expensive, slow, risky) or get stuck on a dead model. Isotrieve lets you switch without re-embedding: fit a lightweight linear transform from ~2K calibration texts, apply it to stored vectors, and gate the migration on measured retrieval retention. 87-91% retention on BEIR benchmarks.

## Install

```bash
pip install isotrieve
```

Python >= 3.10. Core deps: numpy, scikit-learn, typer, rich.

Optional extras:
- `pip install isotrieve[chroma]` — ChromaDB adapter
- `pip install isotrieve[langchain]` — LangChain embeddings shim
- `pip install isotrieve[llamaindex]` — LlamaIndex query wrapper
- `pip install isotrieve[sentence-transformers]` — local model support
- `pip install isotrieve[qdrant]` — Qdrant store adapter
- `pip install isotrieve[openai]` — OpenAI client shim
- `pip install isotrieve[all]` — everything above

## 5-minute trial: query-time wrapper

Zero writes to your vector store. Map new-model queries into legacy space on-the-fly. Fully reversible.

### LlamaIndex

```python
from isotrieve.wrappers.llamaindex import IsotrieveEmbedding
from isotrieve.mapping.registry import load_mapping

mapping = load_mapping("mapping.isotrieve")
wrapper = IsotrieveEmbedding(
    new_model_embedder=your_llamaindex_embedder,
    transform_artifact_path="mapping.isotrieve",
)
# Use wrapper anywhere LlamaIndex expects a BaseEmbedding
# Queries are mapped; document embeddings raise IsotrieveWrapperUsageError
```

### OpenAI client

```python
import openai
from isotrieve.wrappers.openai_shim import IsotrieveOpenAI

client = openai.OpenAI()
shim = IsotrieveOpenAI(client, "mapping.isotrieve")
response = shim.embeddings.create(input=["query text"], model="text-embedding-3-small")
# response.data[0].embedding is now in legacy-model space
```

### LangChain

```python
from isotrieve.adapters.langchain import IsotrieveEmbeddings
from langchain_openai import OpenAIEmbeddings

mapping = Mapping.load("mapping.isotrieve")
base = OpenAIEmbeddings(model="text-embedding-3-small")
ae = IsotrieveEmbeddings(mapping, base)

from langchain_chroma import Chroma
db = Chroma.from_documents(docs, embedding=ae)
results = db.similarity_search("query", k=10)
```

## Quality gate

Before migrating anything, verify the transform preserves retrieval quality:

```bash
isotrieve gate --mapping mapping.isotrieve \
          --source-vectors X_sample.npy \
          --target-vectors Y_sample.npy
```

Output: retention table (Recall@1/5/10, MRR), bootstrap confidence intervals, per-metric pass/fail, and a one-line verdict. Exit code 0 for PASS, 1 for WARN/FAIL — use it in CI.

## Full migration

```bash
# 1. Plan cost
isotrieve plan --source-model ada-002 --target-model te3-large --corpus-size 1000000

# 2. Calibrate
isotrieve calibrate --source-vectors X.npy --target-vectors Y.npy -o mapping.isotrieve

# 3. Gate
isotrieve gate --mapping mapping.isotrieve --source-vectors X.npy --target-vectors Y.npy

# 4. Migrate
isotrieve transform --mapping mapping.isotrieve --source-dir ./old_store --target-dir ./new_store
```

### Serve mode (zero corpus writes)

Map queries on-the-fly without touching stored data:

```python
from isotrieve.serve import QueryAdapter

qa = QueryAdapter.load("mapping.isotrieve")
legacy_vec = qa.map_query(new_model_embed(query))
```

## Adapter status

| Store | Serve mode | Offline migration | Status |
|-------|-----------|-------------------|--------|
| ChromaDB | `IsotrieveChromaFunction` | `migrate_collection()` | Supported |
| LangChain | `IsotrieveEmbeddings` | via store adapter | Supported |
| LlamaIndex | `IsotrieveEmbedding` wrapper | via store adapter | Query wrapper |
| OpenAI | `IsotrieveOpenAI` shim | N/A | Query shim |
| Qdrant | `QdrantStore` | checkpointed in-place | Supported |
| Pinecone | — | shadow-namespace | Planned |

## Claims policy

Every quantitative claim in this README or docs references a committed artifact in `benchmarks/results/` and a row in `isotrieve-python/CLAIMS.md`. No exceptions. If a number isn't in CLAIMS.md, it isn't a claim.

### Adapter comparison (SciFact, MiniLM→bge-large, K=4000, 3 seeds)

| Adapter | nDCG@10 retention | Notes |
|---------|------------------|-------|
| Ridge | 0.871 ± 0.006 | Default. Fast, stable. |
| LowRank | 0.857 ± 0.009 | Compressed matrix. ~1% worse. |
| MLP | 0.727 ± 0.007 | No tuning. Linear wins. |

### K-sweep (all adapters averaged, SciFact, 3 seeds)

| K | nDCG@10 retention | Gate |
|---|------------------|------|
| 500 | 0.671 ± 0.041 | WARN |
| 1000 | 0.735 ± 0.058 | WARN |
| 2000 | 0.785 ± 0.052 | PASS |
| 4000 | 0.832 ± 0.061 | PASS |

### Same-dim pair (bge-large→e5-large, 1024→1024)

| Metric | Value |
|--------|-------|
| Floor (raw cross-space) | 0.0 |
| Isotrieve (mapped) | 0.667 |
| Ceiling (full re-embed) | 0.722 |
| Retention | 0.923 ± 0.010 |

Same dimension ≠ same space. e5 models require "query: "/"passage: " prefixes; without them ceiling drops to 0.36.

### Confidence flags (predictive across both pairs)

| Pair | High-conf R@10 | Low-conf R@10 | Gap |
|------|---------------|---------------|-----|
| bge→e5 | 0.955 | 0.637 | 0.318 |
| MiniLM→bge | 0.875 | 0.651 | 0.224 |

### Score recalibration (MiniLM→bge, rectangular)

| Threshold | Raw recall | + Recalibration | Δ |
|-----------|-----------|-----------------|---|
| τ = 0.60 | 78% | 100% | +22% |
| τ = 0.70 | 27% | 67% | +40% |

## When NOT to use Isotrieve

- Maximum retrieval quality matters more than cost → re-embed
- Calibration domain mismatches corpus (e.g., code index calibrated on prose)
- Quality gate returns FAIL → do not migrate; re-embed
- You need unsupervised migration (Isotrieve requires paired calibration)
- K < 2000 (quality degrades significantly below this)

## Anti-patterns

- Do not mix vectors from different models in one collection
- Do not assume same dimensionality means compatibility
- Do not skip the quality gate
- Do not use MLP adapter (0.727 vs 0.871 for Ridge, same cost)

## How it works

1. Embed K texts with source and target models → matrices X, Y
2. Fit ridge map Y = [X | 1] W (handles unequal dims)
3. Hold out 10% to estimate quality
4. Transform corpus: V' = normalize(V @ W) (streaming batches)
5. Write to new collection; keep old as rollback

## Prior art

Engineering, not research. Built on:
- vec2vec (Jha et al., 2025)
- Drift-Adapter (EMNLP 2025)
- Platonic Representation Hypothesis (Huh et al., 2024)

## Security

Embedding translation enables inversion-style attacks. Treat mapped vectors with same sensitivity as source text.

## License

Apache-2.0
