Metadata-Version: 2.4
Name: poison-vectors
Version: 0.2.0
Summary: POC: query-conditional poison vectors for vector search relevance feedback
Author: James Burton
License-Expression: PolyForm-Noncommercial-1.0.0
Project-URL: Homepage, https://github.com/jamesburton/poison-vectors
Project-URL: Repository, https://github.com/jamesburton/poison-vectors
Project-URL: Issues, https://github.com/jamesburton/poison-vectors/issues
Keywords: vector-search,embeddings,relevance-feedback,retrieval,rag
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Provides-Extra: real
Requires-Dist: sentence-transformers>=2.7; extra == "real"
Requires-Dist: datasets>=2.19; extra == "real"
Provides-Extra: plot
Requires-Dist: matplotlib>=3.8; extra == "plot"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# Poison Vectors

**A vector-store MCP server with built-in negative-feedback learning.** Index
documents, search semantically — and when a result is a *false match*, flag it so
that **searches similar to the one that found it also exclude it**, *without*
harming queries where that document is genuinely relevant. Backed by your choice of
store: **in-memory**, **flat-file**, **SQLite (sqlite-vec)**, or **Qdrant**.

> "Poison" here is **benign** — a stored *negative-feedback* signal, not the
> adversarial *data-poisoning* attacks the term usually means in ML.

This repository is the full arc: a validated research result, a packaged Python
library, a numerically-certified C# port, and a published **MCP server** an agent
can call over `dnx`.

## The result, in one table

The headline metric is **collateral damage** — does suppressing a false match
*here* harm the queries where that document is relevant? Measured on real homonym
data (CoarseWSD-20, held-out nDCG@10 delta; positive = net benefit, worst = the
collateral floor):

| Mechanism | λ=1 mean / worst | λ=5 mean / worst |
|---|---|---|
| naive (global suppression) | −0.115 / −0.83 | −0.648 / **−1.00** |
| query-conditional | +0.001 / −0.31 | −0.027 / **−1.00** |
| **contrastive (separating-direction)** | **+0.003 / −0.23** | **+0.004 / −0.26** |

**Verdict: GO.** Query-conditional poisoning controls the collateral damage that
naive global suppression inflicts; the **contrastive (separating-direction)**
mechanism is the standout — the only one that stays net-positive with a **bounded
worst-case** (never the −1.0 wipeout), robust to its strength parameter. Validated
on a falsifiable synthetic corpus and confirmed on real data (SciFact + homonyms).

Full numbers and methodology: **[Research & Results](docs/research-and-results.md)**.

### Seeing it work (live, through the MCP server)

A word-sense example — "python" the language vs. the snake. Six docs indexed; the
query *"best way to learn python programming for data analysis"* initially ranks
two **snake** docs above a genuinely-relevant programming doc. After one
`record_false_match` on the top snake doc (anchored to a good doc):

| Doc | Before | After feedback | A *dissimilar* query<br>("care for a pet snake") |
|---|---|---|---|
| `prog-1` (language)        | 0.67 (#1) | 0.67 (#1) ✓ | 0.11 |
| `prog-2` (language)        | 0.27 (#4) | **0.16 (#2)** ↑ | 0.16 |
| `snake-2` (flagged)        | 0.35 (#2) | **−2.49 (last)** ↓ | **0.51 (#1)** ✓ |
| `snake-1` (*not* flagged)  | 0.32 (#3) | **−1.31** ↓ *(generalized)* | **0.50 (#2)** ✓ |
| `monty-1` (*not* flagged)  | 0.19 | **−0.59** ↓ *(generalized)* | 0.15 |

Feedback on one doc generalizes to the whole wrong sense **and to similar future
queries**, while the *same* docs stay top-ranked for a dissimilar query where they
are correct — **query-conditional suppression, zero collateral.**

## How it works (the short version)

Each mechanism subtracts a non-negative penalty from the base cosine score. The
winning mechanism penalizes a candidate's lean along the **separating direction**
between the flagged false match and an anchor (the query, or a known good doc),
**gated by how similar the new query is to the one that flagged it**:

```
score(d) = sim(q, d) − λ · maxₚ [ gate(sim(q, q_trigₚ)) · max(0, sim(d, sₚ)) ]
           where sₚ = normalize(bad_p − anchor)
```

So suppression radiates outward from the original query and fades for unrelated
queries — and only along the axis that made the match *wrong*, leaving relevant
neighbours intact. Two empirical corrections are load-bearing in every mechanism:
penalties are **clipped at 0** (never boost a doc) and poisons aggregate by **max**
(not sum). Details: **[Design & Implementation](docs/design-and-implementation.md)**.

## What's in this repo

| Component | What it is | Where |
|---|---|---|
| **`poison-vectors`** (Python, 0.2.0) | The research testbed **and** the packaged library: `PoisonStore` overlay, mechanisms, JSON persistence. NumPy-only core. | `poison_vectors/`, `pyproject.toml` |
| **`PoisonVectors`** (C#, net10.0) | A numerically-certified port of the scoring engine — `VecMath`, the three mechanisms, `PoisonStore`, the JSON persistence envelope. | `csharp/PoisonVectors/` |
| **`PoisonVectors` MCP server** | The agent-facing vector store: an ONNX MiniLM embedder + a **pluggable datastore** (in-memory, flat-file, SqliteVec, or Qdrant) exposed over MCP (stdio), published to NuGet and runnable via **`dnx`**. Semantic search with a negative-feedback (poison) overlay. | `csharp/PoisonVectors.Cli/` |

The Python and C# implementations are kept equivalent by **golden cross-checks**:
the C# rerank reproduces Python's scores to ~1e-9, the persistence envelope
round-trips **both directions** (Python↔C#), and the C# ONNX embedder matches
Python `sentence-transformers/all-MiniLM-L6-v2` to ~6 decimals.

## Quickstart

### Python — research & library

```bash
pip install -e ".[dev]"
python -m pytest -q                                   # 77 tests
python -m poison_vectors.experiments.milestone_a      # synthetic go/no-go
python examples/ann_overlay.py                        # worked ANN-overlay example

pip install -e ".[real]"                              # for real-data runs
python -m poison_vectors.experiments.coarsewsd_run    # real homonyms (CoarseWSD-20)
```

```python
from poison_vectors import PoisonStore, Mechanism

store = PoisonStore(mechanism=Mechanism.CONTRASTIVE, lam=5.0)
store.add_feedback(query_vec, bad_doc_vec, good_doc_vec)      # vectors in
reranked = store.rerank(query_vec, candidates=[(doc_id, vec), ...], k=10)
```

### C# — library & tests

```bash
cd csharp
dotnet test PoisonVectors.sln          # 104 tests (net10.0)
```

### MCP server — run via `dnx` (needs the .NET 10 SDK)

The server is published to NuGet as an [`mcpserver` tool](https://www.nuget.org/packages/PoisonVectors):

```bash
dnx PoisonVectors --yes -- serve        # MCP server over stdio (what a client launches)
dnx PoisonVectors -- embed "some text"  # quick embed check
dnx PoisonVectors -- demo               # before/after rerank demo
```

First run downloads the `all-MiniLM-L6-v2` ONNX model (~90 MB) to a local cache
(`%LOCALAPPDATA%/poison-vectors/models/all-MiniLM-L6-v2`, overridable via
`POISON_VECTORS_MODEL_DIR`).

**Register it in a client** with the built-in `setup` command (interactive by
default; flags for automation):

```bash
dnx PoisonVectors -- setup --client claude --scope project-local --yes
```

This writes/merges a project-scoped `.mcp.json` so Claude Code can launch the
server. MCP tools exposed: `index_document`, `search`, `remove_document`,
`record_false_match` (optional good-doc anchor; returns a record id),
`list_poisons`, `get_poison`, `search_poisons`, `remove_poison`,
`remove_poisons_for_doc`, `clear_poisons`, `save_store`/`load_store` (full corpus+poison snapshot export/import).

Full guides: **[Usage](docs/usage.md)** (Python + C#/MCP).

## Storage providers & configuration

The MCP server persists both the corpus and the poison-record set to a pluggable
datastore. Select it with an environment variable, overridable by a `serve` flag
(precedence: **flag > env var > default**).

| Setting | Env var | `serve` flag | Default |
|---|---|---|---|
| Provider | `POISON_VECTORS_STORE` | `--store` | `flatfile` |
| Flat-file directory | `POISON_VECTORS_STORE_DIR` | `--store-dir` | `~/.poison-vectors` |
| SqliteVec file | `POISON_VECTORS_STORE_FILE` | `--store-file` | `~/.poison-vectors/poison-vectors.db` |
| Qdrant host | `POISON_VECTORS_QDRANT_HOST` | `--qdrant-host` | `localhost` |
| Qdrant port (gRPC) | `POISON_VECTORS_QDRANT_PORT` | `--qdrant-port` | `6334` |
| Qdrant API key | `POISON_VECTORS_QDRANT_API_KEY` | `--qdrant-api-key` | (none) |

Valid providers: `memory`, `flatfile`, `sqlitevec`, `qdrant`. The default, `flatfile`,
persists both the corpus and the poison records under `~/.poison-vectors` — **state is
durable and shared across launches**. `sqlitevec` is **now available** — ANN-backed via
the `sqlite-vec` extension; select it with `--store sqlitevec` (db file at
`POISON_VECTORS_STORE_FILE`, default `~/.poison-vectors/poison-vectors.db`). Selecting
`sqlitevec` downloads the `sqlite-vec` v0.1.9 native extension on first use to
`~/.poison-vectors/native/` (cache dir overridable via `POISON_VECTORS_VEC0_DIR`; point at a prebuilt extension with `POISON_VECTORS_VEC0_PATH`). `qdrant` is **available** — ANN-backed via the official `Qdrant.Client` (gRPC); select it with `--store qdrant`. The `qdrant` provider expects a reachable Qdrant server (e.g. `docker run -p 6334:6334 qdrant/qdrant`).

## Documentation

- **[Research & Results](docs/research-and-results.md)** — concept, research
  question, prior art, methodology, all milestone results, references.
- **[Design & Implementation](docs/design-and-implementation.md)** — architecture,
  mechanism math, metrics, the falsifiable synthetic corpus, module map, how to
  extend.
- **[Usage](docs/usage.md)** — install, run experiments, use as a library, the
  C# port, and the MCP server / `dnx` / `setup`.
- **[Roadmap](docs/roadmap.md)** — the productization plan and its delivery status.
- Process artifacts (specs, plans, milestone findings): [`docs/superpowers/`](docs/superpowers/).

## Status

All three roadmap phases are **shipped**: the Python module, the C# port (certified
equivalent), and the MCP server (published, `dnx`-distributed, exercised live in
Claude Code). The server is now being expanded into a **pluggable vector store**:
the `IVectorStore` abstraction, addressable poison records (per-id and per-doc
removal, `get`/`search`), `remove_document`, and env+CLI store configuration are in
place; the flat-file provider is now durable (v2 snapshot: corpus + poison records persist
under `~/.poison-vectors`); the SqliteVec provider is now available (ANN via
`sqlite-vec`; `--store sqlitevec`); the Qdrant provider is now available (ANN via
`Qdrant.Client`; `--store qdrant`). All four providers ship. Licensed under PolyForm Noncommercial 1.0.0 (see `LICENSE`).

## Data & licensing

The synthetic corpus is generated in-process. Real-data loaders download on
demand: **SciFact** (BEIR) and **CoarseWSD-20** (homonyms, **CC BY-NC 4.0,
NonCommercial — research only**; downloaded to a gitignored cache, never
committed). The repository is licensed under the **[PolyForm Noncommercial
1.0.0](LICENSE)** license — noncommercial use and modification are permitted;
no warranty is provided. Future versions may be released under different terms.
