Metadata-Version: 2.5
Name: ws-rag-26
Version: 0.2.0
Summary: Composable RAG primitives on Qdrant — a library, a CLI, and an MCP server
Project-URL: Homepage, https://github.com/tenPro4/ws-rag-26
Project-URL: Repository, https://github.com/tenPro4/ws-rag-26
Project-URL: Changelog, https://github.com/tenPro4/ws-rag-26/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/tenPro4/ws-rag-26/issues
Author: tenPro4
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,hybrid-search,mcp,qdrant,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.13
Requires-Dist: langchain-qdrant>=1.1.0
Requires-Dist: langchain-text-splitters>=1.1.2
Requires-Dist: markitdown[pdf]>=0.1.7
Requires-Dist: pydantic>=2.13.4
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: qdrant-client>=1.18.0
Provides-Extra: all
Requires-Dist: cohere>=5.0.0; extra == 'all'
Requires-Dist: fastembed>=0.8.0; extra == 'all'
Requires-Dist: langchain-anthropic>=1.0; extra == 'all'
Requires-Dist: langchain-community>=0.4.0; extra == 'all'
Requires-Dist: langchain-google-genai>=4.0; extra == 'all'
Requires-Dist: langchain-groq>=1.0; extra == 'all'
Requires-Dist: langchain-huggingface>=1.0; extra == 'all'
Requires-Dist: langchain-ollama>=1.0; extra == 'all'
Requires-Dist: langchain-openai>=1.0; extra == 'all'
Requires-Dist: mcp>=2.0.0; extra == 'all'
Requires-Dist: unstructured-client>=0.40.0; extra == 'all'
Requires-Dist: unstructured>=0.20.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: langchain-anthropic>=1.0; extra == 'anthropic'
Provides-Extra: cohere
Requires-Dist: cohere>=5.0.0; extra == 'cohere'
Provides-Extra: fastembed
Requires-Dist: fastembed>=0.8.0; extra == 'fastembed'
Requires-Dist: langchain-community>=0.4.0; extra == 'fastembed'
Provides-Extra: google
Requires-Dist: langchain-google-genai>=4.0; extra == 'google'
Provides-Extra: groq
Requires-Dist: langchain-groq>=1.0; extra == 'groq'
Provides-Extra: huggingface
Requires-Dist: langchain-huggingface>=1.0; extra == 'huggingface'
Provides-Extra: hybrid
Requires-Dist: fastembed>=0.8.0; extra == 'hybrid'
Provides-Extra: mcp
Requires-Dist: mcp>=2.0.0; extra == 'mcp'
Provides-Extra: ollama
Requires-Dist: langchain-ollama>=1.0; extra == 'ollama'
Provides-Extra: openai
Requires-Dist: langchain-openai>=1.0; extra == 'openai'
Provides-Extra: openrouter
Requires-Dist: langchain-openrouter>=0.2.4; extra == 'openrouter'
Requires-Dist: openrouter<1.0.0,>=0.9.2; extra == 'openrouter'
Provides-Extra: unstructured
Requires-Dist: unstructured-client>=0.40.0; extra == 'unstructured'
Requires-Dist: unstructured>=0.20.0; extra == 'unstructured'
Description-Content-Type: text/markdown

# ws-rag-26

Composable RAG primitives on Qdrant, with three ways in: a Python library, a
`wsrag` command, and an MCP server. Ingest documents, search them with metadata
filters, and get plain dicts back.

The design bet: **no service you have to run first**. Qdrant runs embedded in
your process, MarkItDown parses files without a container, and everything else
is optional. A single `pip install` gets you from nothing to a working index.

## Install

```bash
pip install ws-rag-26          # core: embedded Qdrant + MarkItDown
pip install "ws-rag-26[openai]"   # plus an embedding/LLM provider
```

Optional extras: `openai`, `anthropic`, `google`, `ollama`, `groq`,
`openrouter`, `huggingface`, `hybrid` (sparse vectors via fastembed),
`unstructured` (title-aware chunking via an unstructured-api container), `all`.

## Configure

Three environment variables are enough:

```bash
QDRANT_PATH=./qdrant_data     # embedded mode — no server, no Docker
AI_PROVIDER=openai            # applies to both embeddings and the LLM
AI_KEY=sk-...
```

`AI_PROVIDER` and `AI_KEY` are fallbacks; `AI_EMBEDDING_PROVIDER` /
`AI_LLM_PROVIDER` override them per role when you want a different model for
each. A `.env` file is loaded automatically.

Set **exactly one** of `QDRANT_PATH` and `QDRANT_URL`. Both together is refused
rather than resolved: the URL would win and the path — along with everything
already stored in it — would be silently ignored, which reads as data loss.

Anything beyond the three variables goes in YAML or in keyword arguments:

```python
rag = WsRag(
    vectorstore={"distance": "cosine"},   # or euclid / dot / manhattan
    splitter={"chunk_size": 1200},
    retriever={"strategy": "multi-query-hybrid", "top_k": 8},
)
```

`distance` is fixed when the collection is created and cannot be changed
afterwards — pick it before your first ingestion.

Configuration merges in four layers, each winning over the one before:

```
built-in defaults  <  environment  <  YAML file  <  keyword arguments
```

Environment alone must be enough, because an MCP host can only inject env vars.
A checked-in YAML file expresses a stronger intent than the ambient environment,
so it wins; an argument passed in code wins over everything.

## Use

```python
from wsrag import WsRag

with WsRag() as rag:
    rag.ingest(["report.pdf", "notes.md"], domain="finance")

    hits = rag.search("What was Q3 revenue?", domain="finance", top_k=5)
    for hit in hits:
        print(hit["metadata"]["file_name"], "→", hit["content"][:120])
```

Results are plain dicts (`content`, `metadata`, and `score` when a strategy
produces one), so they serialise to JSON without LangChain on the other end.

Use `with` — or call `close()`. In embedded mode Qdrant holds an **exclusive
lock** on its directory until the client is released, including against the next
run of the same script.

### Ingesting

```python
rag.ingest(["a.pdf"], domain="finance", sub_domain="payroll")
rag.ingest(["a.pdf"], force=True)                 # re-ingest regardless
rag.ingest_directory("./docs", recursive=True)    # filtered by loader.extensions
rag.ingest_bytes(uploaded, "report.pdf")          # for uploads and MCP clients
```

Every call returns a JSON-serialisable summary:

```python
{"total": 2, "processed": 1, "skipped": 1, "failed": 0, "chunks_created": 12, "errors": []}
```

**Re-ingestion is cheap and safe.** Two levels of deduplication:

1. The **file hash** — an unchanged file is hashed, recognised, and skipped
   before it is parsed, chunked or embedded. The read is not free; everything
   after it is.
2. **Chunk hashes** — a document that changed in one paragraph re-embeds that
   paragraph, not the whole file. Chunks that vanished from the new version are
   deleted, so edited-away text stops being answerable.

Chunk identity keys on the document's `source`, which is why `ingest_bytes`
records the file name you pass rather than the temp path it writes: a random
path each time would make every re-upload look like a brand new document.

### Searching

```python
rag.search("query")                                  # config defaults
rag.search("query", top_k=10, strategy="multi-query-hybrid")
rag.search("query", filters={"domain": ["finance", "ops"], "fiscal_year": 2024})
```

Different fields AND together; a list within one field ORs. `domain=` and
`sub_domain=` are shorthands that merge into `filters`.

Four strategies: `basic`, `hybrid`, `multi-query-vector`, `multi-query-hybrid`.
The multi-query ones ask the LLM for rephrasings and fuse the results with
Reciprocal Rank Fusion; without an LLM configured they degrade to a single
query rather than failing. Note that dense-vs-hybrid is decided when the store
is built (`use_sparse` plus fastembed), not per query — see the note in
`wsrag/retrieval/strategies.py`.

### Reranking (optional)

A cross-encoder reorders the retrieved candidates far more accurately than
vector similarity can. Off by default — it costs an extra API call per search:

```bash
pip install "ws-rag-26[cohere]"
RERANK_ENABLED=true
RERANK_KEY=...        # or COHERE_API_KEY
```

The key alone does not switch it on. A Cohere key is often present for something
else, and reading consent into it would bill you for a stage you never asked
for — so the switch stays explicit, in the environment or in code:

```python
rag = WsRag(rerank={"enabled": True})
rag.search("query", top_k=5)                 # fetches 20, returns the best 5
rag.search("query", top_k=5, rerank=False)   # skip it for this call
rag.search("query", top_k=5, rerank=True)    # rerank even with enabled=False
```

With reranking on, the vector search over-fetches `top_k × 4` candidates. That
multiplier is the whole mechanism: a reranker handed exactly `top_k` hits can
only permute them, so no document outside the original `top_k` could ever reach
the answer.

**Reranking never breaks retrieval.** A rate limit, a network failure, an
uninstalled SDK — each degrades to plain truncation, which is exactly what you
would have got without it. It is a quality layer over results that are already
correct, so it is never allowed to turn a working search into an error.

The one exception is `rerank=True`. Naming it explicitly is a request, not a
preference, so if no reranker can be built for it — no API key, an unknown
provider — you get a `ConfigError` saying which. Silence there would be
indistinguishable from success. Configuration is the line: weather is not.

### Metadata

Every chunk carries pipeline-owned system fields (`source`, `file_name`,
`file_hash`, `chunk_index`, `total_chunks`, …) plus domain fields you declare.
With an LLM configured, domain fields are extracted from the document; anything
you pass explicitly is never re-derived.

Declare your own schema in YAML and point `WS_RAG_METADATA_CONFIG` at it:

```yaml
fields:
  - name: domain
    type: string
    values: [finance, production, hr]
  - name: fiscal_year
    type: integer
  - name: parties
    type: list
```

Fields are deliberately **flat**. A hierarchy is expressed as parallel fields
(`domain` + `sub_domain`), never as a path string — a path can only be
prefix-matched, whereas parallel fields filter at any level independently.

### Contextual retrieval (optional)

A chunk pulled out of a document loses what made it findable. Turning this on
embeds each chunk together with a short LLM-written preamble saying what it is
and where it sits, so the vector describes a self-contained passage:

```python
rag = WsRag(contextual={"enabled": True})
rag.ingest(["handbook.pdf"], force=True)   # see below for why force
```

**It costs one LLM call per chunk**, not per document, which is why it is off by
default — switching it on multiplies the price of loading a corpus by its chunk
count. `evals/` exists to measure whether that buys you anything on your own
data before you pay for it.

The calls run concurrently, eight at a time by default, because serially a
thousand-chunk corpus is a thousand round trips end to end. The first call is
sent on its own so the document excerpt — identical in every prompt, and much
larger than the chunk — is in the provider's prompt cache before the rest fan
out. Lower the bound if ingestion starts drawing rate-limit errors; `1` restores
the fully serial behaviour:

```python
rag = WsRag(contextual={"enabled": True, "max_concurrency": 4})
```

The same three settings are reachable from the environment, which is what makes
them usable from an MCP host:

```bash
WS_RAG_CONTEXTUAL_ENABLED=true
WS_RAG_CONTEXTUAL_MAX_CONCURRENCY=4
WS_RAG_CONTEXTUAL_MAX_DOCUMENT_CHARS=8000
```

A chunk whose call fails is embedded as-is rather than failing the document, and
the count of those is logged as a warning — worth watching after raising the
bound, because that is where a rate limit shows up.

Chunk identity is still taken from the original text, deliberately: an LLM
preamble is not reproducible, and hashing it would make every re-ingestion look
like a changed document. The consequence is that flipping this setting does not
re-embed anything by itself — pass `force=True` once after changing it.

### Introspection and maintenance

```python
rag.stats()                      # collection + active component configuration
rag.list_domains()               # {"domain": [...], "sub_domain": [...]}
rag.field_values("domain")       # unique values, for building a filter UI
rag.extract_filters(query)       # what the LLM reads out of a query, no search
rag.get_filter_context(query)    # the same, rendered as a prompt block
rag.delete_by_filter({"domain": "finance"})
rag.delete_by_file_hash(file_hash)
rag.reset()                      # drop and recreate — irreversible
```

`delete_by_filter` refuses an empty filter: Qdrant reads an empty condition list
as *match everything*, so forwarding one would wipe the collection while looking
like a targeted delete.

## Command line

```bash
wsrag ingest ./docs --recursive --domain finance
wsrag list-domains
wsrag search "Q3 revenue" --domain finance --top-k 5
wsrag field-values domain
wsrag stats
```

**stdout is JSON, always** — one object per run, `{"ok": true, ...}` or
`{"ok": false, "error": ...}`. Logs and tracebacks go to stderr, and a failure
also exits non-zero. Both signals are emitted because callers check different
ones: a shell script reads `$?`, a model parses the JSON.

`--filter KEY=VALUE` is repeatable, and repeating a key ORs its values. Values
are read as JSON scalars, which is how a field's type reaches Qdrant intact:

```bash
wsrag search "budget" --filter fiscal_year=2024 --filter domain=finance --filter domain=ops
```

`fiscal_year=2024` is the number 2024, `domain=finance` is the string. This is
not cosmetic — Qdrant's equality match compares by type, so filtering on the
string `"2024"` against a payload holding the number `2024` matches nothing and
reports it as zero results rather than as an error. When a field really does
hold digits as text, quote inside the value: `--filter code='"2024"'`.

`reset`, `delete-by-filter` and `delete-by-file-hash` are deliberately **not**
commands. The usual caller here is a model shelling out, and an irreversible
operation whose blast radius is the whole collection is not something to hand
one. They stay on the Python API.

## MCP server

```bash
pip install "ws-rag-26[mcp]"
```

```json
{
  "mcpServers": {
    "wsrag": {
      "command": "wsrag-mcp",
      "env": { "QDRANT_PATH": "/abs/path/qdrant_data", "AI_PROVIDER": "openai", "AI_KEY": "sk-..." }
    }
  }
}
```

Six tools: `list_domains`, `get_field_values`, `search`, `ingest_document`,
`ingest_text`, `get_stats`. They are thin and deterministic on purpose — routing,
query reformulation and synthesis stay with the host model, which is already an
LLM and already has the conversation. Wrapping an agent inside a tool that an
agent calls means two models negotiating through a JSON boundary: more latency,
more cost, and a decision the user cannot see.

Configuration comes from environment variables only, because that is all an MCP
host can inject — so every optional stage carries its own switch here. The
`search` tool deliberately takes no `rerank` argument, which makes
`RERANK_ENABLED` the only way a host can reach that stage at all:

```json
"env": {
  "QDRANT_PATH": "/abs/path/qdrant_data",
  "AI_PROVIDER": "openai",
  "AI_KEY": "sk-...",
  "RERANK_ENABLED": "true",
  "RERANK_KEY": "...",
  "WS_RAG_CONTEXTUAL_ENABLED": "true",
  "WS_RAG_CONTEXTUAL_MAX_CONCURRENCY": "4"
}
```

Embedded Qdrant locks its directory, so nothing else may hold
the same path while the server runs — including the `wsrag` CLI. Run both
against one `QDRANT_PATH` and whichever starts second fails; use `QDRANT_URL`
if you need them side by side.

### Claude Code

```bash
wsrag install              # ./.claude/ for this project (default)
wsrag install --user       # ~/.claude/ for every project
wsrag install --skill      # or --commands, to install just one
```

Two things land, and they do different jobs:

- **`/wsrag-search`** — a skill carrying the *working rules* the tool
  descriptions have no room for: call `list_domains` before filtering, never
  guess a domain, what to try when a search comes back empty, why `score` is
  `null` and must never be reported as confidence. Its body loads only when
  triggered, and `references/field-reference.md` only when the model reaches
  for it, so none of it costs context until it is needed.
- **`/ingest-document <path>`** and **`/search <question>`** — slash commands.
  Deterministic manual entry points for when you would rather type the
  operation than describe it.

They do not compete. The MCP tool descriptions say *what exists*, the skill says
*how to judge*, the commands say *do exactly this*. Give a command a
`description` in its frontmatter and the model can invoke it too; leave it out
to keep it a human-only path.

A misconfigured server still starts. The problem is reported through the first
tool call the model makes, naming the missing variable — a host that cannot
start a server shows the user far less than a tool result can.

## Embedded vs remote

| | Embedded (`QDRANT_PATH`) | Remote (`QDRANT_URL`) |
|---|---|---|
| Setup | none | a Qdrant server |
| Processes | exactly one at a time | many |
| Payload indexes | ignored (local Qdrant scans) | used |
| Suits | MCP server, notebook, CLI | FastAPI with >1 worker |

Multi-worker deployments must use `QDRANT_URL` — the embedded directory lock is
per-process and there is no way around it.

`docker-compose.yml` brings up both optional services:

```bash
docker compose up -d qdrant        # a Qdrant server on :6333
docker compose up -d               # plus unstructured-api on :8000
```

Then set `QDRANT_URL=http://localhost:6333` and **remove `QDRANT_PATH`** —
wsrag refuses a configuration with both rather than silently picking one,
because the loser takes every document stored in it out of sight.

## Security note

A metadata filter is **not** a security boundary. Anyone who can call `search`
can pass any filter, so every domain in a collection is readable by every
caller. Isolate sensitive material in a separate collection, not behind a
`domain` value.

## Development

```bash
uv sync
uv run pytest                  # unit tests
uv run pytest -m integration   # real embedded Qdrant on a temp directory
uv run ruff check src tests
```

Integration tests are excluded from the default run. They spin up a real Qdrant
and use deterministic hash-based embeddings, so they need no network and no
model download.

```bash
uv run mypy                    # the suite runs clean; py.typed ships the types
uv build                       # sdist + wheel into dist/
```

## License

MIT — see [LICENSE](LICENSE). Changes are recorded in
[CHANGELOG.md](CHANGELOG.md).
