Metadata-Version: 2.5
Name: ws-rag-26
Version: 0.1.0
Summary: Composable RAG primitives on Qdrant — a library, a CLI, and an MCP server
Project-URL: Homepage, https://github.com/tenPro4/ws-rag-26
Project-URL: Repository, https://github.com/tenPro4/ws-rag-26
Project-URL: Changelog, https://github.com/tenPro4/ws-rag-26/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/tenPro4/ws-rag-26/issues
Author: tenPro4
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,hybrid-search,mcp,qdrant,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.13
Requires-Dist: langchain-qdrant>=1.1.0
Requires-Dist: langchain-text-splitters>=1.1.2
Requires-Dist: markitdown[pdf]>=0.1.7
Requires-Dist: pydantic>=2.13.4
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: qdrant-client>=1.18.0
Provides-Extra: all
Requires-Dist: cohere>=5.0.0; extra == 'all'
Requires-Dist: fastembed>=0.8.0; extra == 'all'
Requires-Dist: langchain-anthropic>=1.0; extra == 'all'
Requires-Dist: langchain-community>=0.4.0; extra == 'all'
Requires-Dist: langchain-google-genai>=4.0; extra == 'all'
Requires-Dist: langchain-groq>=1.0; extra == 'all'
Requires-Dist: langchain-huggingface>=1.0; extra == 'all'
Requires-Dist: langchain-ollama>=1.0; extra == 'all'
Requires-Dist: langchain-openai>=1.0; extra == 'all'
Requires-Dist: mcp>=2.0.0; extra == 'all'
Requires-Dist: unstructured-client>=0.40.0; extra == 'all'
Requires-Dist: unstructured>=0.20.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: langchain-anthropic>=1.0; extra == 'anthropic'
Provides-Extra: cohere
Requires-Dist: cohere>=5.0.0; extra == 'cohere'
Provides-Extra: fastembed
Requires-Dist: fastembed>=0.8.0; extra == 'fastembed'
Requires-Dist: langchain-community>=0.4.0; extra == 'fastembed'
Provides-Extra: google
Requires-Dist: langchain-google-genai>=4.0; extra == 'google'
Provides-Extra: groq
Requires-Dist: langchain-groq>=1.0; extra == 'groq'
Provides-Extra: huggingface
Requires-Dist: langchain-huggingface>=1.0; extra == 'huggingface'
Provides-Extra: hybrid
Requires-Dist: fastembed>=0.8.0; extra == 'hybrid'
Provides-Extra: mcp
Requires-Dist: mcp>=2.0.0; extra == 'mcp'
Provides-Extra: ollama
Requires-Dist: langchain-ollama>=1.0; extra == 'ollama'
Provides-Extra: openai
Requires-Dist: langchain-openai>=1.0; extra == 'openai'
Provides-Extra: openrouter
Requires-Dist: langchain-openrouter>=0.2.4; extra == 'openrouter'
Requires-Dist: openrouter<1.0.0,>=0.9.2; extra == 'openrouter'
Provides-Extra: unstructured
Requires-Dist: unstructured-client>=0.40.0; extra == 'unstructured'
Requires-Dist: unstructured>=0.20.0; extra == 'unstructured'
Description-Content-Type: text/markdown

# ws-rag-26

Composable RAG primitives on Qdrant, with three ways in: a Python library, a
`wsrag` command, and an MCP server. Ingest documents, search them with metadata
filters, and get plain dicts back.

The design bet: **no service you have to run first**. Qdrant runs embedded in
your process, MarkItDown parses files without a container, and everything else
is optional. A single `pip install` gets you from nothing to a working index.

## Install

```bash
pip install ws-rag-26          # core: embedded Qdrant + MarkItDown
pip install "ws-rag-26[openai]"   # plus an embedding/LLM provider
```

Optional extras: `openai`, `anthropic`, `google`, `ollama`, `groq`,
`openrouter`, `huggingface`, `hybrid` (sparse vectors via fastembed),
`unstructured` (title-aware chunking via an unstructured-api container), `all`.

## Configure

Three environment variables are enough:

```bash
QDRANT_PATH=./qdrant_data     # embedded mode — no server, no Docker
AI_PROVIDER=openai            # applies to both embeddings and the LLM
AI_KEY=sk-...
```

`AI_PROVIDER` and `AI_KEY` are fallbacks; `AI_EMBEDDING_PROVIDER` /
`AI_LLM_PROVIDER` override them per role when you want a different model for
each. A `.env` file is loaded automatically.

Set **exactly one** of `QDRANT_PATH` and `QDRANT_URL`. Both together is refused
rather than resolved: the URL would win and the path — along with everything
already stored in it — would be silently ignored, which reads as data loss.

Anything beyond the three variables goes in YAML or in keyword arguments:

```python
rag = WsRag(
    vectorstore={"distance": "cosine"},   # or euclid / dot / manhattan
    splitter={"chunk_size": 1200},
    retriever={"strategy": "multi-query-hybrid", "top_k": 8},
)
```

`distance` is fixed when the collection is created and cannot be changed
afterwards — pick it before your first ingestion.

Configuration merges in four layers, each winning over the one before:

```
built-in defaults  <  environment  <  YAML file  <  keyword arguments
```

Environment alone must be enough, because an MCP host can only inject env vars.
A checked-in YAML file expresses a stronger intent than the ambient environment,
so it wins; an argument passed in code wins over everything.

## Use

```python
from wsrag import WsRag

with WsRag() as rag:
    rag.ingest(["report.pdf", "notes.md"], domain="finance")

    hits = rag.search("What was Q3 revenue?", domain="finance", top_k=5)
    for hit in hits:
        print(hit["metadata"]["file_name"], "→", hit["content"][:120])
```

Results are plain dicts (`content`, `metadata`, and `score` when a strategy
produces one), so they serialise to JSON without LangChain on the other end.

Use `with` — or call `close()`. In embedded mode Qdrant holds an **exclusive
lock** on its directory until the client is released, including against the next
run of the same script.

### Ingesting

```python
rag.ingest(["a.pdf"], domain="finance", sub_domain="payroll")
rag.ingest(["a.pdf"], force=True)                 # re-ingest regardless
rag.ingest_directory("./docs", recursive=True)    # filtered by loader.extensions
rag.ingest_bytes(uploaded, "report.pdf")          # for uploads and MCP clients
```

Every call returns a JSON-serialisable summary:

```python
{"total": 2, "processed": 1, "skipped": 1, "failed": 0, "chunks_created": 12, "errors": []}
```

**Re-ingestion is cheap and safe.** Two levels of deduplication:

1. The **file hash** — an unchanged file is hashed, recognised, and skipped
   before it is parsed, chunked or embedded. The read is not free; everything
   after it is.
2. **Chunk hashes** — a document that changed in one paragraph re-embeds that
   paragraph, not the whole file. Chunks that vanished from the new version are
   deleted, so edited-away text stops being answerable.

Chunk identity keys on the document's `source`, which is why `ingest_bytes`
records the file name you pass rather than the temp path it writes: a random
path each time would make every re-upload look like a brand new document.

### Searching

```python
rag.search("query")                                  # config defaults
rag.search("query", top_k=10, strategy="multi-query-hybrid")
rag.search("query", filters={"domain": ["finance", "ops"], "fiscal_year": 2024})
```

Different fields AND together; a list within one field ORs. `domain=` and
`sub_domain=` are shorthands that merge into `filters`.

Four strategies: `basic`, `hybrid`, `multi-query-vector`, `multi-query-hybrid`.
The multi-query ones ask the LLM for rephrasings and fuse the results with
Reciprocal Rank Fusion; without an LLM configured they degrade to a single
query rather than failing. Note that dense-vs-hybrid is decided when the store
is built (`use_sparse` plus fastembed), not per query — see the note in
`wsrag/retrieval/strategies.py`.

### Reranking (optional)

A cross-encoder reorders the retrieved candidates far more accurately than
vector similarity can. Off by default — it costs an extra API call per search:

```bash
pip install "ws-rag-26[cohere]"
RERANK_KEY=...        # or COHERE_API_KEY
```

```python
rag = WsRag(rerank={"enabled": True})
rag.search("query", top_k=5)                 # fetches 20, returns the best 5
rag.search("query", top_k=5, rerank=False)   # skip it for this call
rag.search("query", top_k=5, rerank=True)    # rerank even with enabled=False
```

With reranking on, the vector search over-fetches `top_k × 4` candidates. That
multiplier is the whole mechanism: a reranker handed exactly `top_k` hits can
only permute them, so no document outside the original `top_k` could ever reach
the answer.

**Reranking never breaks retrieval.** A rate limit, a network failure, an
uninstalled SDK — each degrades to plain truncation, which is exactly what you
would have got without it. It is a quality layer over results that are already
correct, so it is never allowed to turn a working search into an error.

The one exception is `rerank=True`. Naming it explicitly is a request, not a
preference, so if no reranker can be built for it — no API key, an unknown
provider — you get a `ConfigError` saying which. Silence there would be
indistinguishable from success. Configuration is the line: weather is not.

### Metadata

Every chunk carries pipeline-owned system fields (`source`, `file_name`,
`file_hash`, `chunk_index`, `total_chunks`, …) plus domain fields you declare.
With an LLM configured, domain fields are extracted from the document; anything
you pass explicitly is never re-derived.

Declare your own schema in YAML and point `WS_RAG_METADATA_CONFIG` at it:

```yaml
fields:
  - name: domain
    type: string
    values: [finance, production, hr]
  - name: fiscal_year
    type: integer
  - name: parties
    type: list
```

Fields are deliberately **flat**. A hierarchy is expressed as parallel fields
(`domain` + `sub_domain`), never as a path string — a path can only be
prefix-matched, whereas parallel fields filter at any level independently.

### Contextual retrieval (optional)

A chunk pulled out of a document loses what made it findable. Turning this on
embeds each chunk together with a short LLM-written preamble saying what it is
and where it sits, so the vector describes a self-contained passage:

```python
rag = WsRag(contextual={"enabled": True})
rag.ingest(["handbook.pdf"], force=True)   # see below for why force
```

**It costs one LLM call per chunk**, not per document, which is why it is off by
default — switching it on multiplies the price of loading a corpus by its chunk
count. `evals/` exists to measure whether that buys you anything on your own
data before you pay for it.

Chunk identity is still taken from the original text, deliberately: an LLM
preamble is not reproducible, and hashing it would make every re-ingestion look
like a changed document. The consequence is that flipping this setting does not
re-embed anything by itself — pass `force=True` once after changing it.

### Introspection and maintenance

```python
rag.stats()                      # collection + active component configuration
rag.list_domains()               # {"domain": [...], "sub_domain": [...]}
rag.field_values("domain")       # unique values, for building a filter UI
rag.extract_filters(query)       # what the LLM reads out of a query, no search
rag.get_filter_context(query)    # the same, rendered as a prompt block
rag.delete_by_filter({"domain": "finance"})
rag.delete_by_file_hash(file_hash)
rag.reset()                      # drop and recreate — irreversible
```

`delete_by_filter` refuses an empty filter: Qdrant reads an empty condition list
as *match everything*, so forwarding one would wipe the collection while looking
like a targeted delete.

## Command line

```bash
wsrag ingest ./docs --recursive --domain finance
wsrag list-domains
wsrag search "Q3 revenue" --domain finance --top-k 5
wsrag field-values domain
wsrag stats
```

**stdout is JSON, always** — one object per run, `{"ok": true, ...}` or
`{"ok": false, "error": ...}`. Logs and tracebacks go to stderr, and a failure
also exits non-zero. Both signals are emitted because callers check different
ones: a shell script reads `$?`, a model parses the JSON.

`--filter KEY=VALUE` is repeatable, and repeating a key ORs its values. Values
are read as JSON scalars, which is how a field's type reaches Qdrant intact:

```bash
wsrag search "budget" --filter fiscal_year=2024 --filter domain=finance --filter domain=ops
```

`fiscal_year=2024` is the number 2024, `domain=finance` is the string. This is
not cosmetic — Qdrant's equality match compares by type, so filtering on the
string `"2024"` against a payload holding the number `2024` matches nothing and
reports it as zero results rather than as an error. When a field really does
hold digits as text, quote inside the value: `--filter code='"2024"'`.

`reset`, `delete-by-filter` and `delete-by-file-hash` are deliberately **not**
commands. The usual caller here is a model shelling out, and an irreversible
operation whose blast radius is the whole collection is not something to hand
one. They stay on the Python API.

## MCP server

```bash
pip install "ws-rag-26[mcp]"
```

```json
{
  "mcpServers": {
    "wsrag": {
      "command": "wsrag-mcp",
      "env": { "QDRANT_PATH": "/abs/path/qdrant_data", "AI_PROVIDER": "openai", "AI_KEY": "sk-..." }
    }
  }
}
```

Six tools: `list_domains`, `get_field_values`, `search`, `ingest_document`,
`ingest_text`, `get_stats`. They are thin and deterministic on purpose — routing,
query reformulation and synthesis stay with the host model, which is already an
LLM and already has the conversation. Wrapping an agent inside a tool that an
agent calls means two models negotiating through a JSON boundary: more latency,
more cost, and a decision the user cannot see.

Configuration comes from environment variables only, because that is all an MCP
host can inject. Embedded Qdrant locks its directory, so nothing else may hold
the same path while the server runs — including the `wsrag` CLI. Run both
against one `QDRANT_PATH` and whichever starts second fails; use `QDRANT_URL`
if you need them side by side.

### Claude Code

```bash
wsrag install              # ./.claude/ for this project (default)
wsrag install --user       # ~/.claude/ for every project
wsrag install --skill      # or --commands, to install just one
```

Two things land, and they do different jobs:

- **`/wsrag-search`** — a skill carrying the *working rules* the tool
  descriptions have no room for: call `list_domains` before filtering, never
  guess a domain, what to try when a search comes back empty, why `score` is
  `null` and must never be reported as confidence. Its body loads only when
  triggered, and `references/field-reference.md` only when the model reaches
  for it, so none of it costs context until it is needed.
- **`/ingest-document <path>`** and **`/search <question>`** — slash commands.
  Deterministic manual entry points for when you would rather type the
  operation than describe it.

They do not compete. The MCP tool descriptions say *what exists*, the skill says
*how to judge*, the commands say *do exactly this*. Give a command a
`description` in its frontmatter and the model can invoke it too; leave it out
to keep it a human-only path.

A misconfigured server still starts. The problem is reported through the first
tool call the model makes, naming the missing variable — a host that cannot
start a server shows the user far less than a tool result can.

## Embedded vs remote

| | Embedded (`QDRANT_PATH`) | Remote (`QDRANT_URL`) |
|---|---|---|
| Setup | none | a Qdrant server |
| Processes | exactly one at a time | many |
| Payload indexes | ignored (local Qdrant scans) | used |
| Suits | MCP server, notebook, CLI | FastAPI with >1 worker |

Multi-worker deployments must use `QDRANT_URL` — the embedded directory lock is
per-process and there is no way around it.

`docker-compose.yml` brings up both optional services:

```bash
docker compose up -d qdrant        # a Qdrant server on :6333
docker compose up -d               # plus unstructured-api on :8000
```

Then set `QDRANT_URL=http://localhost:6333` and **remove `QDRANT_PATH`** —
wsrag refuses a configuration with both rather than silently picking one,
because the loser takes every document stored in it out of sight.

## Security note

A metadata filter is **not** a security boundary. Anyone who can call `search`
can pass any filter, so every domain in a collection is readable by every
caller. Isolate sensitive material in a separate collection, not behind a
`domain` value.

## Development

```bash
uv sync
uv run pytest                  # unit tests
uv run pytest -m integration   # real embedded Qdrant on a temp directory
uv run ruff check src tests
```

Integration tests are excluded from the default run. They spin up a real Qdrant
and use deterministic hash-based embeddings, so they need no network and no
model download.

```bash
uv run mypy                    # the suite runs clean; py.typed ships the types
uv build                       # sdist + wheel into dist/
```

## License

MIT — see [LICENSE](LICENSE). Changes are recorded in
[CHANGELOG.md](CHANGELOG.md).
