Metadata-Version: 2.4
Name: qdrant-lens
Version: 0.1.11
Summary: A knowledge browser and audit layer for Qdrant vector databases
Project-URL: Homepage, https://github.com/Shaik-Hamzah123/qdrant-lens
Project-URL: Repository, https://github.com/Shaik-Hamzah123/qdrant-lens
Project-URL: Bug Tracker, https://github.com/Shaik-Hamzah123/qdrant-lens/issues
License: MIT
Requires-Python: >=3.12
Requires-Dist: aiosqlite>=0.22.1
Requires-Dist: apscheduler<4,>=3.10
Requires-Dist: bcrypt>=5.0.0
Requires-Dist: fastapi>=0.139.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: python-multipart>=0.0.32
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: qdrant-client>=1.18.0
Requires-Dist: rich>=15.0.0
Requires-Dist: sqlalchemy>=2.0.51
Requires-Dist: typer>=0.26.8
Requires-Dist: uvicorn>=0.51.0
Requires-Dist: whenever>=0.10.1
Provides-Extra: all
Requires-Dist: asyncpg>=0.31.0; extra == 'all'
Requires-Dist: chonkie[all]>=1.7.0; extra == 'all'
Requires-Dist: deepagents>=0.6.12; extra == 'all'
Requires-Dist: fastembed>=0.8.0; extra == 'all'
Requires-Dist: langchain-litellm>=0.7.0; extra == 'all'
Requires-Dist: langchain-openai>=1.3.4; extra == 'all'
Requires-Dist: langchain>=1.3.12; extra == 'all'
Requires-Dist: langgraph>=1.2.8; extra == 'all'
Requires-Dist: liteparse>=2.5.0; extra == 'all'
Requires-Dist: neo4j>=6.2.0; extra == 'all'
Requires-Dist: pathspec>=1.1.1; extra == 'all'
Requires-Dist: pdfitdown>=4.2.0; extra == 'all'
Requires-Dist: pdfplumber>=0.11.10; extra == 'all'
Requires-Dist: pillow>=10.0.0; extra == 'all'
Requires-Dist: pymupdf>=1.28.0; extra == 'all'
Requires-Dist: pytesseract>=0.3.13; extra == 'all'
Provides-Extra: chat
Requires-Dist: deepagents>=0.6.12; extra == 'chat'
Requires-Dist: langchain-litellm>=0.7.0; extra == 'chat'
Requires-Dist: langchain-openai>=1.3.4; extra == 'chat'
Requires-Dist: langchain>=1.3.12; extra == 'chat'
Requires-Dist: langgraph>=1.2.8; extra == 'chat'
Provides-Extra: dev
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: embed
Requires-Dist: fastembed>=0.8.0; extra == 'embed'
Provides-Extra: graph
Requires-Dist: neo4j>=6.2.0; extra == 'graph'
Provides-Extra: ingest
Requires-Dist: chonkie[all]>=1.7.0; extra == 'ingest'
Requires-Dist: fastembed>=0.8.0; extra == 'ingest'
Requires-Dist: liteparse>=2.5.0; extra == 'ingest'
Requires-Dist: pathspec>=1.1.1; extra == 'ingest'
Requires-Dist: pdfitdown>=4.2.0; extra == 'ingest'
Requires-Dist: pdfplumber>=0.11.10; extra == 'ingest'
Requires-Dist: pillow>=10.0.0; extra == 'ingest'
Requires-Dist: pymupdf>=1.28.0; extra == 'ingest'
Requires-Dist: pytesseract>=0.3.13; extra == 'ingest'
Provides-Extra: liteparse
Requires-Dist: liteparse>=2.5.0; extra == 'liteparse'
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.31.0; extra == 'postgres'
Description-Content-Type: text/markdown

# qdrant-lens

![PyPI](https://img.shields.io/pypi/v/qdrant-lens) ![Python](https://img.shields.io/badge/python-%3E%3D3.12-blue) ![License](https://img.shields.io/badge/license-MIT-green)

**A knowledge browser and audit layer for Qdrant vector databases.**

![qdrant-lens](assets/asset_1.png)

Qdrant's own dashboard is a point browser — it shows raw vectors and payloads. qdrant-lens sits above it: it shows developers *what the team has actually ingested*, organised by source and document type, with a full audit trail, schema enforcement, and an LLM-powered knowledge layer (OKF). It is a drop-in wrapper around `QdrantClient` — existing code requires no changes.

---

## Features

- **Knowledge tree** — browse ingested content by source → document type → document → chunk, not by raw vector ID; click any chunk to inspect its full `_lens` metadata
- **Audit layer** — every upsert and delete is intercepted, signed with the actor identity and timestamp, and written to an append-only sidestore (SQLite or Postgres); nothing in the cluster changes without a trace
- **`_lens` schema enforcement** — every point carries a validated metadata namespace; violations are surfaced before bad data reaches the index
- **Sync-aware ingest** — chunk-hash dedup means re-ingesting a folder only re-embeds what actually changed; unchanged files are skipped silently
- **Hybrid search** — dense + BM25 or SPLADE sparse vectors via FastEmbed; pass `--bm25` at the CLI or `bm25_enable=True` on the client
- **Parser-agnostic pipeline** — LiteParse (default, Rust-core) handles PDFs, Office docs, images, and Jupyter notebooks; falls back to a pdfplumber/pytesseract stack automatically; `.ipynb` routed to a dedicated notebook parser
- **Open Knowledge Format (OKF)** — auto-generates `.okf.md` concept sidecars per file using a deepagents LLM pass; directory-level agent calls rebuild `index.md` with correct type grouping and handle orphan cleanup; `lens okf` CLI manages the knowledge base independently of ingest
- **Dashboard UI** — React frontend served at port 2873; tabs for Knowledge Tree, Upload, History, Analytics, Graph, and Schedules; upload accepts drag-and-drop or Cmd/Ctrl+V clipboard paste; connect by entering the API URL and API key — the header shows your username and role
- **RBAC user management** — four roles (viewer/developer/owner/admin), per-user bcrypt API keys, two-step provisioning via `lens user bootstrap / create / activate`
- **Chat agent** — ask questions about a collection in plain English, backed by live tree/history/search tools; destructive operations require explicit typed confirmation
- **Export** — dump any collection to portable JSONL or JSON (vectors + metadata) with `lens export`

---

## Install

```bash
pip install qdrant-lens                    # core: client wrapper + dashboard API
pip install "qdrant-lens[ingest]"          # + PDF/OCR/chunking pipeline (LiteParse included)
pip install "qdrant-lens[embed]"           # + FastEmbed (--embed-model CLI flag, sparse vectors)
pip install "qdrant-lens[chat]"            # + LangGraph chat agent + OKF generation
pip install "qdrant-lens[all]"             # everything
```

`uv` is recommended for projects: `uv add "qdrant-lens[ingest,embed,chat]"`. See [docs/quickstart.md](docs/quickstart.md) for Docker and `.env` setup.

---

## Quickstart

### Day 1 — browse an existing collection (zero code changes)

```bash
pip install qdrant-lens
lens serve --url http://localhost:6333
# UI at http://localhost:2873
# Knowledge Tree renders from your existing collection immediately
```

### Day 2 — drop-in client with audit log

```python
from qdrant_lens import QdrantLensClient

# Drop-in replacement for QdrantClient — all existing calls unchanged
client = QdrantLensClient(
    url="http://localhost:6333",
    lens_actor="my-pipeline",   # labels every audit event
)

# Existing calls are intercepted, validated, and audited automatically
client.upsert(collection_name="docs", points=[...])

# New lens-only methods
tree       = client.tree("docs")
events     = client.history("docs", last_n=20)
violations = client.validate("docs")
```

Async variant:

```python
from qdrant_lens import AsyncQdrantLensClient

client = AsyncQdrantLensClient(url="...", lens_actor="pipeline")
tree = await client.tree("docs")
```

### Ingest a folder

```bash
# Basic — discover, parse, chunk, embed, upsert
lens ingest folder ./my-docs \
  --collection my-collection \
  --source github \
  --embed-model BAAI/bge-small-en-v1.5 \
  --actor ci-pipeline

# With BM25 hybrid search
lens ingest folder ./my-docs --collection my-collection --source local \
  --embed-model BAAI/bge-small-en-v1.5 --bm25

# With OKF knowledge sidecars (requires [chat] + LLM API key)
lens ingest folder ./my-docs --collection my-collection --source local \
  --embed-model BAAI/bge-small-en-v1.5 \
  --okf-generate --okf-model openai/gpt-4.1-mini

# In-memory Qdrant — test without a running server
lens ingest folder ./my-docs --collection test --source local \
  --embed-model BAAI/bge-small-en-v1.5 --memory
```

Or from Python:

```python
from fastembed import TextEmbedding
from qdrant_lens import QdrantLensClient

client = QdrantLensClient(url="http://localhost:6333")
model  = TextEmbedding("BAAI/bge-small-en-v1.5")

results = client.ingest_folder(
    "./my-docs",
    collection_name="my-collection",
    source="github",
    embed_fn=lambda texts: [v.tolist() for v in model.embed(texts)],
    bm25_enable=True,
    okf_generate=True,
    okf_model="openai/gpt-4.1-mini",
)
```

---

## CLI reference

| Command | Description |
|---------|-------------|
| `lens serve` | Start the dashboard API + UI at `http://localhost:2873` |
| `lens collections` | List all collections as a table |
| `lens tree` | Print the knowledge tree for a collection |
| `lens history` | Show the ingestion audit log for a collection |
| `lens validate` | Check `_lens` schema on all points; exit 1 on violations |
| `lens export <collection>` | Export points to JSONL or JSON (vectors + metadata) |
| `lens init` | Scaffold `.lensignore`, `lens.yaml`, and `SETUP_GUIDE.md` |
| `lens migrate` | Migrate unstructured points to `_lens` schema (Phase 2) |
| **Ingest subcommands** | |
| `lens ingest folder <path>` | Ingest all supported files in a folder |
| `lens ingest file <path>` | Ingest a single file |
| `lens ingest status` | Show last ingest run summary from audit history |
| `lens ingest list-models` | List available FastEmbed embedding models |
| **OKF subcommands** | |
| `lens okf prune [folder]` | Delete `.okf.md` sidecars whose source file no longer exists |
| `lens okf fix-index [folder]` | Rebuild `index.md` from existing sidecar frontmatter |
| `lens okf update <file>` | Refresh one sidecar's description, tags, and body |

Key flags for `lens ingest folder`:

| Flag | Description |
|------|-------------|
| `--collection` / `-c` | Target collection name |
| `--source` / `-s` | Source label (e.g. `github`, `gdrive`) |
| `--embed-model` / `-m` | FastEmbed model (e.g. `BAAI/bge-small-en-v1.5`) |
| `--actor` | Actor label written to every audit event |
| `--bm25` | Enable BM25 sparse vectors |
| `--sparse-model` | Custom sparse model (overrides `--bm25`) |
| `--okf-generate` | Generate OKF sidecars via LLM |
| `--okf-model` | LiteLLM model string for OKF (e.g. `openai/gpt-4.1-mini`) |
| `--memory` | Use in-memory Qdrant — no server needed |
| `--parser` | `liteparse` (default) or `auto` |
| `--chunk-size` | Tokens per chunk (default 512) |
| `--overlap` | Token overlap between chunks (default 50) |

---

## Open Knowledge Format (OKF)

OKF auto-generates `.okf.md` YAML+markdown sidecars alongside each ingested file, acting as LLM-readable concept cards for RAG systems. Each sidecar contains a structured frontmatter block (`type`, `title`, `description`, `tags`, `resource`, `timestamp`) and a short prose summary.

During `lens ingest folder`, a single agent call per directory processes all added, updated, and deleted files in one pass and rebuilds `index.md` with correct type grouping. Orphaned sidecars (source file deleted) are detected and removed automatically.

Standalone management:

```bash
lens okf prune ./my-docs               # delete sidecars with no source file (--dry-run to preview)
lens okf fix-index ./my-docs/api/      # rebuild index.md from existing sidecar frontmatter
lens okf update ./my-docs/api/auth.py  # refresh one sidecar from current file content
```

Or via the Python API:

```python
from qdrant_lens.ingest.okf_editor import OKFEditor

editor = OKFEditor(root_dir="./my-docs", model="openai/gpt-4.1-mini")
editor.prune_orphans(dry_run=True)          # list orphans without deleting
editor.fix_index()                           # rebuild all index.md files under root
editor.update_sidecar("./my-docs/api/auth.py")
```

See [docs/okf.md](docs/okf.md) for the full schema reference and deletion behaviour.

---

## Dashboard UI

`lens serve` starts the FastAPI backend on port 2873 and serves the React frontend at `/`. Connect by entering the API URL and optional `LENS_API_KEY` on the connect screen — the header then shows the actor name from `LENS_ACTOR`.

| Tab | What it shows |
|-----|---------------|
| Knowledge Tree | source → doc_type → document → chunk hierarchy; Chunk Inspector; hover-to-delete |
| Upload | Drag-and-drop or Cmd/Ctrl+V paste files; configure collection, source, embed model; per-file results |
| History | Append-only audit log with time, action, actor, and chunk count |
| Analytics | Search query tracking and retrieval metrics |
| Graph | Document relationship graph (payload-derived, no Neo4j required for basic graph) |
| Schedules | View and manage scheduled ingest jobs from `lens.yaml` |

---

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `QDRANT_URL` | `http://localhost:6333` | Qdrant instance URL |
| `QDRANT_API_KEY` | — | Qdrant API key |
| `LENS_API_KEY` | — | Bootstrap admin key — used as the single auth key until the first RBAC user is created via `lens user bootstrap`; has no effect once the `lens_users` table is non-empty |
| `LENS_DB_URL` | `sqlite:///lens.db` | Sidestore database (SQLite or Postgres URL) |
| `LENS_ACTOR` | `pipeline` | Actor name written to audit events in bootstrap mode only; once RBAC users exist, the authenticated username is used instead |
| `LENS_CHAT_MODEL` | `openai/gpt-4.1-mini` | LLM for the chat agent and OKF generation |
| `OPENAI_API_KEY` | — | Required for OpenAI-hosted OKF/chat models |
| `NEO4J_URL` | — | Neo4j bolt URL (Graph RAG only) |
| `NEO4J_PASSWORD` | — | Neo4j password (Graph RAG only) |

---

## Documentation

| Guide | Description |
|-------|-------------|
| [Quickstart](docs/quickstart.md) | Install, first run, Day 1/Day 2, env setup |
| [Ingest Pipeline](docs/ingest.md) | File ingestion, parsers, chunking, dedup, hybrid search |
| [Client Reference](docs/client.md) | QdrantLensClient API, audit log, schema, core concepts |
| [Dashboard](docs/dashboard.md) | `lens serve`, UI views, FastAPI routes, CLI commands |
| [UI Guide](docs/ui-guide.md) | Dashboard walkthrough — Knowledge Tree, Upload, History, Analytics |
| [OKF Reference](docs/okf.md) | Open Knowledge Format — sidecars, editor, CLI, deletion behaviour |
| [Export & Migration](docs/export.md) | Export collections to JSONL/JSON; migration recipes |
| [Parsers](docs/parsers.md) | `liteparse` vs `auto`, system deps, troubleshooting |
| [Testing](docs/testing.md) | Test suite overview, per-module test descriptions |
| [Contributing](docs/contributing.md) | Project structure, module map, adding parsers/routes/fields |

---

## Prerequisites

| Requirement | Version | Notes |
|-------------|---------|-------|
| Python | >= 3.12 | |
| Qdrant | >= 1.18.0 | Self-hosted, Docker, or [Qdrant Cloud](https://cloud.qdrant.io) |
| Docker | any | For `docker compose up` (optional). **Docker Hub login required** — run `docker login` first, or use the [Qdrant binary](https://github.com/qdrant/qdrant/releases) or `brew install qdrant` |
| Tesseract | any | OCR only — `sudo apt install tesseract-ocr` / `brew install tesseract`. Set `TESSDATA_PREFIX` if OCR fails at runtime |
| LibreOffice | any | Office → PDF conversion in the `auto` parser stack |
| Neo4j | >= 6.0 | Graph RAG only — `docker compose --profile graph up` |

---

## Compatibility

- **qdrant-client >= 1.18.0 required.** Earlier versions are not supported.
- **Python >= 3.12.** 3.10 and 3.11 may work but are untested.

See [COMPATIBILITY.md](COMPATIBILITY.md) for the full version matrix.

---

## Built with

| Project | Role |
|---------|------|
| [Chonkie](https://www.chonkie.ai/) | Chunking — token-aware `RecursiveChunker`, `SemanticChunker`, and `CodeChunker` |
| [LiteParse](https://github.com/run-llama/liteparse) | Default parser — PDFs, Office docs, images without a LibreOffice dependency |
| [PdfItDown](https://github.com/AstraBert/PdfItDown) | Office and image → PDF conversion in the `auto` fallback stack |
| [FastEmbed](https://github.com/qdrant/fastembed) | Dense and sparse (BM25/SPLADE) embeddings for the CLI and hybrid search |
| [deepagents](https://github.com/run-llama/deepagents) | Filesystem-scoped LLM agents powering OKF sidecar generation and editing |
| [LangGraph](https://github.com/langchain-ai/langgraph) | Chat agent orchestration for the `POST /chat/{collection}` route |
