qdrant-lens
Ingest pipeline — file to vector
Showing the OKF branch when okf_enabled=True
1
Entry
Discover
Scan folder, apply .lensignore patterns, filter to supported extensions. Notebooks always go to NotebookParser regardless of any flag.
discovery.py → list[Path]
2
Non-native files
Convert
Office / image / zip → PDF via PdfItDown. Skipped entirely for native text files (.md .py .txt etc.) and when using parser="liteparse".
converter.py
3
Branch point
Route & detect
For .md files: is_okf() peeks at the first 30 frontmatter lines for a non-empty type: field. OKF-detected files take the left lane. All others fall through to the right.
is_okf() — okf_parser.py router.py — auto/liteparse
Parallel extraction paths
◈ OKF path okf_enabled=True
OKFParser.extract() strips YAML frontmatter, returns (body, "okf")
Frontmatter parsed: type, title, description, resource, tags, timestamp
doc_type = "okf" → SemanticChunker
Frontmatter merged into _lens: is_okf=True, okf_type, title, description, resource, okf_timestamp
tags → top-level payload["tags"]
▤ Standard path
LiteParseParser or PlainTextParser (for .md .txt .py etc.)
doc_type set by extension: markdown / code / text / pdf / slide
Chunker chosen per doc_type: Semantic / Code / Recursive / slide_splitter
No OKF fields in _lens
4
Quality gate
Validate extraction
Check for empty, short, noisy, or repetitive text. PDF failures trigger OCR fallback before applying on_extraction_failure policy.
validation.py → ValidationResult
5
Split
Chunk
Split text using the chunker selected by doc_type. Chunkers can be overridden per type via chunker_overrides={"okf": ...}.
chunker.py → list[Chunk]
6
Deduplication
Hash check
chunk_hash = "sha256:" + sha256(text)[:16]. Each hash is checked via Qdrant scroll filter. Existing chunks are skipped — no re-embedding.
pipeline.py → skip if exists
7
Final step
Embed + Upsert
Call embed_fn(texts) → dense vectors. Optionally compute sparse vectors (BM25/SPLADE) if bm25_enable=True. Upsert all new chunks with full _lens payload. Write AuditEvent to sidestore.
QdrantLensClient.upsert() LensSidestore → audit log
✓
Output
Result dict
Returns {"status": "ok", "chunks_upserted": N, "doc_type": "okf", "parser_used": "OKFParser", ...}. OKF chunks are now searchable via _lens.is_okf == True filter in the chat agent.
Branch / decision
Quality gate
Transform
Dedup
Write
Passthrough