Metadata-Version: 2.4
Name: koji-ingest
Version: 0.5.0
Summary: Ingestion pipeline for the koji-db database: parsing, chunking, and multi-vector embeddings
Project-URL: Homepage, https://github.com/tkr-projects/tkr-koji
Project-URL: Repository, https://github.com/tkr-projects/tkr-koji
Project-URL: Source, https://github.com/tkr-projects/tkr-koji/tree/main/koji-ingest
Author: Koji Team
License-Expression: Apache-2.0
Keywords: chunking,docling,embeddings,ingestion,koji,rag,vector
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: structlog>=23.1
Provides-Extra: colnomic
Requires-Dist: colpali-engine<1,>=0.3.9; extra == 'colnomic'
Requires-Dist: pillow>=10; extra == 'colnomic'
Requires-Dist: torch>=2.2; extra == 'colnomic'
Requires-Dist: transformers<6,>=4.45; extra == 'colnomic'
Provides-Extra: cuda
Requires-Dist: accelerate>=0.30; extra == 'cuda'
Requires-Dist: bitsandbytes>=0.43; (platform_system == 'Linux' and platform_machine == 'x86_64') and extra == 'cuda'
Provides-Extra: dev
Requires-Dist: pillow>=10; extra == 'dev'
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: document
Requires-Dist: docling-core>=2.3; extra == 'document'
Requires-Dist: docling<3,>=2.94; extra == 'document'
Provides-Extra: enrichment
Requires-Dist: mlx-embeddings>=0.1.0; extra == 'enrichment'
Requires-Dist: mlx-vlm>=0.4.3; extra == 'enrichment'
Requires-Dist: mlx>=0.20; extra == 'enrichment'
Provides-Extra: koji
Requires-Dist: pyarrow>=14.0; extra == 'koji'
Provides-Extra: rendering
Requires-Dist: pdf2image>=1.16; extra == 'rendering'
Description-Content-Type: text/markdown

# koji-ingest

Ingestion pipeline for Kōji — document parsing, chunking, multi-vector embeddings, and VLM enrichment.

koji-ingest transforms raw documents (PDF, DOCX, PPTX, audio, images, and more) into ColBERT-style multi-vector embeddings ready for storage and semantic search in Kōji.

## Installation

The distribution is published to PyPI as `koji-ingest`; the import module is `koji_ingest`.

```bash
# Core ingestion (parsing, chunking, embeddings)
pip install koji-ingest

# With document parsing support (Docling, PDF)
pip install "koji-ingest[document]"

# With LibreOffice-based rendering (DOCX/PPTX page images)
pip install "koji-ingest[rendering]"

# With MLX VLM enrichment engine (Apple Silicon)
pip install "koji-ingest[enrichment]"

# With Kōji storage integration (PyArrow)
pip install "koji-ingest[koji]"

# With the ColNomic multi-vector engine (colpali-engine + torch + transformers)
pip install "koji-ingest[colnomic]"

# Linux + NVIDIA only: enable 4bit/8bit quantization (bitsandbytes)
pip install "koji-ingest[colnomic,cuda]"
```

### Embedding-engine install matrix

| Platform | Install | Supported `quantization` |
| --- | --- | --- |
| Apple Silicon (MPS) | `pip install "koji-ingest[colnomic]"` | `fp16`, `bf16`, `fp32` |
| CPU-only | `pip install "koji-ingest[colnomic]"` | `fp16`, `bf16`, `fp32` |
| Linux + NVIDIA CUDA | `pip install "koji-ingest[colnomic,cuda]"` | `fp16`, `bf16`, `fp32`, `4bit`, `8bit` |

`bitsandbytes` is CUDA-only, so the `[cuda]` extra is a no-op on macOS — requesting `quantization="4bit"` on a non-CUDA device raises a clear error at config-validation time rather than at model load.

## Features

### Multi-Vector Embeddings

koji-ingest generates ColBERT-style multi-vector embeddings where each document is represented by a set of per-token vectors, enabling MaxSim late-interaction scoring:

```python
from koji_ingest import MultiVectorEmbedding
import numpy as np

data = np.random.randn(10, 128).astype(np.float32)
emb = MultiVectorEmbedding(num_tokens=10, dim=128, data=data)

blob = emb.to_blob()          # serialize for Lance storage
recovered = MultiVectorEmbedding.from_blob(blob)
```

### Document Parsing and Chunking

Parse and chunk documents in a variety of formats via Docling:

```python
from koji_ingest import parse, chunk, IngestConfig

config = IngestConfig()
parsed = parse("report.pdf", config=config)
chunks = chunk(parsed, config=config)
```

Supported formats include PDF, DOCX, PPTX, Markdown, HTML, images, and audio files.

### DenseTextEngine (Lightweight Embedding)

For text-only workloads without vision, use `DenseTextEngine` backed by `mlx-embeddings`:

```python
from koji_ingest import DenseTextEngine

engine = DenseTextEngine()
embedding = engine.embed("The quick brown fox")
```

### Gemma 4 E4B VLM Enrichment (Apple Silicon)

Augment parsed documents with VLM-generated descriptions, code analysis, formula interpretations, and document summaries using the Gemma 4 E4B model via `mlx-vlm`:

```python
from koji_ingest.enrichment import GemmaEnrichmentEngine

engine = GemmaEnrichmentEngine()
enriched = await engine.enrich(parsed_content)
# enriched.figures now contain model-generated captions
# enriched.summary contains a document-level summary
```

### GPU / Metal Memory Management

Release cached GPU or Metal memory between batch operations to prevent OOM errors:

```python
from koji_ingest.gpu import release_gpu_memory

for batch in large_corpus:
    process(batch)
    release_gpu_memory()   # returns allocations to OS after each batch
```

### LibreOffice Page Renderer

Convert DOCX and PPTX slides to page images for visual ingestion:

```python
from koji_ingest.parser import parse

# Renders pages via LibreOffice headless, returns per-page images in ParsedContent
result = parse("slides.pptx", config=config)
```

### Audio Chunking

Ingest long-form audio with automatic chunking:

```python
from koji_ingest import IngestConfig, AudioConfig

config = IngestConfig(audio=AudioConfig(chunk_duration_secs=30))
result = parse("interview.mp3", config=config)
```

### Full Ingestion Pipeline

```python
from koji_ingest import Ingester, IngestConfig

ingester = Ingester(config=IngestConfig())
result = await ingester.ingest("document.pdf")

print(result.chunks)       # List[TextChunk]
print(result.embeddings)   # List[MultiVectorEmbedding]
```

## License

Apache-2.0
