Metadata-Version: 2.4
Name: koji-ingest
Version: 0.1.0rc0
Summary: Ingestion pipeline for the koji-db database: parsing, chunking, and multi-vector embeddings
Project-URL: Homepage, https://github.com/tuckertucker/tkr-koji
Project-URL: Repository, https://github.com/tuckertucker/tkr-koji
Project-URL: Source, https://github.com/tuckertucker/tkr-koji/tree/main/shikomi
Author: Koji Team
License-Expression: Apache-2.0
Keywords: chunking,docling,embeddings,ingestion,koji,rag,vector
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: structlog>=23.1
Provides-Extra: dev
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: document
Requires-Dist: docling-core>=2.3; extra == 'document'
Requires-Dist: docling<3,>=2.81; extra == 'document'
Provides-Extra: enrichment
Requires-Dist: mlx-embeddings>=0.1.0; extra == 'enrichment'
Requires-Dist: mlx-vlm>=0.4.3; extra == 'enrichment'
Requires-Dist: mlx>=0.20; extra == 'enrichment'
Provides-Extra: koji
Requires-Dist: pyarrow>=14.0; extra == 'koji'
Provides-Extra: rendering
Requires-Dist: pdf2image>=1.16; extra == 'rendering'
Description-Content-Type: text/markdown

# Shikomi

Ingestion pipeline for Koji - parsing, chunking, and multi-vector embeddings.

## Installation

```bash
pip install shikomi
```

## Usage

```python
from shikomi import MultiVectorEmbedding, TextChunk, IngestConfig
import numpy as np

# Create an embedding
data = np.random.randn(10, 128).astype(np.float32)
emb = MultiVectorEmbedding(num_tokens=10, dim=128, data=data)

# Serialize to BLOB format for storage
blob = emb.to_blob()

# Deserialize from BLOB
recovered = MultiVectorEmbedding.from_blob(blob)
```

## License

MIT
