Metadata-Version: 2.4
Name: omni-ingest
Version: 0.1.1
Summary: Universal knowledge ingestion framework for multi-modal AI systems. Ingest documents, audio, and structured data into RAG-ready knowledge bases with domain-aware pipelines.
Author: A4i-tech
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pydantic-settings>=2.14.1
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: sqlalchemy[asyncio]>=2.0.50
Requires-Dist: aiosqlite>=0.22.1
Requires-Dist: fsspec>=2026.4.0
Requires-Dist: fastmcp>=3.4.2
Requires-Dist: adlfs>=2026.5.0
Requires-Dist: qdrant-client>=1.18.0
Requires-Dist: pypdf>=6.13.2
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: python-docx>=1.2.0
Requires-Dist: lancedb>=0.33.0
Requires-Dist: azure-cognitiveservices-speech>=1.50.0
Requires-Dist: azure-ai-translation-text>=2.0.0
Requires-Dist: pydantic-ai-slim[openai]>=2.30.0
Requires-Dist: python-magic>=0.4.27
Requires-Dist: python-magic-bin>=0.4.14 ; sys_platform == 'win32'
Requires-Dist: datamodel-code-generator>=0.63.0
Requires-Dist: jq>=1.11.0
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: logfire>=4.37.0
Requires-Dist: rich>=15.0.0
Requires-Dist: pymupdf>=1.27.2.3
Requires-Dist: sqlmodel>=0.0.39
Requires-Dist: azure-ai-documentintelligence>=1.0.2
Requires-Dist: azure-core[aio]>=1.41.0
Requires-Dist: pydantic-ai-slim[anthropic]>=2.0.0 ; extra == 'anthropic'
Requires-Dist: pydantic-ai-slim[bedrock]>=2.0.0 ; extra == 'bedrock'
Requires-Dist: pydantic-ai-slim[cohere]>=2.0.0 ; extra == 'cohere'
Requires-Dist: pydantic-ai-slim[google]>=2.0.0 ; extra == 'google'
Requires-Dist: pydantic-ai-slim[groq]>=2.0.0 ; extra == 'groq'
Requires-Dist: pydantic-ai-slim[huggingface]>=2.0.0 ; extra == 'huggingface'
Requires-Dist: pydantic-ai-slim[mistral]>=2.0.0 ; extra == 'mistral'
Requires-Dist: pydantic-ai-slim[openrouter]>=2.0.0 ; extra == 'openrouter'
Requires-Dist: pydantic-ai-slim[sentence-transformers]>=2.0.0 ; extra == 'sentence-transformers'
Requires-Dist: azure-search-documents>=12.0.0 ; extra == 'storage'
Requires-Dist: chromadb>=1.5.9 ; extra == 'storage'
Requires-Dist: neo4j>=6.2.0 ; extra == 'storage'
Requires-Dist: pymongo>=4.17.0 ; extra == 'storage'
Requires-Dist: pydantic-ai-slim[voyageai]>=2.0.0 ; extra == 'voyageai'
Requires-Dist: pydantic-ai-slim[xai]>=2.0.0 ; extra == 'xai'
Requires-Python: >=3.11
Project-URL: Documentation, https://omniingest.onrender.com
Project-URL: Homepage, https://omniingest.onrender.com
Project-URL: Source, https://github.com/A4i-tech/OmniIngest
Provides-Extra: anthropic
Provides-Extra: bedrock
Provides-Extra: cohere
Provides-Extra: google
Provides-Extra: groq
Provides-Extra: huggingface
Provides-Extra: mistral
Provides-Extra: openrouter
Provides-Extra: sentence-transformers
Provides-Extra: storage
Provides-Extra: voyageai
Provides-Extra: xai
Description-Content-Type: text/markdown

<div align="center">
  <picture>
    <img src="https://gist.githubusercontent.com/a4i-architect/9f6bf6c997ed2c171aed62e1c1583f2d/raw/ec0034143bb0d9d6906c5cce62b4edd13730a9be/omni_ingest.svg" alt="Omni Ingest">
  </picture>
</div>
<div align="center">
  <h3>Multimodal ingestion that does not dictate your stack</h3>
</div>
<div align="center">
 <a href="https://github.com/A4i-tech/OmniIngest/blob/a4i/main/LICENSE"><img src="https://img.shields.io/github/license/A4i-tech/OmniIngest" alt="License"/></a>
 <a href="https://www.python.org/downloads/release/python-3110/"><img src="https://img.shields.io/badge/python-3.11+-blue.svg" alt="Python Version"/></a>
 <a href="https://pypi.python.org/pypi/omni-ingest"><img src="https://img.shields.io/pypi/v/omni-ingest.svg" alt="PyPI"/></a>
 <a href="https://github.com/A4i-tech/OmniIngest/actions/workflows/main.yaml"><img src="https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/a4i-architect/9f6bf6c997ed2c171aed62e1c1583f2d/raw/omni_ingest.json" alt="Coverage"/></a>
 <a href="https://github.com/A4i-tech/OmniIngest/actions/workflows/main.yaml"><img src="https://github.com/A4i-tech/OmniIngest/actions/workflows/main.yaml/badge.svg?branch=a4i/main" alt="CI"/></a>
</div>

---

OmniIngest is a multimodal ingestion framework for building reusable knowledge pipelines. It turns documents, web pages, images, and audio into validated knowledge items without the faff of stitching together one-off preprocessing scripts.

**Documentation:** [omniingest.onrender.com](https://omniingest.onrender.com/)

## Why use OmniIngest

- **Multimodal processing.** PDF, DOCX, HTML, images, audio, and plain text use one pipeline model.
- **Multiple model providers.** Every model-backed step can select its own Pydantic AI model. Chat and embedding models are configured independently.
- **Storage-independent pipelines.** YAML describes transformation. CLI or Python caller chooses JSON, vector store, graph store, or another output.
- **Structured when needed.** JSON Schema output, jq transforms, and step validation turn model results into predictable data.
- **Built for sizeable jobs.** fsspec resources, concurrent stages, progress events, and stage checkpoints keep long ingestion runs manageable.

## Quick start

Install command-line tool with uv:

```bash
uv tool install omni-ingest
```

Create a pipeline which extracts text and makes sentence-aware chunks:

```yaml title="school_notes.yaml"
pipeline_id: school_notes
domain_profile: education

steps:
  - agent: text
  - agent: chunking
    config:
      chunk_size: 900
      overlap: 120
```

Run it against a PDF:

```bash
omni-ingest school_notes.yaml \
  --input lessons/soil-erosion.pdf \
  --output build/soil-erosion.json
```

YAML stops at transformation. Pick storage at run time:

```bash
omni-ingest school_notes.yaml --input lessons/soil-erosion.pdf --output - | jq '.items | length'
omni-ingest retrieval.yaml --input handbooks/*.pdf --output vector:default
```

## Learn more

- [Command-line usage](https://omniingest.onrender.com/cli/) covers files, remote resources, stdin, outputs, and resume.
- [Pipeline playbook](https://omniingest.onrender.com/pipelines/) builds from simple YAML to structured output, validation, components, and model tools.
- [Working with event stream](https://omniingest.onrender.com/events/) covers progress and live application integration.
- [API reference](https://omniingest.onrender.com/api/) documents Python interfaces.

## Development

```bash
git clone https://github.com/A4i-tech/OmniIngest.git
cd OmniIngest
uv sync --dev
uv run pytest tests/unit
```
