Metadata-Version: 2.4
Name: ragmill
Version: 0.4.1
Summary: A lightweight, zero-config local pipeline engine for AI data ingestion, semantic chunking, embeddings, and vector search — with optional cloud backends and a REST API.
Project-URL: Homepage, https://github.com/Abdullahbinaqeel/RAGMill
Project-URL: Documentation, https://abdullahbinaqeel.github.io/RAGMill/
Project-URL: Repository, https://github.com/Abdullahbinaqeel/RAGMill
Project-URL: Issues, https://github.com/Abdullahbinaqeel/RAGMill/issues
Author-email: Abdullah Bin Aqeel <abdulbinaqeel@gmail.com>
License: MIT
License-File: LICENSE
Keywords: data-pipeline,llm-ingestion,onnx,rag,semantic-chunking,vector-embeddings,vector-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Provides-Extra: all
Requires-Dist: beautifulsoup4>=4.12; extra == 'all'
Requires-Dist: fastapi>=0.110.0; extra == 'all'
Requires-Dist: google-genai>=1.0.0; extra == 'all'
Requires-Dist: numpy>=1.22.0; extra == 'all'
Requires-Dist: onnxruntime>=1.14.0; extra == 'all'
Requires-Dist: openai>=1.0.0; extra == 'all'
Requires-Dist: openpyxl>=3.1; extra == 'all'
Requires-Dist: pillow>=10.0; extra == 'all'
Requires-Dist: pinecone>=5.0; extra == 'all'
Requires-Dist: pydantic>=2.0; extra == 'all'
Requires-Dist: pypdf>=4.0; extra == 'all'
Requires-Dist: pytesseract>=0.3.10; extra == 'all'
Requires-Dist: python-docx>=1.0; extra == 'all'
Requires-Dist: python-dotenv>=1.0.0; extra == 'all'
Requires-Dist: python-multipart>=0.0.9; extra == 'all'
Requires-Dist: python-pptx>=1.0; extra == 'all'
Requires-Dist: qdrant-client>=1.9.0; extra == 'all'
Requires-Dist: striprtf>=0.0.26; extra == 'all'
Requires-Dist: tokenizers>=0.15.0; extra == 'all'
Requires-Dist: uvicorn[standard]>=0.27.0; extra == 'all'
Provides-Extra: chat
Requires-Dist: llama-cpp-python>=0.3.0; extra == 'chat'
Provides-Extra: chat-gemini
Requires-Dist: google-genai>=1.0.0; extra == 'chat-gemini'
Provides-Extra: chat-openai
Requires-Dist: openai>=1.0.0; extra == 'chat-openai'
Provides-Extra: config-ui
Requires-Dist: fastapi>=0.110.0; extra == 'config-ui'
Requires-Dist: numpy>=1.22.0; extra == 'config-ui'
Requires-Dist: pydantic>=2.0; extra == 'config-ui'
Requires-Dist: python-dotenv>=1.0.0; extra == 'config-ui'
Requires-Dist: uvicorn[standard]>=0.27.0; extra == 'config-ui'
Provides-Extra: dev
Requires-Dist: beautifulsoup4>=4.12; extra == 'dev'
Requires-Dist: black>=23.0; extra == 'dev'
Requires-Dist: fastapi>=0.110.0; extra == 'dev'
Requires-Dist: google-genai>=1.0.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: numpy>=1.22.0; extra == 'dev'
Requires-Dist: onnxruntime>=1.14.0; extra == 'dev'
Requires-Dist: openai>=1.0.0; extra == 'dev'
Requires-Dist: openpyxl>=3.1; extra == 'dev'
Requires-Dist: pillow>=10.0; extra == 'dev'
Requires-Dist: pinecone>=5.0; extra == 'dev'
Requires-Dist: pydantic>=2.0; extra == 'dev'
Requires-Dist: pypdf>=4.0; extra == 'dev'
Requires-Dist: pytesseract>=0.3.10; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: python-docx>=1.0; extra == 'dev'
Requires-Dist: python-dotenv>=1.0.0; extra == 'dev'
Requires-Dist: python-multipart>=0.0.9; extra == 'dev'
Requires-Dist: python-pptx>=1.0; extra == 'dev'
Requires-Dist: qdrant-client>=1.9.0; extra == 'dev'
Requires-Dist: reportlab>=4.0; extra == 'dev'
Requires-Dist: striprtf>=0.0.26; extra == 'dev'
Requires-Dist: tokenizers>=0.15.0; extra == 'dev'
Requires-Dist: tomli>=2.0; (python_version < '3.11') and extra == 'dev'
Requires-Dist: uvicorn[standard]>=0.27.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Provides-Extra: docx
Requires-Dist: python-docx>=1.0; extra == 'docx'
Provides-Extra: embeddings
Requires-Dist: numpy>=1.22.0; extra == 'embeddings'
Requires-Dist: onnxruntime>=1.14.0; extra == 'embeddings'
Requires-Dist: tokenizers>=0.15.0; extra == 'embeddings'
Provides-Extra: ocr
Requires-Dist: pillow>=10.0; extra == 'ocr'
Requires-Dist: pytesseract>=0.3.10; extra == 'ocr'
Provides-Extra: office
Requires-Dist: beautifulsoup4>=4.12; extra == 'office'
Requires-Dist: openpyxl>=3.1; extra == 'office'
Requires-Dist: python-pptx>=1.0; extra == 'office'
Requires-Dist: striprtf>=0.0.26; extra == 'office'
Provides-Extra: pdf
Requires-Dist: pypdf>=4.0; extra == 'pdf'
Provides-Extra: pinecone
Requires-Dist: numpy>=1.22.0; extra == 'pinecone'
Requires-Dist: pinecone>=5.0; extra == 'pinecone'
Provides-Extra: qdrant
Requires-Dist: numpy>=1.22.0; extra == 'qdrant'
Requires-Dist: qdrant-client>=1.9.0; extra == 'qdrant'
Provides-Extra: server
Requires-Dist: fastapi>=0.110.0; extra == 'server'
Requires-Dist: pydantic>=2.0; extra == 'server'
Requires-Dist: python-dotenv>=1.0.0; extra == 'server'
Requires-Dist: python-multipart>=0.0.9; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.27.0; extra == 'server'
Description-Content-Type: text/markdown

<p align="center">
  <img src="logo.png" alt="RAGMill logo" width="200">
</p>

<h1 align="center">RAGMill</h1>

[![PyPI](https://img.shields.io/pypi/v/ragmill.svg?cacheSeconds=3600)](https://pypi.org/project/ragmill/)
[![CI](https://github.com/Abdullahbinaqeel/RAGMill/actions/workflows/ci.yml/badge.svg)](https://github.com/Abdullahbinaqeel/RAGMill/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python](https://img.shields.io/pypi/pyversions/ragmill.svg)](https://pypi.org/project/ragmill/)

A lightweight, zero-config local pipeline engine for AI data ingestion,
semantic chunking, embeddings, vector search, and retrieval-augmented chat
— fully offline by default (no API keys, ever), with optional cloud
backends (Pinecone, Qdrant), a REST API, a standalone setup UI, and Docker
support.

RAGMill takes a folder of documents and turns it into a searchable,
question-answerable knowledge base in a few lines of Python — or entirely
from the command line. It handles the full RAG pipeline end to end: parsing
files, splitting them into overlapping semantic chunks, generating
embeddings, storing vectors, retrieving the most relevant passages for a
query, and generating grounded, cited answers on top of them. Everything
runs on your machine out of the box; the cloud and hosted-LLM integrations
are strictly opt-in.

The core install has **zero dependencies** and works with `.txt`/`.md` files
straight away. Every heavier capability — PDF/DOCX parsing, local ONNX
embeddings, the local LLM, the REST server, and cloud backends — ships as an
optional extra, so you only install what you actually use.

### Highlights

- **Offline-first, no keys required** — local embeddings (ONNX) and a local
  LLM (Qwen2.5-1.5B via `llama-cpp-python`) run entirely on your machine.
- **Full RAG pipeline** — ingest → chunk (configurable size/overlap) →
  embed → store → semantic search → grounded chat, all in one package.
- **Multiple file formats** — text (`.txt`, `.md`, `.log`, `.rst`), data
  (`.csv`, `.tsv`), documents (`.pdf`, `.docx`, `.rtf`, `.html`), office
  (`.xlsx`, `.pptx`), and images (`.png`, `.jpg`, `.tiff`, …) plus scanned
  PDFs via OCR.
- **Pluggable vector stores** — local SQLite by default; switch to Pinecone
  or Qdrant with a couple of env vars, no code changes.
- **Swappable chat backends** — local LLM, Gemini, or OpenAI, selected at
  runtime via `RAGMILL_CHAT_BACKEND`.
- **Incremental sync** — keep a store in step with a folder, adding,
  updating, and deleting only what changed.
- **Backend migration** — export a local store to JSONL and import it into a
  cloud backend (or vice versa).
- **Multiple interfaces** — a Python API, a `ragmill` CLI, a FastAPI REST
  server (with a browser chatbox), and a standalone setup UI that writes your
  `.env` for you.
- **Docker-ready** — compose profiles for both SQLite and Qdrant.

## Install

```bash
pip install ragmill                          # core only (txt/md), zero dependencies
pip install ragmill[all]                     # everything installable from wheels (PDF, DOCX, embeddings, server, cloud backends)
pip install ragmill[embeddings]              # + local ONNX embeddings
pip install ragmill[chat]                    # + local LLM for retrieval-augmented answers (no API key) — see note below
pip install ragmill[chat-gemini]             # + Gemini as the chat backend (needs GEMINI_API_KEY)
pip install ragmill[chat-openai]             # + ChatGPT as the chat backend (needs OPENAI_API_KEY)
pip install ragmill[pinecone]                # + Pinecone cloud backend
pip install ragmill[qdrant]                  # + Qdrant cloud backend
pip install ragmill[server]                  # + FastAPI REST API
pip install ragmill[config-ui]               # + standalone setup UI (writes .env)
```

> **`[all]` does not include the local LLM.** `llama-cpp-python` publishes no PyPI
> wheels for recent versions, so pip builds it from a 70 MB+ source archive that
> vendors llama.cpp — which needs a C++ toolchain, and on Windows overruns the
> 260-character `MAX_PATH` limit while unpacking:
>
> ```
> ERROR: Could not install packages due to an OSError: [Errno 2]
> No such file or directory: 'C:\\Users\\...\\vendor\\llama.cpp\\tools\\ui\\...'
> ```
>
> Keeping it out of `[all]` means `pip install ragmill[all]` installs from wheels
> alone on every platform. To add the local LLM, install a prebuilt wheel from the
> project's own index — no compiler, no long-path problem:
>
> ```bash
> pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
> pip install ragmill[all]
> ```
>
> `ragmill[chat]` still works if you have a working C++ toolchain (and, on
> Windows, long paths enabled).

## Quick start

### Ingest + chunk + embed + search (local SQLite)

> **Note:** This example requires the `embeddings` extra:
> `pip install ragmill[embeddings]`

```python
from ragmill import RAGEngine
from ragmill.embeddings import EmbeddingModel
from ragmill.vector_store import VectorStore

chunks = RAGEngine().execute_pipeline("./my_documents")

model = EmbeddingModel()
vectors = model.embed([c["content"] for c in chunks])

store = VectorStore("my_store.db")
store.add(chunks, vectors)

query = model.embed(["how does the overlap work?"])[0]
for r in store.search(query, top_k=3):
    print(r["score"], r["metadata"]["filename"], "->", r["content"][:80])
```

### Keep a store in sync with a folder

```python
from ragmill import RAGEngine
from ragmill.embeddings import EmbeddingModel
from ragmill.vector_store import VectorStore
from ragmill.sync import sync_directory

engine = RAGEngine()
model = EmbeddingModel()
store = VectorStore("my_store.db")

result = sync_directory("./my_documents", engine, model, store)
print(result)  # {"added": 2, "updated": 1, "skipped": 40, "deleted": 1}
```

## Ask questions, get grounded answers

Retrieval-augmented answer generation, with a choice of three backends —
selected via `RAGMILL_CHAT_BACKEND` (default `local`):

| Backend | Install | Needs a key? | Notes |
|---|---|---|---|
| `local` (default) | `ragmill[chat]` | No | Qwen2.5-1.5B-Instruct via `llama-cpp-python`. Downloads once (~1.1GB) to `~/.cache/ragmill/models`, then runs fully offline. |
| `gemini` | `ragmill[chat-gemini]` | `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) | Google's Gemini API. Best answer quality if you're online and have a key. |
| `openai` | `ragmill[chat-openai]` | `OPENAI_API_KEY` | OpenAI's Chat Completions API (ChatGPT). |

```bash
pip install ragmill[chat]           # local model (default, no key)
ragmill chat                        # interactive terminal Q&A over your ingested docs
```

```bash
# Switch to Gemini
pip install ragmill[chat-gemini]
export RAGMILL_CHAT_BACKEND=gemini
export GEMINI_API_KEY=xxxxxxxx
ragmill chat

# Or ChatGPT
pip install ragmill[chat-openai]
export RAGMILL_CHAT_BACKEND=openai
export OPENAI_API_KEY=xxxxxxxx
ragmill chat
```

```python
from ragmill.chat import generate_answer
answer = generate_answer("what does the overlap parameter do?", results)
```

All three backends share the same `generate_answer(query, chunks)` call — the
backend is picked at call time from `RAGMILL_CHAT_BACKEND`, so switching is
just an env var change, no code change. The [setup UI](#standalone-setup-ui)
below lets you pick a backend and enter its key without touching the
environment by hand.

Per-backend overrides:
- Local: `RAGMILL_CHAT_MODEL_REPO`, `RAGMILL_CHAT_MODEL_FILE`, `RAGMILL_CHAT_N_CTX`
- Gemini: `RAGMILL_GEMINI_MODEL` (default `gemini-flash-latest`)
- OpenAI: `RAGMILL_OPENAI_MODEL` (default `gpt-4o-mini`)

## Use a cloud vector store

Set environment variables and the store backend switches automatically:

```bash
export RAGMILL_STORE_TYPE=pinecone
export RAGMILL_PINECONE_API_KEY=xxxxxxxx
export RAGMILL_PINECONE_ENVIRONMENT=us-west1-gcp
export RAGMILL_PINECONE_INDEX_NAME=ragmill
```

```python
from ragmill.vector_store import store_from_config
from ragmill.config import RAGMillConfig

config = RAGMillConfig.from_env()
store = store_from_config(config)   # returns a PineconeVectorStore
```

Or for Qdrant (local via Docker, or a managed Qdrant Cloud cluster):

```bash
export RAGMILL_STORE_TYPE=qdrant
export RAGMILL_QDRANT_URL=http://localhost:6333          # or your cloud cluster URL
export RAGMILL_QDRANT_API_KEY=xxxxxxxx                    # required for Qdrant Cloud
export RAGMILL_QDRANT_COLLECTION_NAME=ragmill
```

Payload indexes (`filename`, `source_file`) are created automatically the
first time a collection is set up — Qdrant Cloud rejects filtered
search/delete/sync operations without them, while local/self-hosted Qdrant
is more lenient about it.

## Migrate between backends

Export your local SQLite store to JSONL, then import into a cloud store:

```bash
# 1. Export from SQLite
ragmill export ./backup.jsonl

# 2. Switch env to point at Pinecone
export RAGMILL_STORE_TYPE=pinecone
export RAGMILL_PINECONE_API_KEY=xxx

# 3. Import into Pinecone
ragmill import ./backup.jsonl
```

## REST API

```bash
pip install ragmill[server]
ragmill serve
# or: uvicorn ragmill.server:app --host 0.0.0.0 --port 8000
```

| Method | Path         | Description                             |
|--------|--------------|------------------------------------------|
| POST   | `/ingest`    | Ingest a directory                        |
| POST   | `/sync`      | Incremental sync                          |
| POST   | `/search`    | Search chunks                             |
| POST   | `/chat`      | Ask a question, get a grounded answer     |
| GET    | `/count`     | Number of stored chunks                   |
| POST   | `/export`    | Export store to JSONL                     |
| POST   | `/import`    | Import JSONL into store                   |
| GET    | `/health`    | Health check                              |
| GET    | `/`          | Minimal terminal-style chatbox (test `/chat` in a browser) |

## Standalone setup UI

A separate, minimal web UI for filling in optional config — cloud vector
store credentials, which chat backend to use (local/Gemini/ChatGPT) and its
key, or an override for the local chat model — without hand-editing
anything. It runs as its own server/process (a different port than
`ragmill serve`), so it's clearly a one-time setup tool independent of
wherever RAGMill actually runs as a dependency.

```bash
pip install ragmill[config-ui]
ragmill configure   # http://127.0.0.1:8090 by default — binds to localhost only
```

Fill in what you need and click "Save configuration" — it writes only the
fields you filled in to a local `.env` file (via `python-dotenv`, preserving
any unrelated lines already there). RAGMill loads that `.env` automatically
on the next run. Remember to add `.env` to `.gitignore` — nothing is ever
written into source code.

## Docker

```bash
# SQLite backend
docker compose --profile sqlite up

# Qdrant backend (spins up a Qdrant container too)
docker compose --profile qdrant up
```

## CLI

```bash
ragmill ingest ./docs       # Ingest + embed files
ragmill sync ./docs         # Incremental sync
ragmill search "query"      # Search
ragmill chat                # Interactive Q&A over stored chunks (local LLM)
ragmill count                # Chunk count
ragmill serve               # Start API
ragmill export ./out.jsonl  # Export
ragmill import ./in.jsonl   # Import
ragmill configure           # Standalone setup UI (writes .env)
ragmill --version           # Print the installed version
```

## Configuration

All settings are controlled via environment variables (or a `.env` file —
see the setup UI above):

| Variable | Default | Description |
|---|---|---|
| `RAGMILL_STORE_TYPE` | `sqlite` | `sqlite`, `pinecone`, or `qdrant` |
| `RAGMILL_SQLITE_PATH` | `./ragmill.db` | Path to SQLite database file |
| `RAGMILL_EMBEDDING_MODEL` | `Xenova/all-MiniLM-L6-v2` | Hugging Face model for embeddings |
| `RAGMILL_EMBEDDING_DIM` | `384` | Embedding vector dimension |
| `RAGMILL_PINECONE_API_KEY` | — | Pinecone API key |
| `RAGMILL_PINECONE_ENVIRONMENT` | — | Pinecone environment |
| `RAGMILL_PINECONE_INDEX_NAME` | `ragmill` | Pinecone index name |
| `RAGMILL_QDRANT_URL` | — | Qdrant server/cluster URL |
| `RAGMILL_QDRANT_API_KEY` | — | Qdrant API key (required for Qdrant Cloud) |
| `RAGMILL_QDRANT_COLLECTION_NAME` | `ragmill` | Qdrant collection name |
| `RAGMILL_CHAT_BACKEND` | `local` | `local`, `gemini`, or `openai` |
| `RAGMILL_CHAT_MODEL_REPO` | `Qwen/Qwen2.5-1.5B-Instruct-GGUF` | Local chat model's Hugging Face repo |
| `RAGMILL_CHAT_MODEL_FILE` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Local chat model's GGUF filename |
| `RAGMILL_CHAT_N_CTX` | `4096` | Local chat model's context window (tokens) |
| `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) | — | Required when `RAGMILL_CHAT_BACKEND=gemini` |
| `RAGMILL_GEMINI_MODEL` | `gemini-flash-latest` | Gemini model name |
| `OPENAI_API_KEY` | — | Required when `RAGMILL_CHAT_BACKEND=openai` |
| `RAGMILL_OPENAI_MODEL` | `gpt-4o-mini` | OpenAI model name |
| `RAGMILL_CHUNK_SIZE` | `500` | Max chunk size in characters |
| `RAGMILL_OVERLAP` | `50` | Chunk overlap in characters |
| `RAGMILL_HOST` | `127.0.0.1` | Server bind address |
| `RAGMILL_PORT` | `8000` | Server port |

## Supported file types

| Formats | Extra needed |
|---|---|
| `.txt`, `.md`, `.log`, `.rst`, `.csv`, `.tsv` | none (core) |
| `.pdf` | `ragmill[pdf]` |
| `.docx` | `ragmill[docx]` |
| `.html`, `.htm`, `.rtf`, `.xlsx`, `.pptx` | `ragmill[office]` |
| `.png`, `.jpg`, `.jpeg`, `.tiff`, `.bmp`, `.gif`, and scanned/image PDFs (OCR) | `ragmill[ocr]` |

OCR requires the system `tesseract` binary (and `pdftoppm`/poppler for scanned
PDFs); it is English-only by default. Scanned PDFs with no text layer fall back
to OCR automatically when the `ocr` extra is installed.

## Project structure

```
ragmill/
├── src/ragmill/
│   ├── __init__.py            # Public API exports
│   ├── engine.py              # RAGEngine: ingestion + chunking
│   ├── parsers.py             # PDF/DOCX text extractors
│   ├── embeddings.py          # Local ONNX embedding model
│   ├── chat.py                # Local LLM answer generation (llama-cpp-python)
│   ├── vector_store.py        # BaseVectorStore ABC + SQLiteVectorStore
│   ├── pinecone_store.py      # Pinecone backend (optional)
│   ├── qdrant_store.py        # Qdrant backend (optional)
│   ├── config.py              # RAGMillConfig: env/.env-based configuration
│   ├── sync.py                # Incremental directory sync
│   ├── export.py              # JSONL export/import for migration
│   ├── server.py              # FastAPI REST API + chat UI (optional)
│   ├── static/                # Terminal-style chatbox HTML for server.py's `/`
│   ├── config_ui.py           # Standalone setup UI, separate server (optional)
│   ├── config_ui_static/      # Setup UI's HTML form
│   └── __main__.py            # CLI entry point
├── tests/
├── docs/                      # MkDocs documentation source
├── Dockerfile
└── docker-compose.yml
```

## Documentation

Full documentation is published at
**<https://abdullahbinaqeel.github.io/RAGMill/>**.

- [Quickstart](docs/quickstart.md) · [Installation](docs/installation.md) · [Configuration](docs/guide/configuration.md)
- [CLI](docs/guide/cli.md) · [REST API](docs/guide/rest-api.md) · [Chat](docs/guide/chat.md)
- [Vector stores](docs/guide/vector-stores.md) · [Migration](docs/guide/migration.md)
- [API reference](docs/api-reference.md) · [FAQ](docs/faq.md)

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) to get
started, and note our [Code of Conduct](CODE_OF_CONDUCT.md). Found a security
issue? Please report it privately via [SECURITY.md](SECURITY.md) — not a public
issue. Maintainers: see [RELEASING.md](RELEASING.md) for the release process.

## License

[MIT](LICENSE) © Abdullah Bin Aqeel

