Metadata-Version: 2.4
Name: vecbee
Version: 0.1.0
Summary: A lightweight, open-source vector-database migration & infrastructure CLI (Alembic for vector databases).
Project-URL: Homepage, https://github.com/Caoquyen1913/vecbee
Project-URL: Repository, https://github.com/Caoquyen1913/vecbee
Project-URL: Documentation, https://github.com/Caoquyen1913/vecbee#readme
Project-URL: Issues, https://github.com/Caoquyen1913/vecbee/issues
Project-URL: Changelog, https://github.com/Caoquyen1913/vecbee/blob/main/CHANGELOG.md
Author: AI Outsourcing Studio
Author-email: Kawin Tran <caoquyen1913@gmail.com>
Maintainer-email: Kawin Tran <caoquyen1913@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: blue-green,cli,embeddings,migration,rag,vector-database
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: aiosqlite>=0.20
Requires-Dist: langchain-text-splitters>=0.2
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: import-linter>=2.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == 'postgres'
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.9; extra == 'qdrant'
Description-Content-Type: text/markdown

<!-- Logo & assets live in ./assets and are referenced by absolute raw URL so they
     render identically on GitHub and on PyPI (PyPI cannot resolve relative paths). -->
<p align="center">
  <img src="https://raw.githubusercontent.com/Caoquyen1913/vecbee/main/assets/vecbee-logo.svg" alt="vecbee" width="440">
</p>

<h3 align="center">Alembic for vector databases.</h3>

<p align="center">
  Migrate, sync, and re-embed vector data across stores with blue-green
  zero-downtime cutovers — all driven by one declarative <code>vector.config.yaml</code>.
</p>

<p align="center">
  <a href="https://pypi.org/project/vecbee/"><img src="https://img.shields.io/pypi/v/vecbee.svg?color=F59003&label=pypi" alt="PyPI version"></a>
  <a href="https://pypi.org/project/vecbee/"><img src="https://img.shields.io/pypi/pyversions/vecbee.svg?color=3B82F6" alt="Python versions"></a>
  <a href="https://github.com/Caoquyen1913/vecbee/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/vecbee.svg?color=10B981" alt="License: Apache-2.0"></a>
  <a href="https://github.com/Caoquyen1913/vecbee/actions/workflows/ci.yml"><img src="https://github.com/Caoquyen1913/vecbee/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <img src="https://img.shields.io/badge/typed-mypy%20strict-2A6DB0.svg" alt="mypy strict">
</p>

---

## Why vecbee?

Relational databases have had migration tooling for decades — Alembic, Flyway, Rails
migrations. Vector databases have had almost nothing. When your embedding model changes,
your chunking strategy improves, or you outgrow your vector store, you're left writing
one-off scripts, praying nothing breaks, and taking downtime while you rebuild an index.

**vecbee treats vector migrations like the versioned, reviewable, reversible operations
they should be.** It re-embeds and re-chunks into a *shadow* collection, verifies it, then
does an **atomic alias swap** — your application never queries a half-built index, and if
something's wrong you roll back with one command.

```bash
pip install vecbee
vecbee init      # scaffolds a runnable project — zero API keys, zero infra
vecbee up        # runs the full pipeline for $0 (mock embeddings + local store)
```

That first `up` runs end-to-end with **no paid API calls and no external services**, so you
can see exactly what vecbee does before pointing it at anything real.

---

## Features

- 🐝 **Blue-green zero-downtime cutovers** — migrations build a shadow collection and flip a
  stable alias atomically. Readers never see a partial index.
- ⏪ **Reversible** — `vecbee down` reverts the most recently applied migration.
- 🔀 **Three migration modes** — `sql_chunk_embed` (chunk + embed rows from any SQL source),
  `re_embed` (re-embed existing content with a new model), and `pure_copy` (move vectors
  verbatim, zero embedding calls).
- 📈 **Incremental sync** — an `updated_at` watermark means re-runs only process changed rows,
  not the whole table.
- 🔎 **Drift detection** — `vecbee status` reconciles applied migration history against your
  local migration files and reports drift before you touch production.
- 🧾 **Dead-letter queue** — every failed record is written to `dlq/` as self-describing
  JSONL. Nothing is ever silently dropped.
- 🔁 **Resumable & rate-limit aware** — upsert-by-source-id makes runs idempotent; exponential
  backoff handles `429`s from embedding APIs.
- 🧩 **Pluggable** — sources, targets, and embedding providers are swappable drivers;
  third-party plugins register via a standard entry-point group.
- 💸 **Zero-cost default path** — `mock` provider + `local` SQLite store means CI and first-run
  cost nothing and need no network.
- 🎛️ **Declarative** — one `vector.config.yaml` is the whole contract. No imperative glue code.

---

## Quickstart

### 1. Install

```bash
pip install vecbee
# optional extras for real targets/providers:
pip install "vecbee[qdrant]"      # Qdrant vector target
pip install "vecbee[postgres]"    # Postgres/asyncpg source
```

### 2. Scaffold a project

```bash
vecbee init
```

This writes a runnable `vector.config.yaml` (wired to the `mock` provider + `local` store),
a sample SQL source fixture, and a `migrations/` directory.

### 3. Run the pipeline — for $0

```bash
vecbee up
```

`up` chunks and embeds the sample data into a shadow collection, then atomically swaps the
`docs` alias to it. No API keys, no Docker, no cost.

### 4. Point it at your own data

Edit the `source` block in `vector.config.yaml` to your real database, swap `provider.driver`
to a real embedding API, and re-run `vecbee up`.

---

## How it works

```
                    ┌─────────────────────────────────────────────┐
   vector.config    │  vecbee up                                   │
   .yaml  ────────► │                                              │
                    │  source ──► chunk ──► embed ──► shadow write  │
                    │   (SQL)     (recursive)  (provider)   │       │
                    │                                       ▼       │
   migrations/  ──► │                              ┌── reconcile ──┐│
   0001_*.yaml      │                              │  verify count ││
                    │                              └──────┬────────┘│
                    │        readers query  ◄── alias ────┘         │
                    │        "docs"          atomic swap (cutover)  │
                    └─────────────────────────────────────────────┘
                    failed records ──► dlq/*.jsonl   (never dropped)
```

The **alias** (`collection:` in your config) is the stable name your application queries.
A migration never mutates the live collection in place — it builds a new shadow collection
and, only once written and reconciled, repoints the alias. That repoint is the cutover.

---

## Commands

| Command | What it does |
| --- | --- |
| `vecbee init` | Scaffold a runnable project (`vector.config.yaml` + sample fixtures + `migrations/`). |
| `vecbee status` | Reconcile applied migration history vs. local migration files; report drift. |
| `vecbee up` | Blue-green migration into a shadow collection + atomic cutover. |
| `vecbee down` | Revert the most recently applied migration. |
| `vecbee run-ext <name>` | Load and run one extension standalone (no args → list discoverable extensions). |
| `vecbee --version` | Print the installed version. |

---

## Configuration

`vecbee init` generates this annotated starter. Every block below already works end-to-end
against the zero-cost defaults; edit `source` to migrate your own data.

```yaml
source:
  driver: sql                       # read rows from any SQLAlchemy-supported database
  dsn: "sqlite+aiosqlite:///./.vecbee/sample_data/sample_sql_source.db"
  table: documents
  updated_at_column: updated_at     # enables incremental sync

target:
  driver: local                     # zero-infra SQLite store with real atomic alias-swap
  collection: docs                  # the stable alias your application queries
  path: "./.vecbee/local_target.db"

provider:
  driver: mock                      # deterministic, zero-cost, offline embeddings
  dimensions: 8

chunker:
  driver: recursive                 # langchain-text-splitters RecursiveCharacterTextSplitter
  chunk_size: 1000
  chunk_overlap: 200

state_store:
  dsn: "sqlite+aiosqlite:///./.vecbee/state.db"   # tracks migration history

migrations:
  dir: "./migrations"               # your NNNN_<slug>.yaml migration files

dlq:
  path: "./dlq"                     # failed records land here as JSONL

concurrency: 8                      # in-flight records
batch_size: 64                      # records per batch
```

Values support `${ENV_VAR}` interpolation for secrets (see [`.env.example`](.env.example)).

### Drivers & providers

| Kind | Driver | Notes |
| --- | --- | --- |
| **Source** | `sql` | Any SQLAlchemy dialect (SQLite, Postgres via `[postgres]`, …). |
| **Source** | `vector_local` | Read vectors back out of a local store (e.g. for `pure_copy`). |
| **Target** | `local` | Zero-infra SQLite store with WAL-backed atomic alias swap. |
| **Target** | `qdrant` | Real [Qdrant](https://qdrant.tech) target (`pip install "vecbee[qdrant]"`). |
| **Provider** | `mock` | Deterministic offline embeddings — $0, CI-safe. |
| **Provider** | `openai_compat` | Any OpenAI-compatible embeddings endpoint (key-gated). |
| **Chunker** | `recursive` | `RecursiveCharacterTextSplitter` with configurable size/overlap. |

---

## Architecture

vecbee enforces one hard rule: **`engine/` is pure domain logic and must never import
`display/` or Rich.** The engine emits plain dataclasses; `display/` is the only place Rich
is used to render them. This boundary is enforced in CI by
[import-linter](https://github.com/seddonym/import-linter), so the core stays testable,
scriptable, and free of presentation concerns.

```
src/vecbee/
├── abc/         # provider / source / target / hooks interfaces (the contracts)
├── engine/      # pure orchestration: pipeline, reconciler, drift, dlq, backoff, modes
├── drivers/     # concrete sources, targets, providers, chunker
├── config/      # vector.config.yaml loading, validation, secrets
├── cli/         # Typer commands (init / status / up / down / run-ext)
├── display/     # Rich rendering — the ONLY place Rich is imported
├── plugins/     # entry-point discovery for third-party drivers
└── state/       # migration-history persistence
```

---

## Extending vecbee

Third-party drivers register under the `vecbee.plugins` entry-point group. In your plugin
package's `pyproject.toml`:

```toml
[project.entry-points."vecbee.plugins"]
my_target = "my_pkg.my_module:MyTargetDriver"
```

Then `vecbee run-ext` will discover it, and it becomes usable as a `driver:` in config.

---

## Development

```bash
git clone https://github.com/Caoquyen1913/vecbee.git
cd vecbee
pip install -e ".[dev,qdrant]"

ruff check .        # lint
mypy                # strict type-check on engine/abc/config
lint-imports        # enforce the engine ↛ display boundary
pytest              # full test suite
```

The optional Qdrant integration check runs against a real dockerized Qdrant:

```bash
docker compose up -d qdrant
QDRANT_URL=http://localhost:6333 python scripts/qdrant_integration_check.py
docker compose down
```

---

## Project status

**Alpha (`0.1.0`).** The interface is exercised by a large passing test suite and every
acceptance criterion, but has no real-world usage history yet. Expect the API to stabilize
toward `0.2.0`. Feedback and issues are very welcome.

## Contributing

Contributions are welcome! Please open an issue to discuss substantial changes first. Before
submitting a PR, make sure `ruff`, `mypy`, `lint-imports`, and `pytest` all pass locally
(they're gated in CI). See [PUBLISHING.md](PUBLISHING.md) for the release process.

## License

[Apache-2.0](LICENSE) © Kawin Tran ([@Caoquyen1913](https://github.com/Caoquyen1913)) and
[AI Outsourcing Studio](https://github.com/Caoquyen1913/ai-outsourcing-studio).
