Metadata-Version: 2.4
Name: vecbee
Version: 0.1.1
Summary: A lightweight, open-source vector-database migration & infrastructure CLI (Alembic for vector databases).
Project-URL: Homepage, https://github.com/Caoquyen1913/vecbee
Project-URL: Repository, https://github.com/Caoquyen1913/vecbee
Project-URL: Documentation, https://github.com/Caoquyen1913/vecbee#readme
Project-URL: Issues, https://github.com/Caoquyen1913/vecbee/issues
Project-URL: Changelog, https://github.com/Caoquyen1913/vecbee/blob/main/CHANGELOG.md
Author: AI Outsourcing Studio
Author-email: Kawin Tran <caoquyen1913@gmail.com>
Maintainer-email: Kawin Tran <caoquyen1913@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai,alembic,blue-green,blue-green-deployment,cli,data-pipeline,database-migration,embedding-migration,embeddings,llm,migration,pinecone,qdrant,rag,re-embedding,retrieval-augmented-generation,schema-migration,semantic-search,sqlalchemy,vector-database,vector-db,vector-store,zero-downtime
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: aiosqlite>=0.20
Requires-Dist: langchain-text-splitters>=0.2
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: import-linter>=2.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == 'postgres'
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.9; extra == 'qdrant'
Description-Content-Type: text/markdown

<!-- Logo & assets live in ./assets and are referenced by absolute raw URL so they
     render identically on GitHub and on PyPI (PyPI cannot resolve relative paths). -->
<p align="center">
  <img src="https://raw.githubusercontent.com/Caoquyen1913/vecbee/main/assets/vecbee-logo.svg" alt="vecbee" width="440">
</p>

<h3 align="center">Alembic for vector databases.</h3>

<p align="center">
  Migrate, sync, and re-embed vector data across stores with blue-green
  zero-downtime cutovers — all driven by one declarative <code>vector.config.yaml</code>.
</p>

<p align="center">
  <a href="https://pypi.org/project/vecbee/"><img src="https://img.shields.io/pypi/v/vecbee?color=F59003&label=pypi" alt="PyPI version"></a>
  <a href="https://pypi.org/project/vecbee/"><img src="https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-3B82F6" alt="Python 3.10 | 3.11 | 3.12"></a>
  <a href="https://github.com/Caoquyen1913/vecbee/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-10B981" alt="License: Apache-2.0"></a>
  <a href="https://github.com/Caoquyen1913/vecbee/actions/workflows/ci.yml"><img src="https://github.com/Caoquyen1913/vecbee/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
  <img src="https://img.shields.io/badge/mypy-strict-2A6DB0" alt="mypy strict">
</p>

---

> **vecbee** is an open-source command-line tool for **vector-database migrations** — it
> re-embeds, re-chunks, and moves embedding data between vector stores (Qdrant, a local
> SQLite store, and more) with **blue-green, zero-downtime cutovers**, driven by a single
> declarative `vector.config.yaml`. Think **Alembic or Flyway, but for vector databases and
> RAG pipelines**.

## Why vecbee?

Relational databases have had migration tooling for decades — Alembic, Flyway, Rails
migrations. Vector databases have had almost nothing. When your embedding model changes,
your chunking strategy improves, or you outgrow your vector store, you're left writing
one-off scripts, praying nothing breaks, and taking downtime while you rebuild an index.

**vecbee treats vector migrations like the versioned, reviewable, reversible operations
they should be.** It re-embeds and re-chunks into a *shadow* collection, verifies it, then
does an **atomic alias swap** — your application never queries a half-built index, and if
something's wrong you roll back with one command.

```bash
pip install vecbee
vecbee init      # scaffolds a runnable project — zero API keys, zero infra
vecbee up        # runs the full pipeline for $0 (mock embeddings + local store)
```

That first `up` runs end-to-end with **no paid API calls and no external services**, so you
can see exactly what vecbee does before pointing it at anything real.

---

## Features

- 🐝 **Blue-green zero-downtime cutovers** — migrations build a shadow collection and flip a
  stable alias atomically. Readers never see a partial index.
- ⏪ **Reversible** — `vecbee down` reverts the most recently applied migration.
- 🔀 **Three migration modes** — `sql_chunk_embed` (chunk + embed rows from any SQL source),
  `re_embed` (re-embed existing content with a new model), and `pure_copy` (move vectors
  verbatim, zero embedding calls).
- 📈 **Incremental sync** — an `updated_at` watermark means re-runs only process changed rows,
  not the whole table.
- 🔎 **Drift detection** — `vecbee status` reconciles applied migration history against your
  local migration files and reports drift before you touch production.
- 🧾 **Dead-letter queue** — every failed record is written to `dlq/` as self-describing
  JSONL. Nothing is ever silently dropped.
- 🔁 **Resumable & rate-limit aware** — upsert-by-source-id makes runs idempotent; exponential
  backoff handles `429`s from embedding APIs.
- 🧩 **Pluggable** — sources, targets, and embedding providers are swappable drivers;
  third-party plugins register via a standard entry-point group.
- 💸 **Zero-cost default path** — `mock` provider + `local` SQLite store means CI and first-run
  cost nothing and need no network.
- 🎛️ **Declarative** — one `vector.config.yaml` is the whole contract. No imperative glue code.

---

## Quickstart

### 1. Install

```bash
pip install vecbee
# optional extras for real targets/providers:
pip install "vecbee[qdrant]"      # Qdrant vector target
pip install "vecbee[postgres]"    # Postgres/asyncpg source
```

### 2. Scaffold a project

```bash
vecbee init
```

This writes a runnable `vector.config.yaml` (wired to the `mock` provider + `local` store),
a sample SQL source fixture, and a `migrations/` directory.

### 3. Run the pipeline — for $0

```bash
vecbee up
```

`up` chunks and embeds the sample data into a shadow collection, then atomically swaps the
`docs` alias to it. No API keys, no Docker, no cost.

### 4. Point it at your own data

Edit the `source` block in `vector.config.yaml` to your real database, swap `provider.driver`
to a real embedding API, and re-run `vecbee up`.

---

## How it works

<p align="center">
  <img src="https://raw.githubusercontent.com/Caoquyen1913/vecbee/main/assets/vecbee-flow.png" alt="vecbee pipeline: vector.config.yaml + migrations feed Source (SQL) → Chunk (recursive) → Embed (provider) → Shadow collection → Reconcile (verify count) → Atomic cutover (alias swap to &quot;docs&quot;, zero read-gap) → Readers query &quot;docs&quot;; failed records go to dlq/*.jsonl" width="840">
</p>

The **alias** (`collection:` in your config) is the stable name your application queries.
A migration never mutates the live collection in place — it builds a new shadow collection
and, only once written and reconciled, repoints the alias. That repoint is the cutover.

---

## Commands

| Command | What it does |
| --- | --- |
| `vecbee init` | Scaffold a runnable project (`vector.config.yaml` + sample fixtures + `migrations/`). |
| `vecbee status` | Reconcile applied migration history vs. local migration files; report drift. |
| `vecbee up` | Blue-green migration into a shadow collection + atomic cutover. |
| `vecbee down` | Revert the most recently applied migration. |
| `vecbee run-ext <name>` | Load and run one extension standalone (no args → list discoverable extensions). |
| `vecbee --version` | Print the installed version. |

---

## Configuration

`vecbee init` generates this annotated starter. Every block below already works end-to-end
against the zero-cost defaults; edit `source` to migrate your own data.

```yaml
source:
  driver: sql                       # read rows from any SQLAlchemy-supported database
  dsn: "sqlite+aiosqlite:///./.vecbee/sample_data/sample_sql_source.db"
  table: documents
  updated_at_column: updated_at     # enables incremental sync

target:
  driver: local                     # zero-infra SQLite store with real atomic alias-swap
  collection: docs                  # the stable alias your application queries
  path: "./.vecbee/local_target.db"

provider:
  driver: mock                      # deterministic, zero-cost, offline embeddings
  dimensions: 8

chunker:
  driver: recursive                 # langchain-text-splitters RecursiveCharacterTextSplitter
  chunk_size: 1000
  chunk_overlap: 200

state_store:
  dsn: "sqlite+aiosqlite:///./.vecbee/state.db"   # tracks migration history

migrations:
  dir: "./migrations"               # your NNNN_<slug>.yaml migration files

dlq:
  path: "./dlq"                     # failed records land here as JSONL

concurrency: 8                      # in-flight records
batch_size: 64                      # records per batch
```

Values support `${ENV_VAR}` interpolation for secrets (see [`.env.example`](.env.example)).

### Drivers & providers

| Kind | Driver | Notes |
| --- | --- | --- |
| **Source** | `sql` | Any SQLAlchemy dialect (SQLite, Postgres via `[postgres]`, …). |
| **Source** | `vector_local` | Read vectors back out of a local store (e.g. for `pure_copy`). |
| **Target** | `local` | Zero-infra SQLite store with WAL-backed atomic alias swap. |
| **Target** | `qdrant` | Real [Qdrant](https://qdrant.tech) target (`pip install "vecbee[qdrant]"`). |
| **Provider** | `mock` | Deterministic offline embeddings — $0, CI-safe. |
| **Provider** | `openai_compat` | Any OpenAI-compatible embeddings endpoint (key-gated). |
| **Chunker** | `recursive` | `RecursiveCharacterTextSplitter` with configurable size/overlap. |

---

## Architecture

vecbee enforces one hard rule: **`engine/` is pure domain logic and must never import
`display/` or Rich.** The engine emits plain dataclasses; `display/` is the only place Rich
is used to render them. This boundary is enforced in CI by
[import-linter](https://github.com/seddonym/import-linter), so the core stays testable,
scriptable, and free of presentation concerns.

```
src/vecbee/
├── abc/         # provider / source / target / hooks interfaces (the contracts)
├── engine/      # pure orchestration: pipeline, reconciler, drift, dlq, backoff, modes
├── drivers/     # concrete sources, targets, providers, chunker
├── config/      # vector.config.yaml loading, validation, secrets
├── cli/         # Typer commands (init / status / up / down / run-ext)
├── display/     # Rich rendering — the ONLY place Rich is imported
├── plugins/     # entry-point discovery for third-party drivers
└── state/       # migration-history persistence
```

---

## Extending vecbee

Third-party drivers register under the `vecbee.plugins` entry-point group. In your plugin
package's `pyproject.toml`:

```toml
[project.entry-points."vecbee.plugins"]
my_target = "my_pkg.my_module:MyTargetDriver"
```

Then `vecbee run-ext` will discover it, and it becomes usable as a `driver:` in config.

---

## Development

```bash
git clone https://github.com/Caoquyen1913/vecbee.git
cd vecbee
pip install -e ".[dev,qdrant]"

ruff check .        # lint
mypy                # strict type-check on engine/abc/config
lint-imports        # enforce the engine ↛ display boundary
pytest              # full test suite
```

The optional Qdrant integration check runs against a real dockerized Qdrant:

```bash
docker compose up -d qdrant
QDRANT_URL=http://localhost:6333 python scripts/qdrant_integration_check.py
docker compose down
```

---

## vecbee vs. a hand-rolled migration script

| | Hand-rolled script | **vecbee** |
| --- | --- | --- |
| Zero-downtime cutover | Manual, error-prone | ✅ Atomic alias swap, built in |
| Rollback | Rewrite/rerun by hand | ✅ `vecbee down` |
| Failed records | Often silently lost | ✅ Dead-letter queue (JSONL) |
| Resume after crash | Usually restart from zero | ✅ Idempotent upsert-by-source-id |
| Rate-limit handling | Ad-hoc `sleep()` | ✅ Exponential backoff |
| Drift / history | None | ✅ `vecbee status` reconciles history |
| Config | Scattered in code | ✅ One declarative `vector.config.yaml` |
| Cost to try | — | ✅ `$0` mock + local default path |

## FAQ

**What is vecbee?**
vecbee is an open-source Python CLI for migrating vector-database data — re-embedding,
re-chunking, and moving embeddings between vector stores with blue-green, zero-downtime
cutovers. It's often described as "Alembic for vector databases."

**Is vecbee like Alembic or Flyway, but for vector databases?**
Yes. Alembic and Flyway version and migrate relational schemas; vecbee versions and migrates
vector data — the embeddings, chunks, and collections that power RAG and semantic search.

**Which vector databases does vecbee support?**
Today: [Qdrant](https://qdrant.tech) and a zero-infrastructure local SQLite-backed vector
store (with real atomic alias-swap semantics). More targets are on the roadmap, and the
driver/plugin system lets you add your own (Pinecone, Weaviate, pgvector, …).

**How does vecbee achieve zero-downtime migrations?**
It uses a blue-green cutover: the migration writes into a new *shadow* collection, verifies
it by reconciling record counts, then atomically repoints a stable alias. Readers always
query the alias, so they never observe a partial or half-rebuilt index.

**When should I use vecbee instead of a custom re-embedding script?**
When you care about not taking downtime, being able to roll back, not losing failed records,
resuming after a crash, and having a reviewable migration history — vecbee gives you all of
that out of the box instead of reimplementing it per script.

**Does vecbee need an API key or a paid service to try?**
No. The default path uses a `mock` embedding provider and a `local` SQLite store, so
`vecbee init && vecbee up` runs end-to-end with zero API keys, zero network, and zero cost.

**What embedding providers does vecbee work with?**
A deterministic `mock` provider (for CI and first runs) and any OpenAI-compatible embeddings
endpoint via the `openai_compat` driver. More providers can be added as plugins.

**Is vecbee free and open source?**
Yes — Apache-2.0 licensed, installable from PyPI with `pip install vecbee`.

## Project status

**Alpha (`0.1.x`).** The interface is exercised by a large passing test suite and every
acceptance criterion, but has no real-world usage history yet. Expect the API to stabilize
toward `0.2.0`. Feedback and issues are very welcome.

## Contributing

Contributions are welcome! Please open an issue to discuss substantial changes first. Before
submitting a PR, make sure `ruff`, `mypy`, `lint-imports`, and `pytest` all pass locally
(they're gated in CI). See [PUBLISHING.md](PUBLISHING.md) for the release process.

## License

[Apache-2.0](LICENSE) © Kawin Tran ([@Caoquyen1913](https://github.com/Caoquyen1913)) and
[AI Outsourcing Studio](https://github.com/Caoquyen1913/ai-outsourcing-studio).
