Metadata-Version: 2.4
Name: milvus-lite
Version: 3.1.1
Summary: Lightweight version of Milvus for local development and testing
Project-URL: Homepage, https://github.com/milvus-io/milvus-lite
Project-URL: Repository, https://github.com/milvus-io/milvus-lite
Author: Milvus Contributors
License: Apache-2.0
License-File: LICENSE
Keywords: bm25,database,embedding,milvus,search,vector
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: faiss-cpu>=1.7.4
Requires-Dist: grpcio>=1.50
Requires-Dist: numpy>=1.24
Requires-Dist: pyarrow>=15.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: chinese
Requires-Dist: jieba>=0.42; extra == 'chinese'
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

<div align="center">
    <img src="https://raw.githubusercontent.com/milvus-io/milvus-lite/refs/heads/main/milvus_lite_logo.png" width="60%"/>
</div>

<h3 align="center">
    Milvus Lite - a lightweight, local Milvus for development and testing
</h3>

<p align="center">
    <a href="https://github.com/milvus-io/milvus-lite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue" alt="License"></a>
    <a href="https://pypi.org/project/milvus-lite/"><img src="https://img.shields.io/pypi/v/milvus-lite?label=PyPI" alt="PyPI"></a>
    <img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python">
</p>

# Milvus Lite

Milvus Lite is a lightweight version of [Milvus](https://github.com/milvus-io/milvus) for local development, notebooks, tests, and small-scale AI applications. It exposes the familiar `pymilvus` API through a local `.db` file or an embedded gRPC server, so code written against Milvus Lite can move to Milvus Standalone, Milvus Distributed, or Zilliz Cloud with minimal changes.

This generation of Milvus Lite is rebuilt from scratch in pure Python. Instead of wrapping the full C++ Milvus core, it uses a local LSM-style storage engine with WAL, in-memory memtables, immutable Parquet segments, segment-level FAISS indexes, scalar filtering, BM25 full text search, and a Milvus-compatible gRPC adapter.

<div align="center">
    <img src="https://milvus.io/docs/v2.5.x/assets/select-deployment-option.png" width="80%"/>
</div>

Milvus Lite is intended for prototyping and local workloads. For large-scale production deployments, use [Milvus Standalone](https://milvus.io/docs/install-overview.md#Milvus-Standalone), [Milvus Distributed](https://milvus.io/docs/install-overview.md#Milvus-Distributed), or [Zilliz Cloud](https://zilliz.com/cloud).

## Highlights

- Drop-in local usage with `MilvusClient("./demo.db")`.
- Pure Python implementation with inspectable code and Python stack traces.
- Dense vector search, sparse BM25 search, and hybrid search.
- FAISS-backed vector indexes and `INVERTED` scalar indexes.
- Milvus-style scalar and geometry filters, dynamic fields, JSON fields, array fields, partitions, aliases, iterators, and group-by search.
- Named databases and embedded-engine collection snapshots.
- Optional schema functions for BM25 and text embedding.
- Standalone local gRPC server for multi-client development.

## Requirements

- Python 3.10 or newer
- macOS, Linux, or Windows where the Python dependencies are available

Continuous integration tests cover Linux and macOS on Python 3.10–3.13 and Windows on Python 3.10. Installation on other Windows/Python combinations depends on compatible wheels being available for dependencies such as `faiss-cpu` and `pyarrow`.

Core dependencies are installed by default: `pyarrow`, `numpy`, `faiss-cpu`, and `grpcio`. `pymilvus` is intentionally not a dependency of `milvus-lite`; it installs Milvus Lite as its local backend.

## Installation

Install Milvus Lite directly:

```bash
pip install -U milvus-lite
```

Most users install Milvus Lite through the `pymilvus` extra:

```bash
pip install -U "pymilvus[milvus-lite]"
```

If you already have the original C++/CGo-based `milvus-lite` package installed, upgrade in a clean environment or reinstall it explicitly:

```bash
pip uninstall milvus-lite -y
pip install -U milvus-lite
```

`.db` files created by the original milvus-lite storage format are not compatible with this pure-Python engine. Re-import data into a new database; automatic migration is not available.

## Quick Start

### Local `.db` File

This example requires `pymilvus`; install it with `pip install -U "pymilvus[milvus-lite]"`.

Passing a local `.db` path to `MilvusClient` starts Milvus Lite automatically.

```python
from pymilvus import MilvusClient

client = MilvusClient("./demo.db")

client.create_collection(
    collection_name="docs",
    dimension=4,
)

data = [
    {"id": 1, "vector": [0.1, 0.2, 0.3, 0.4], "text": "python tutorial", "category": "tech"},
    {"id": 2, "vector": [0.2, 0.1, 0.4, 0.3], "text": "machine learning", "category": "ai"},
    {"id": 3, "vector": [0.9, 0.1, 0.1, 0.1], "text": "travel notes", "category": "life"},
]
client.insert("docs", data)

results = client.search(
    collection_name="docs",
    data=[[0.1, 0.2, 0.3, 0.4]],
    limit=2,
    filter="category in ['tech', 'ai']",
    output_fields=["text", "category"],
)
print(results)

rows = client.query(
    collection_name="docs",
    filter="category == 'tech'",
    output_fields=["text"],
)
print(rows)

client.delete("docs", ids=[3])
client.drop_collection("docs")
```

### Standalone gRPC Server

Use server mode when you want a long-running local process or more than one client.

```bash
milvus-lite server --data-dir ./data --port 19530
```

```python
from pymilvus import MilvusClient

client = MilvusClient(uri="http://127.0.0.1:19530")
```

### Embedded Engine API

Use the internal Python engine directly when you do not need the Milvus protocol adapter.

```python
from milvus_lite import CollectionSchema, DataType, FieldSchema, MilvusLite

schema = CollectionSchema(fields=[
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="vec", dtype=DataType.FLOAT_VECTOR, dim=4),
    FieldSchema(name="category", dtype=DataType.VARCHAR, max_length=64),
])

with MilvusLite("./data") as db:
    col = db.create_collection("docs", schema)
    col.insert([
        {"id": 1, "vec": [0.1, 0.2, 0.3, 0.4], "category": "tech"},
        {"id": 2, "vec": [0.2, 0.1, 0.4, 0.3], "category": "ai"},
    ])
    col.create_index("vec", {
        "index_type": "HNSW",
        "metric_type": "COSINE",
        "params": {"M": 16, "efConstruction": 200},
    })
    col.load()

    results = col.search(
        [[0.1, 0.2, 0.3, 0.4]],
        top_k=2,
        metric_type="COSINE",
        expr="category == 'tech'",
        output_fields=["category"],
    )
    print(results)
```

## Full Text and Hybrid Search

Milvus Lite supports BM25 through Milvus schema functions. Text fields are analyzed on insert and written to a sparse vector field.

```python
from pymilvus import DataType, Function, FunctionType, MilvusClient

client = MilvusClient("./fts.db")

schema = MilvusClient.create_schema(auto_id=True)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field(
    "text",
    DataType.VARCHAR,
    max_length=65535,
    enable_analyzer=True,
    enable_match=True,
)
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
schema.add_function(Function(
    name="bm25",
    function_type=FunctionType.BM25,
    input_field_names=["text"],
    output_field_names=["sparse"],
))

client.create_collection("articles", schema=schema)
client.insert("articles", [
    {"text": "machine learning algorithms"},
    {"text": "deep learning neural networks"},
    {"text": "database systems and search"},
])

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="sparse",
    index_type="SPARSE_INVERTED_INDEX",
    metric_type="BM25",
    params={},
)
client.create_index("articles", index_params)
client.load_collection("articles")

results = client.search(
    "articles",
    data=["machine learning"],
    anns_field="sparse",
    limit=3,
    output_fields=["text"],
)
print(results)
```

Hybrid search combines multiple ANN routes, such as dense vector search plus BM25, with `WeightedRanker` or `RRFRanker`. Request-level `FunctionType.RERANK` is also supported for model rerank and numeric decay rerank.

## Filtering

Milvus Lite implements Milvus-style scalar expressions for search, query, get, and delete paths.

```python
client.query("docs", filter="age > 25 and status == 'active'")
client.query("docs", filter="category in ['tech', 'science']")
client.query("docs", filter="name like 'John%'")
client.query("docs", filter='array_contains(tags, "python")')
client.query("docs", filter="array_length(scores) >= 3")
client.query("docs", filter='$meta["color"] == "red"')
client.query("docs", filter='exists metadata["source"]')
client.query("docs", filter='$meta["priority"] is not null')
client.query("docs", filter="text_match(text, 'machine learning')")
client.query("places", filter="ST_WITHIN(location, 'POLYGON((0 0, 10 0, 10 10, 0 10, 0 0))')")
```

JSON path and dynamic-field expressions support `exists`, `is null`, and `is not null`. A missing JSON key and an explicit JSON `null` are both treated as null. `GEOMETRY` fields use WKT values and support `ST_CONTAINS`, `ST_WITHIN`, `ST_INTERSECTS`, `ST_ISVALID`, and `ST_DWITHIN`; spatial predicates currently use scan-based filtering.

## Dump Data

Milvus Lite includes a compatibility dump command that exports a collection to local Milvus BulkWriter JSON files.

Dump a local `.db` database:

```bash
milvus-lite dump -d ./demo.db -c docs -p ./dump
```

Or dump from a running Milvus-compatible endpoint:

```bash
milvus-lite dump --uri http://127.0.0.1:19530 -c docs -p ./dump
```

The command reads rows through `pymilvus` query iterators and writes BulkWriter JSON files that can be used with Milvus Bulk Insert or Zilliz Cloud import workflows. Its `pymilvus` and BulkWriter dependencies are expected to come from the `pymilvus` installation that installed Milvus Lite. This is an export tool, not an automatic storage-format migration: original milvus-lite v1 `.db` files still need to be dumped from an environment that can open the v1 database.

## Supported Features

| Area | Support |
|---|---|
| Vector types | `FLOAT_VECTOR`, `SPARSE_FLOAT_VECTOR` |
| Scalar types | `BOOL`, integer types, `FLOAT`, `DOUBLE`, `VARCHAR`, `JSON`, `ARRAY`, `GEOMETRY`, `TIMESTAMPTZ` |
| Metrics | `COSINE`, `L2`, `IP`, `BM25` |
| Indexes | `HNSW`, `HNSW_SQ`, `IVF_FLAT`, `IVF_SQ8`, `FLAT`, `BRUTE_FORCE`, `AUTOINDEX`, sparse `SPARSE_INVERTED_INDEX`, scalar `INVERTED` |
| CRUD | insert, upsert, partial update, delete by ID/filter, get, query, search, truncate |
| Collection management | create, drop, rename, describe, statistics, aliases |
| Databases | create, drop, list, switch, database properties |
| Snapshots | create, list, drop, and restore to a new collection through the embedded engine API |
| Partitions | create, drop, list, partition-specific insert/search, partition key routing |
| Search features | dense and sparse search, multiple dense-vector fields selected with `anns_field`, hybrid search, request-level boost/model/decay reranking, range search, group-by, iterators, offset, `round_decimal` |
| Schema features | auto ID, nullable fields, default values, dynamic fields, BM25 functions, OpenAI text embedding functions |
| Filtering | scalar, JSON path, dynamic field, array, text-match, and scan-based WKT geometry predicates |
| Text search | standard analyzer, optional Jieba analyzer, `text_match`, BM25 sparse inverted index |
| Adapter | gRPC server implementing the supported local `pymilvus` workflow subset |

## Known Limitations

- Single process per `data_dir`; Milvus Lite uses a file lock to protect local storage.
- The gRPC server supports concurrent reads, but writes to the same collection must be serialized; concurrent writers are not supported safely.
- The gRPC adapter implements the supported local `pymilvus` workflow subset. Unsupported Milvus RPCs return `UNIMPLEMENTED`; schema alteration and partition-level load/release are not available.
- No authentication, users, roles, RBAC, or TLS; do not expose the local gRPC server on an untrusted network.
- No binary, float16, bfloat16, or int8 vector fields.
- No Product Quantization indexes.
- `TEXT_EMBEDDING` currently supports OpenAI only, and semantic model reranking currently supports Cohere only; both require external credentials and network access.
- Snapshot operations are currently available through the embedded engine API rather than `pymilvus` Snapshot RPCs.
- Geometry predicates use scan-based filtering; spatial indexes are not available.
- BM25 IDF statistics are segment-local rather than global.
- The engine is designed for local development and small-scale workloads, not distributed production serving.

## Architecture

```text
pymilvus client
      |
      v
adapter/grpc
      |
      v
engine/Collection
      |
      +-- storage/   WAL, MemTable, Parquet segments, deltas, manifest, snapshots
      +-- search/    bitmap pipeline, scalar/geometry filters, dense/sparse executors
      +-- index/     FAISS vector indexes, scalar/sparse inverted indexes
      +-- analyzer/  standard and Jieba analyzers, BM25 term hashing
      +-- function/  BM25, text embedding, hybrid merge, rerank chains
      +-- schema/    data types, validation, Arrow schema builders
```

The storage path is LSM-tree style: writes go to the WAL and MemTable, flush creates immutable Parquet files, and compaction merges segments and garbage-collects tombstones. Vector and scalar indexes are bound to immutable segments and persisted as `.idx` files. Collection snapshots pin immutable data, delta, and index files and preserve copies of the schema and manifest so they can be restored into a new collection. The manifest is the source of truth and is updated atomically.

## Development

```bash
git clone https://github.com/milvus-io/milvus-lite.git
cd milvus-lite
make dev
source .venv/bin/activate
```

Common commands:

| Command | Description |
|---|---|
| `make test` | Run the regular functional suite |
| `make test-fast` | Run the fast local suite |
| `make test-core` | Run schema, storage, search, index, analyzer, engine, function, rerank, embedding tests |
| `make test-compat` | Run gRPC adapter and pymilvus compatibility tests |
| `make test-stability` | Run slow crash and long-run stability tests |
| `make test-soak` | Run explicit large-scale soak tests |
| `make benchmark` | Run performance benchmarks |
| `make coverage` | Run tests with coverage |
| `make build` | Build wheel and source distribution under `dist/` |
| `make serve` | Start a local gRPC server via `milvus-lite server` on port 19530 |
| `make clean` | Remove virtualenv, caches, coverage files, and build artifacts |

## Documentation

Design notes are in [`docs/`](docs/). Start with [`docs/modules.md`](docs/modules.md) for the file-by-file architecture, then read the focused design documents for WAL, filters, indexes, gRPC compatibility, full text search, and search iterators.

## Contributing

Issues and pull requests are welcome. Please read [`CONTRIBUTING.md`](CONTRIBUTING.md) before sending larger changes.

## License

Milvus Lite is licensed under Apache License 2.0. See [`LICENSE`](LICENSE).
