Metadata-Version: 2.4
Name: endee
Version: 0.1.39b1
Summary: Endee is the Next-Generation Vector Database for Scalable, High-Performance AI
Home-page: https://endee.io
Author: Endee Labs
Author-email: dev@endee.io
Project-URL: Documentation, https://docs.endee.io
Keywords: vector database,embeddings,machine learning,AI,similarity search,HNSW,nearest neighbors
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: httpx[http2]>=0.28.1
Requires-Dist: numpy>=2.2.4
Requires-Dist: msgpack>=1.1.0
Requires-Dist: orjson>=3.11.5
Requires-Dist: pydantic<3,>=1.9.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Endee - Python Client

Endee is a high-performance vector database with three retrieval modes you can mix
**per object and per query**:

- **Dense vector search** - semantic ANN over embeddings (HNSW).
- **Keyword search** - sparse embeddings (BM25-style inverted index).
- **Multi-vector search** - many vectors per object, pooled for retrieval and reranked.

A single object can hold a dense field, a sparse field, and a multi-vector field at
once, and a single query can hit any subset - returned per-field or fused with RRF.

## Where it fits

- **Edge / on-device** - `int16` / `int8e` precisions keep memory and footprint small and still provide 97% recall and 1500 QPS.
- **E-commerce** - semantic + keyword hybrid with metadata filters (price, category, …).
- **High-performance agentic AI / RAG** - fast multi-field retrieval and multi-vector reranking for low-latency tool/context lookup.

---

## Concepts

```
Database                      ← top level; one database = one token, fully isolated
└── Collection                ← a named set of typed fields
    └── Object                ← one record
        ├── id                ← unique string key
        ├── meta              ← arbitrary JSON, returned with results
        ├── filter            ← tags used to filter searches
        └── fields            ← the vectors, by field name:
              embedding   (dense)         [0.1, 0.2, ...]
              keywords    (sparse)        {indices, values}
              multivec     (multi_vector)  [[...], [...]]
```

- **Database** - top-level namespace; created by the **root token**, addressed by a **db token**. Databases are isolated from each other.
- **Collection** - declares named, typed fields (`vector` / `sparse` / `multi_vector`) and holds objects.
- **Object** - one item in a collection:
  - **`id`** - unique string identifier (insert-or-replace key).
  - **`meta`** - arbitrary JSON payload, echoed back in search results.
  - **`filter`** - key/value tags queried with filter operators (`$eq`, `$in`, `$range`, …).
  - **`fields`** - the actual vectors, one entry per field you populate.

---

## Install

```bash
pip install endee
```

Python ≥ 3.9. Depends on `numpy`, `msgpack`, `orjson`, `requests`, `httpx`.

---

## Quick start

Authentication is **always on** - every request needs a token.

```python
from endee import Endee

# 1) Admin: create a database with the ROOT token.
#    If the server was started WITHOUT NDD_ROOT_TOKEN, the root token is the
#    insecure dev default "unsafe_root_token".
admin = Endee(token="unsafe_root_token")
admin.set_base_url("http://localhost:8080/api/v2")
db_token = admin.create_database("alice")        # returns the db token, e.g. "alice:9f3..."

# 2) App: everything else uses the DB token.
client = Endee(token=db_token)
client.set_base_url("http://localhost:8080/api/v2")

# 3) Create a collection with a dense field.
client.create_collection(
    name="products",
    fields=[
        {"name": "embedding", "type": "vector",
         "params": {"dimension": 8, "space_type": "cosine", "precision": "int8"}},
    ],
)
collection = client.get_collection("products")

# 4) Insert objects (dense vector per object).
collection.upsert([
    {"id": "p1", "meta": {"name": "Wireless Headphones"}, "filter": {"category": "electronics"},
     "fields": {"embedding": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]}},
    {"id": "p2", "meta": {"name": "Running Shoes"}, "filter": {"category": "footwear"},
     "fields": {"embedding": [0.5, 0.4, 0.3, 0.2, 0.1, 0.0, 0.2, 0.3]}},
])

# 5) Search.
hits = collection.search(fields={"embedding": [0.2, 0.2, 0.3, 0.3, 0.4, 0.5, 0.6, 0.7]}, limit=3)
for h in hits["results"]:
    print(h["id"], round(h["similarity"], 4), h["meta"])
```

> **Tokens:** the **root token** (server's `NDD_ROOT_TOKEN`, or `unsafe_root_token`
> in dev) administers databases; a **db token** (`db_name:secret`, returned by
> `create_database`) does all data work. A missing/invalid token → 401; a read-only
> token on a write → 403.

---

## Field types

Add fields at collection-creation time. Each is a plain dict:

| Type | Keys | Value on upsert | Query shape |
|------|------|-----------------|-------------|
| `vector` | `params`: `dimension`, `space_type`, `precision` | `[0.1, 0.2, ...]` | `[0.2, ...]` |
| `sparse` | `sparse_model` (top-level) | `{"indices":[...], "values":[...]}` | `{"indices":[...], "values":[...]}` |
| `multi_vector` | `params`: vector params **+ `pooling`** (`"mean"`/`"max"`) | `[[...], [...]]` | `[[...], [...]]` |

`space_type`: `cosine` (default, normalized client-side), `l2`, `ip`.
`precision`: `float32`, `float16`, `int16`, `int8`, `int8e`, `binary`.

### Adding a sparse (keyword) field

```python
client.create_collection("docs", fields=[
    {"name": "embedding", "type": "vector",
     "params": {"dimension": 8, "space_type": "cosine", "precision": "int8"}},
    {"name": "keywords", "type": "sparse", "sparse_model": "default"},
])

collection.upsert([{
    "id": "d1",
    "fields": {
        "embedding": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
        "keywords":  {"indices": [3, 17, 42], "values": [0.9, 0.5, 0.2]},   # term ids + weights
    },
}])
```

### Adding a multi-vector field

A multi-vector field stores several vectors per object, pooled into one vector for
retrieval. Two **pooling strategies** are supported:

- **`mean`** (default) - elementwise average of the members.
- **`max`** - elementwise maximum of the members.

```python
client.create_collection("passages", fields=[
    {"name": "embedding", "type": "vector",
     "params": {"dimension": 8, "space_type": "cosine", "precision": "int8"}},
    {"name": "multivec", "type": "multi_vector",
     "params": {"dimension": 8, "space_type": "cosine", "precision": "int8", "pooling": "mean"}},
])

collection.upsert([{
    "id": "x1",
    "fields": {
        "embedding": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
        "multivec":   [[0.1]*8, [0.2]*8, [0.3]*8],   # one vector per token/passage
    },
}])
```

You can set any subset of fields on a given object; `upsert` is insert-or-replace by `id`.

---

## Search

A hit is `{"id", "similarity", "meta", "filter"}`. Results carry `meta`/`filter`
but not the stored vectors - use [`get_objects`](#object-operations) to fetch those.

### Dense only

```python
res = collection.search(fields={"embedding": [0.2]*8}, limit=5)
res["results"]          # → [ {id, similarity, meta, filter}, ... ]
```

### Dense + sparse - without a reranker (per-field results)

Query multiple fields and **omit `reranker`** → you get **each field's own ranked
list**, keyed by field name (scores are per-field and not comparable across fields):

```python
res = collection.search(
    fields={
        "embedding": {"query": [0.2]*8, "limit": 10},
        "keywords":  {"query": {"indices": [3, 17], "values": [0.8, 0.4]}, "limit": 10},
    },
    limit=5,
)
res["results"]["embedding"]   # → [ {id, similarity, ...}, ... ]
res["results"]["keywords"]    # → [ {id, similarity, ...}, ... ]
```

### Dense + sparse - with RRF (one fused list)

Pass `reranker="rrf"` to fuse the per-field lists into a single ranked list via
Reciprocal Rank Fusion:

```python
res = collection.search(
    fields={
        "embedding": {"query": [0.2]*8, "limit": 50},
        "keywords":  {"query": {"indices": [3, 17], "values": [0.8, 0.4]}, "limit": 50},
    },
    limit=10,
    reranker="rrf",
    field_weights={"embedding": 0.6, "keywords": 0.4},   # optional, must sum to 1.0
)
res["results"]          # → [ {id, similarity (fused score), meta, filter}, ... ]
```

### Multi-vector

```python
collection.search(fields={"multivec": [[0.1]*8, [0.2]*8]}, limit=5)         # single field → list
# multi-vector also participates in multi-field + RRF exactly like above.
```

### Filters

`filter` works in any mode - an array of conditions (AND-combined):

```python
collection.search(
    fields={"embedding": [0.2]*8}, limit=5,
    filter=[{"category": {"$eq": "electronics"}}, {"price": {"$range": [10, 100]}}],
)
```

Operators: `$eq`, `$in`, `$range`, `$gt`, `$gte`, `$lt`, `$lte`.

**Return-shape summary**

| Query | `reranker` | `results` |
|-------|-----------|-----------|
| 1 field | (any) | `[hit, ...]` |
| N fields | `None` | `{field: [hit, ...], ...}` |
| N fields | `"rrf"` | `[hit, ...]` (fused) |

---

## Object operations

```python
collection.get_objects(["p1", "p2"])     # full objects: {id, meta, filter, vectors, sparses, multi_vectors}
collection.delete_object("p1")           # {"deleted": "p1"}
collection.delete_by_filter([{"category": {"$eq": "footwear"}}])   # {"deleted": <count>}
collection.update_filters([{"id": "p2", "filter": {"category": "sale"}}])   # {"updated": <count>}
```

## Collection maintenance

```python
collection.describe()                              # {name, fields, created_at, layout_version}
collection.rebuild(field="embedding", m=24, ef_con=200)   # async; poll rebuild_status()
collection.rebuild_status()
collection.shrink()                                # reclaim disk
client.list_collections(); client.delete_collection("products")
```

## Backups (per collection)

A backup captures **one collection**; restoring it creates a **new collection**.
Backups are created on the collection; they're listed/restored/managed at the
database level (the database owns its backups).

```python
collection.create_backup("nightly")                    # back up THIS collection as a backup named "nightly" (async)
client.active_backup()                                  # status of the running backup; poll until it finishes
client.list_backups()                                   # list all backups in this database
client.backup_info("nightly")                           # metadata for the backup named "nightly"
client.restore_backup("nightly", "products_restored")   # restore backup "nightly" into a new collection "products_restored"
client.download_backup("nightly", "/tmp/nightly.tar")   # download backup "nightly" to the local file "/tmp/nightly.tar"
client.upload_backup("/tmp/nightly.tar")                # upload the local .tar file back into this database
client.delete_backup("nightly")                         # delete the backup named "nightly"
```

## Token management

**Admin (root token)** - manage databases and their tokens:

```python
admin.list_databases()                                       # list all databases
admin.get_database("alice")                                  # info for database "alice"
admin.deactivate_database("alice")                           # disable database "alice" (its tokens stop working; data kept)
admin.activate_database("alice")                             # re-enable database "alice"
admin.delete_database("alice")                               # delete database "alice" and ALL its data
admin.create_token("alice", name="ingest", token_type="rw")  # mint token "ingest" for db "alice"; returns the new db token (string)
admin.list_tokens("alice")                                   # list the token names/types of database "alice"
admin.delete_token("alice", "ingest")                        # delete the token named "ingest" from database "alice"
admin.list_db_collections("alice")                           # list collections in database "alice"
admin.list_all_collections()                                 # list collections across all databases
```

**Self-service (db token)** - a database manages its own tokens, no root needed:

```python
client.create_my_token("worker", token_type="r")            # mint token "worker" for YOUR db; returns the new db token (string)
client.list_my_tokens()                                     # list your db's token names/types
client.delete_my_token("worker")                            # delete your db's token named "worker"
```

`token_type`: `rw` (read-write, default) or `r` (read-only - 403 on writes).

## Server info

```python
client.health()    # {"status": "ok", ...}
client.stats()     # {"version", "uptime", "total_requests"}
```

---

## Reference

- **Limits (client-side, fail fast):** ≤ 10,000 objects per `upsert`; `limit` 1-4096;
  `ef_search` 1-1024; filter key ≤ 128 B, string value ≤ 1024 B; vector dim must
  match the field.
- **Errors:** non-2xx raise typed exceptions from `endee` - `EndeeException` (base),
  `APIException`, `AuthenticationException` (401), `ForbiddenException` (403),
  `NotFoundException` (404), `ConflictException` (409), `ServerException` (5xx).
  Bad inputs raise `ValueError` before any network call.

```python
from endee import EndeeException, NotFoundException
try:
    collection.upsert(objects)
except NotFoundException as e:
    ...
except EndeeException as e:
    ...
```

---

## Examples (`scripts/`)

- **`demo.py`** - full collection walkthrough. `python3 scripts/demo.py --token alice:xxxx`
- **`demo_admin.py`** - database administration. `python3 scripts/demo_admin.py --root-token <root>`
- **`test_filters.py`** - filter-operator checks. **`test_bulk_100k.py`** - bulk ingest + recall.
