Metadata-Version: 2.4
Name: endee
Version: 0.1.37b1
Summary: Endee is the Next-Generation Vector Database for Scalable, High-Performance AI
Home-page: https://endee.io
Author: Endee Labs
Author-email: dev@endee.io
Project-URL: Documentation, https://docs.endee.io
Keywords: vector database,embeddings,machine learning,AI,similarity search,HNSW,nearest neighbors
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: httpx[http2]>=0.28.1
Requires-Dist: numpy>=2.2.4
Requires-Dist: msgpack>=1.1.0
Requires-Dist: orjson>=3.11.5
Requires-Dist: pydantic<3,>=1.9.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license-file
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Endee — Python Client

Endee is a high-performance C++ vector database for AI search, RAG, and hybrid
retrieval. A **collection** holds objects, and each object can carry values for
several **named, typed fields** at once — dense vectors, sparse vectors, and
multi-vectors — so a single object can be searched many ways.

- **Multi-field objects** — one object, many vector fields (dense + sparse + multi-vector).
- **Hybrid search** — query any subset of fields; get per-field results, or fuse them with **optional RRF**.
- **Client-side normalization** — cosine vectors are L2-normalized for you.
- **Full control plane** — collections, objects, filters, rebuild/shrink, backups, databases, and tokens.

---

## Table of contents

1. [Install](#install)
2. [Quick start](#quick-start)
3. [Connecting & tokens](#connecting--tokens)
4. [Create a collection](#create-a-collection)
5. [Insert objects (combining vector types)](#insert-objects-combining-vector-types)
6. [Search](#search)
   - [Single field](#single-field-search)
   - [Per-field options](#per-field-options-limit--ef_search)
   - [Multi-field **without** a reranker](#multi-field-search-without-a-reranker)
   - [Multi-field **with** RRF](#multi-field-search-with-rrf)
   - [Filters](#filtered-search)
7. [Object operations](#object-operations)
8. [Collection maintenance](#collection-maintenance)
9. [Backups](#backups)
10. [Database administration (root token)](#database-administration-root-token)
11. [Self-service tokens](#self-service-tokens)
12. [Server info](#server-info)
13. [Reference](#reference)
14. [Error handling](#error-handling)

---

## Install

```bash
pip install endee
```

Requires Python ≥ 3.9. Depends on `numpy`, `msgpack`, `orjson`, `requests`, `httpx`.

---

## Quick start

```python
from endee import Endee

# A "db token" (db_name:secret) scopes you to one database.
client = Endee(token="alice:xxxxxxxx")
client.set_base_url("http://localhost:8080/api/v2")   # include /api/v2

# 1. Create a collection with one of each field type.
client.create_collection(
    name="products",
    fields=[
        {"name": "embedding", "type": "vector",
         "params": {"dimension": 8, "space_type": "cosine", "precision": "int8"}},
        {"name": "keywords", "type": "sparse", "sparse_model": "default"},
        {"name": "colbert", "type": "multi_vector",
         "params": {"dimension": 8, "space_type": "cosine", "precision": "int8",
                    "pooling": "mean"}},
    ],
)
collection = client.get_collection("products")

# 2. Insert an object carrying ALL THREE field types at once.
collection.upsert([{
    "id": "p1",
    "meta": {"name": "Wireless Headphones", "price": 99},
    "filter": {"category": "electronics"},
    "fields": {
        "embedding": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
        "keywords":  {"indices": [3, 17, 42], "values": [0.9, 0.5, 0.2]},
        "colbert":   [[0.1]*8, [0.2]*8],
    },
}])

# 3. Search the dense field.
hits = collection.search(fields={"embedding": [0.2]*8}, limit=5)
for h in hits["results"]:
    print(h["id"], h["similarity"], h["meta"])
```

---

## Connecting & tokens

**Authentication is always required.** The server has no anonymous mode — every
request must carry a token, or it is rejected with **401 Unauthorized**. (Only
`health()` and `stats()` are unauthenticated.)

> **Default root token (dev):** if the server is started **without**
> `NDD_ROOT_TOKEN`, it falls back to the well-known **insecure** token
> **`unsafe_root_token`** (and logs a loud startup warning). Use that value as
> your root token for all root/admin operations on such a server:
> ```python
> admin = Endee(token="unsafe_root_token")
> admin.set_base_url("http://localhost:8080/api/v2")
> ```
> This is for local/dev only — **production must set `NDD_ROOT_TOKEN`.**

There are two kinds of token:

| Token | Format | Used for |
|-------|--------|----------|
| **root token** | the server's `NDD_ROOT_TOKEN` | Database administration (create/delete databases, mint tokens). **Always required to bootstrap** — see below. |
| **db token** | `db_name:secret` | All collection / object / search / backup work inside one database. |

### You always start from the root token

There is **no default database**. Before any data work exists, you must use the
**root token** to create a database — which returns its first **db token**. So
the root token is the entry point to every deployment:

```python
# 1) Admin client — uses the ROOT token (the server's NDD_ROOT_TOKEN).
admin = Endee(token="roottoken")
admin.set_base_url("http://localhost:8080/api/v2")

# 2) Create a database; this returns the db token apps will use.
db_token = admin.create_database("alice", db_type="scale")["db_token"]

# 3) App client — uses that DB token for all collection/data work.
client = Endee(token=db_token)
client.set_base_url("http://localhost:8080/api/v2")
```

Notes:
- The **root token** is the sole admin. It **cannot run data ops directly** — it
  administers databases/tokens, and a db token (which it mints) does the data work.
- `set_base_url` must point at the `/api/v2` root of your server.
- A missing/invalid token → `AuthenticationException` (401); a read-only token on
  a write → `ForbiddenException` (403).

---

## Create a collection

A collection is a set of **named, typed fields**. Each field is a plain dict sent
to the server as-is.

| Field type | Required keys | `params` |
|------------|---------------|----------|
| `vector` | `name`, `type` | `dimension`, `space_type`, `precision` (+ optional `M`, `ef_con`) |
| `sparse` | `name`, `type`, `sparse_model` | — (no `params`) |
| `multi_vector` | `name`, `type` | `dimension`, `space_type`, `precision`, **`pooling`** (`"mean"`/`"max"`) (+ optional `M`, `ef_con`) |

```python
client.create_collection(
    name="products",
    fields=[
        # Dense embedding (single vector per object)
        {"name": "embedding", "type": "vector",
         "params": {"dimension": 768, "space_type": "cosine", "precision": "int8"}},

        # Sparse / keyword field (BM25-style)
        {"name": "keywords", "type": "sparse", "sparse_model": "default"},

        # Multi-vector field (many vectors per object)
        {"name": "colbert", "type": "multi_vector",
         "params": {"dimension": 768, "space_type": "cosine", "precision": "int8",
                    "pooling": "mean"}},
    ],
)
```

- `space_type`: `cosine` (default), `l2`, `ip`.
- `precision`: `float32`, `float16`, `int16` (default), `int8`, `int8e`, `binary`.
- `M` / `ef_con`: HNSW build params (optional; sensible defaults applied).
- **Cosine vectors are L2-normalized by the client** before sending.

Other client-level collection calls:

```python
client.list_collections()              # ["products", ...]
collection = client.get_collection("products")
collection.describe()                  # {name, fields, created_at, layout_version}
client.delete_collection("products")
```

---

## Insert objects (combining vector types)

`collection.upsert(objects)` takes a list of objects. Each object is:

```python
{
    "id":     "<unique string id>",
    "meta":   {...},          # arbitrary JSON, returned in search results (optional)
    "filter": {...},          # tag dict used by filtered search (optional)
    "fields": {               # field_name -> value, ONE entry per field you set
        "<dense_field>":  [float, ...],                       # dense   → list
        "<sparse_field>": {"indices": [...], "values": [...]},# sparse  → dict
        "<multi_field>":  [[float, ...], [float, ...], ...],  # multi   → list of lists
    },
}
```

Per-field value formats:

| Field type | Value format |
|------------|--------------|
| `vector` | `[0.1, 0.2, ...]` — a flat list of floats |
| `sparse` | `{"indices": [3, 17], "values": [0.9, 0.5]}` (also accepts `sparse_indices`/`sparse_values`) |
| `multi_vector` | `[[...], [...], ...]` — a list of equal-length float lists |

### One object, multiple field types

A single object can populate **any combination** of the collection's fields in one
call. They're stored together under the same id and can be searched independently
or together:

```python
collection.upsert([
    {
        "id": "p1",
        "meta": {"name": "Wireless Headphones", "price": 99},
        "filter": {"category": "electronics"},
        "fields": {
            "embedding": [0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80],   # dense
            "keywords":  {"indices": [3, 17, 42], "values": [0.9, 0.5, 0.2]}, # sparse
            "colbert":   [                                                     # multi-vector
                [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
                [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
            ],
        },
    },
    {
        "id": "p2",
        "meta": {"name": "Running Shoes"},
        "filter": {"category": "footwear"},
        "fields": {
            # An object may set only SOME fields — here just dense + sparse.
            "embedding": [0.50, 0.40, 0.30, 0.20, 0.10, 0.05, 0.02, 0.01],
            "keywords":  {"indices": [5, 17, 90], "values": [0.7, 0.6, 0.1]},
        },
    },
])
# → {"upserted": 2}
```

Notes:
- You don't have to fill every field on every object — set whatever subset you have.
- `upsert` is insert-or-replace by `id`.
- Cosine `vector`/`multi_vector` values are normalized client-side before sending.
- Validation is done client-side (fails fast): batch size ≤ 10,000, no duplicate
  ids in a batch, dense/multi dims must match the field, sparse `indices`/`values`
  lengths must match, filter keys ≤ 128 bytes / string values ≤ 1024 bytes.

---

## Search

`collection.search(fields=..., limit=..., reranker=...)` queries one or more
fields **in a single request**. You query a field with the same shape you'd insert:

| Field type | Query shape |
|------------|-------------|
| `vector` | `[0.2, 0.2, ...]` |
| `sparse` | `{"indices": [3, 17], "values": [0.8, 0.4]}` |
| `multi_vector` | `[[...], [...]]` |

Each result **hit** is:

```python
{"id": "p1", "similarity": 0.94, "meta": {...}, "filter": {...}}
```

> Search results carry `meta`/`filter` but **not** the stored vectors. To get the
> vectors back, use [`get_objects`](#object-operations).

### Single-field search

When you query exactly one field, `results` is a ranked **list**:

```python
res = collection.search(fields={"embedding": [0.2]*8}, limit=3)
res["results"]
# [
#   {"id": "p1", "similarity": 0.97, "meta": {...}, "filter": {...}},
#   {"id": "p2", "similarity": 0.88, "meta": {...}, "filter": {...}},
# ]

# sparse
collection.search(fields={"keywords": {"indices": [3, 17], "values": [0.8, 0.4]}}, limit=3)

# multi-vector
collection.search(fields={"colbert": [[0.1]*8, [0.2]*8]}, limit=3)
```

### Per-field options (`limit` / `ef_search`)

Pass a config dict per field to control how many candidates that field fetches and
its HNSW `ef_search`:

```python
collection.search(fields={
    "embedding": {"query": [0.2]*8, "limit": 50, "ef_search": 256},
    "keywords":  {"query": {"indices": [3, 17], "values": [0.8, 0.4]}, "limit": 20},
})
```

### Multi-field search **without** a reranker

When you query **multiple fields and don't pass a reranker**, you get **each
field's own ranked list, unfused** — `results` is a **dict keyed by field name**.
This lets you merge or display them however you like.

```python
res = collection.search(
    fields={
        "embedding": {"query": [0.2, 0.2, 0.3, 0.3], "limit": 2},
        "keywords":  {"query": {"indices": [3, 17], "values": [0.8, 0.4]}, "limit": 2},
    },
    limit=2,
)
```

```json
{
  "results": {
    "embedding": [
      {"id": "p1", "meta": {"name": "Wireless Headphones"},
       "filter": {"category": "electronics"}, "similarity": 0.4938},
      {"id": "p2", "meta": {"name": "Running Shoes"},
       "filter": {"category": "footwear"},    "similarity": 0.4505}
    ],
    "keywords": [
      {"id": "p1", "meta": {"name": "Wireless Headphones"},
       "filter": {"category": "electronics"}, "similarity": 0.9200},
      {"id": "p2", "meta": {"name": "Running Shoes"},
       "filter": {"category": "footwear"},    "similarity": 0.5591}
    ]
  }
}
```

```python
res["results"]["embedding"]   # this field's ranked hits
res["results"]["keywords"]    # that field's ranked hits
```

> `similarity` is in **each field's own scale** (cosine for dense ≈ 0.49, sparse
> dot for keywords ≈ 0.92) — scores are **not comparable across fields**. That's
> exactly why fusion is opt-in.

### Multi-field search **with** RRF

Pass `reranker="rrf"` to fuse the per-field lists into **one ranked list** using
Reciprocal Rank Fusion. Now `results` is a flat list with a single fused score:

```python
res = collection.search(
    fields={
        "embedding": {"query": [0.2, 0.2, 0.3, 0.3], "limit": 50},
        "keywords":  {"query": {"indices": [3, 17], "values": [0.8, 0.4]}, "limit": 50},
        "colbert":   {"query": [[0.1]*8, [0.2]*8], "limit": 50},
    },
    limit=10,
    reranker="rrf",
    # optional: weight each field's contribution (must sum to 1.0; equal by default)
    field_weights={"embedding": 0.5, "keywords": 0.3, "colbert": 0.2},
    # optional: RRF rank constant k (default 60)
    rrf_k=60,
)
```

```json
{
  "results": [
    {"id": "p1", "similarity": 0.0164, "meta": {...}, "filter": {...}},
    {"id": "p3", "similarity": 0.0160, "meta": {...}, "filter": {...}},
    {"id": "p2", "similarity": 0.0160, "meta": {...}, "filter": {...}}
  ]
}
```

**Summary of return shapes**

| Query | `reranker` | `results` shape |
|-------|-----------|-----------------|
| 1 field | (any) | `[ hit, ... ]` |
| N fields | `None` (default) | `{ field_name: [ hit, ... ], ... }` |
| N fields | `"rrf"` | `[ hit, ... ]` (fused) |

`reranker` accepts only `None` or `"rrf"`. `field_weights` / `rrf_k` are used only
when `reranker="rrf"`.

### Filtered search

`filter` applies to any search mode. It's an array of conditions:

```python
collection.search(
    fields={"embedding": [0.2]*8},
    limit=5,
    filter=[{"category": {"$eq": "electronics"}}],
)
```

Operators: `$eq`, `$in`, `$range`, `$gt`, `$gte`, `$lt`, `$lte` (see [Reference](#reference)).

---

## Object operations

```python
# Fetch full stored objects by id (meta, filter, and the stored vectors).
objs = collection.get_objects(["p1", "p2"])
# [
#   {"id": "p1", "meta": {...}, "filter": {...},
#    "vectors": {"embedding": [...]},               # dense fields (normalized)
#    "sparses": {"keywords": {"indices": [...], "values": [...]}},
#    "multi_vectors": {"colbert": [[...], [...]]}},
#   ...
# ]

# Delete a single object by id.
collection.delete_object("p1")                      # {"deleted": "p1"}

# Delete every object matching a filter.
collection.delete_by_filter([{"category": {"$eq": "footwear"}}])   # {"deleted": <count>}

# Update only the filter tags on existing objects (no re-upsert of vectors).
collection.update_filters([
    {"id": "p2", "filter": {"category": "sale"}},
])                                                  # {"updated": <count>}
```

---

## Collection maintenance

```python
collection.describe()                 # {name, fields, created_at, layout_version}

# Rebuild a field's HNSW graph with new M / ef_con (async).
collection.rebuild(field="embedding", m=24, ef_con=200)   # {status: "in_progress", ...}
collection.rebuild_status()                               # {percent_complete, ...}

# Defragment storage in place.
collection.shrink()                   # {"status": "ok", "reclaimed_bytes": <int>}
```

---

## Backups

Backups are per-database. Creating one is per-collection and **asynchronous**.

```python
import time

# Start a backup of a collection (async).
collection.create_backup("nightly")          # {"backup_name": "nightly", "status": "in_progress"}

# Poll for completion.
while client.active_backup().get("active"):
    time.sleep(0.5)

client.list_backups()                         # {backup_name: {metadata}, ...}
client.backup_info("nightly")                 # metadata for one backup

# Restore into a NEW collection.
client.restore_backup("nightly", "products_restored")

# Move backups around.
client.download_backup("nightly", "/tmp/nightly.tar")   # writes a .tar, returns the path
client.upload_backup("/tmp/nightly.tar")                # uploads a .tar into this db

client.delete_backup("nightly")
```

---

## Database administration (root token)

These operate **across databases** and require the **root token** (the server's
`NDD_ROOT_TOKEN`). Create an admin client with that token:

```python
admin = Endee(token="roottoken")
admin.set_base_url("http://localhost:8080/api/v2")
```

### Databases

```python
# Create a database — returns a default db token you hand to an app.
res = admin.create_database("alice", db_type="scale")
db_token = res["db_token"]                     # e.g. "alice:8w12..."
# res == {"db_name": "alice", "db_type": "Scale", "db_token": "...",
#         "name": "default", "message": "Db created successfully"}

admin.list_databases()
# [{"db_name": "alice", "db_type": "Scale", "is_active": True, "created_at": 178...}, ...]

admin.get_database("alice")
# {"db_name": "alice", "db_type": "Scale", "is_active": True,
#  "created_at": 178..., "token_count": 1}

admin.set_database_type("alice", "pro")        # change tier
admin.deactivate_database("alice")             # block its tokens (data is kept)
admin.activate_database("alice")               # re-enable
admin.delete_database("alice")                 # delete db + ALL its data
```

`db_type` is one of `starter`, `pro`, `scale`, `enterprise` (each tier carries
per-tier limits on objects, collections, dimension, etc.).

### Collections across databases (admin view)

```python
admin.list_db_collections("alice")             # ["products", ...] for one db
admin.list_all_collections()                   # [{"db_name": "alice", "collections": [...]}, ...]
admin.delete_db_collection("alice", "products")
```

### Tokens for any database

```python
# Create a named token for a db. token_type: "rw" (default) or "r" (read-only).
admin.create_token("alice", name="ingest", token_type="rw")
# {"db_token": "alice:...", "name": "ingest", "db_name": "alice", "token_type": "rw"}

admin.list_tokens("alice")
# [{"name": "ingest", "db_name": "alice", "token_type": "rw", "created_at": 178...}, ...]

admin.delete_token("alice", "ingest")
```

A **read-only** token (`token_type="r"`) gets a 403 on any mutating request
(upsert, delete, create collection, …).

### Typical admin → app flow

```python
# 1. Admin (root token) provisions a database and gets its token.
admin = Endee(token="roottoken"); admin.set_base_url(".../api/v2")
db_token = admin.create_database("alice", db_type="scale")["db_token"]

# 2. The app uses that db token for all data work.
app = Endee(token=db_token); app.set_base_url(".../api/v2")
app.create_collection("products", fields=[...])
```

---

## Self-service tokens

A **db token** holder can manage their own database's tokens **without** the root
token (e.g. an app team rotating its own keys):

```python
client = Endee(token="alice:xxxx"); client.set_base_url(".../api/v2")

client.create_my_token("worker", token_type="r")   # {"db_token": "alice:...", "token_type": "r"}
client.list_my_tokens()                             # [{"name": "worker", "token_type": "r", ...}, ...]
client.delete_my_token("worker")
```

---

## Server info

```python
client.health()    # {"status": "ok", "timestamp": ...}
client.stats()     # {"version": "...", "uptime": ..., "total_requests": ...}
```

---

## Reference

**Distance / space types** (`space_type`)

| Value | Meaning |
|-------|---------|
| `cosine` | Cosine similarity (vectors L2-normalized client-side). Default. |
| `l2` | Euclidean distance. |
| `ip` | Inner product. |

**Precisions** (`precision`) — memory/accuracy trade-off

`float32`, `float16`, `int16` (default), `int8`, `int8e`, `binary`.

**Database tiers** (`db_type`): `starter`, `pro`, `scale`, `enterprise`.

**Token types** (`token_type`): `rw` (read-write), `r` (read-only).

**Filter operators**

| Operator | Example | Meaning |
|----------|---------|---------|
| `$eq` | `{"category": {"$eq": "news"}}` | equals |
| `$in` | `{"tag": {"$in": ["a", "b"]}}` | in set |
| `$range` | `{"price": {"$range": [10, 100]}}` | inclusive range |
| `$gt` / `$gte` | `{"price": {"$gt": 50}}` | greater than / or equal |
| `$lt` / `$lte` | `{"price": {"$lte": 100}}` | less than / or equal |

**Client-side validation / limits**

| Limit | Value |
|-------|-------|
| Objects per `upsert` | ≤ 10,000 |
| `limit` (top-k) | 1 … 4096 |
| `ef_search` | 1 … 1024 |
| Filter key | ≤ 128 bytes |
| Filter string value | ≤ 1024 bytes |
| Vector dimension | must match the field's configured dimension |

---

## Error handling

Non-2xx responses raise typed exceptions:

```python
from endee import (
    EndeeException,           # base class for all client errors
    APIException,             # generic 4xx (e.g. bad request)
    AuthenticationException,  # 401
    ForbiddenException,       # 403 (e.g. read-only token on a write)
    NotFoundException,        # 404
    ConflictException,        # 409 (e.g. duplicate collection)
    ServerException,          # 5xx
)

try:
    collection.upsert(objects)
except NotFoundException as e:
    print("not found:", e)
except EndeeException as e:           # catches all of the above
    print("request failed:", e)
```

All exception classes are importable from `endee` (and `endee.exceptions`):
`EndeeException` (base), `APIException`, `AuthenticationException`,
`ForbiddenException`, `NotFoundException`, `ConflictException`,
`ServerException`, `SubscriptionException`.

Client-side input mistakes (bad dimensions, mismatched sparse lengths, oversized
batch, sum-of-weights ≠ 1.0, etc.) raise `ValueError` **before** any network call.

---

## End-to-end examples

See `scripts/`:

- **`demo.py`** — full collection walkthrough (create → upsert → every search mode → backups → cleanup). Run with a db token: `python3 scripts/demo.py --token alice:xxxx`.
- **`demo_admin.py`** — database administration. Run with the root token: `python3 scripts/demo_admin.py --root-token roottoken`.
- **`test_bulk_100k.py`** — bulk ingest + self-recall benchmark.
