Metadata-Version: 2.4
Name: zeptodb
Version: 0.1.4
Summary: Python client and DataFrame integration for ZeptoDB — ultra-low latency columnar time-series database
Project-URL: Homepage, https://zeptodb.com
Project-URL: Repository, https://github.com/ZeptoDB/ZeptoDB
Project-URL: Documentation, https://docs.zeptodb.com
Author: ZeptoDB Contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: HFT,analytics,columnar,database,finance,timeseries
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Provides-Extra: all
Requires-Dist: duckdb>=0.9; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: polars>=0.20; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Provides-Extra: arrow
Requires-Dist: pyarrow>=14.0; extra == 'arrow'
Provides-Extra: dev
Requires-Dist: duckdb>=0.9; extra == 'dev'
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: polars>=0.20; extra == 'dev'
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb>=0.9; extra == 'duckdb'
Provides-Extra: pandas
Requires-Dist: numpy>=1.24; extra == 'pandas'
Requires-Dist: pandas>=2.0; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars>=0.20; extra == 'polars'
Description-Content-Type: text/markdown

<div align="center">

# ⚡ ZeptoDB

### Agent Memory for Live Time-Series Data

*Ingest millions of events per second. Retrieve the evidence, context, and cache entries AI agents need before they act.*

[![C++20](https://img.shields.io/badge/C%2B%2B-20-blue?logo=cplusplus)](https://en.cppreference.com/w/cpp/20)
[![LLVM 19](https://img.shields.io/badge/LLVM-19-orange?logo=llvm)](https://llvm.org/)
[![Highway SIMD](https://img.shields.io/badge/SIMD-Highway-green)](https://github.com/google/highway)
[![Tests](https://img.shields.io/badge/tests-830%2B%20passing-brightgreen?logo=googletest)](tests/)
[![License](https://img.shields.io/badge/License-BUSL_1.1-blue)](LICENSE)
[![Website](https://img.shields.io/badge/website-zeptodb.com-00e5a0?logo=astro)](https://zeptodb.com)
[![Discord](https://img.shields.io/discord/1492174712359354590?color=5865F2&logo=discord&logoColor=white&label=Discord)](https://discord.gg/zeptodb)
[![Docker Pulls](https://img.shields.io/docker/pulls/zeptodb/zeptodb?logo=docker)](https://hub.docker.com/r/zeptodb/zeptodb)
[![PyPI](https://img.shields.io/pypi/v/zeptodb?logo=python&logoColor=white)](https://pypi.org/project/zeptodb/)

[Website](https://zeptodb.com) · [Agent Memory](https://zeptodb.com/use-cases/agent-memory/) · [Quick Start](https://zeptodb.com/getting-started/quick_start/) · [Benchmarks](https://zeptodb.com/benchmarks/) · [Docs](https://zeptodb.com/docs/) · [Community](#-community)

</div>

---

<!--
<div align="center">
  <img src="docs/assets/demo.gif" alt="ZeptoDB Demo" width="720">
  <br><em>From zero to query results in 30 seconds</em>
</div>
-->

## What is ZeptoDB?

ZeptoDB is an in-memory columnar database purpose-built for live time-series analytics
and AI agent memory.

It handles **high-throughput ingestion** and **real-time analytical queries** simultaneously — without trade-offs between the two.

The core engine is hardware-software co-optimized: Highway SIMD vectorization,
LLVM JIT compilation, lock-free ring buffers, NUMA-aware allocation, and
UCX/RDMA networking — all working together to eliminate unnecessary copies,
allocations, and cache misses. The AI memory layer adds agent-scoped context for
applications built on live time-series data: client-supplied embedding storage,
parallel filtered context retrieval, optional experimental ANN candidate
generation, token-budget assembly, and exact/semantic prompt cache lookup
without calling LLM providers from the server. Agent examples live
under [`examples/agent_memory`](examples/agent_memory/) and include provider
cache, LangGraph-style memory, optional provider adapters, AgentOps telemetry,
and agent-attached time-series demos for finance, IoT, observability, robotics,
and game/live-ops.

New to the project? Start with the website:

| Path | What it gives you |
|------|-------------------|
| [Agent Memory](https://zeptodb.com/use-cases/agent-memory/) | How live time-series becomes agent context, cache, and replay |
| [Python Agent Memory Quickstart](https://zeptodb.com/use-cases/agent-memory-python-quickstart/) | Copy-paste memory, context, cache, and write-back flow |
| [Benchmarks](https://zeptodb.com/benchmarks/) | Ingestion, query latency, zero-copy Python, and Agent Memory numbers |
| [Features](https://zeptodb.com/features/) | Time-series core, memory layer, APIs, storage, and security |
| [Quick Start](https://zeptodb.com/getting-started/quick_start/) | First local query and client setup |

```
┌─────────────────────────────────────────────────────────────┐
│  Clients: HTTP API · Python DSL · C++ API · Arrow Flight    │
├─────────────────────────────────────────────────────────────┤
│  AI Memory: vertical context · agent telemetry · cache      │
├─────────────────────────────────────────────────────────────┤
│  SQL Engine: Parser (1.5μs) · AST Optimizer · Executor      │
├─────────────────────────────────────────────────────────────┤
│  Execution: Highway SIMD · LLVM JIT · Partition-parallel    │
│  ASOF JOIN · Window JOIN · xbar · EMA · VWAP                │
├─────────────────────────────────────────────────────────────┤
│  Ingestion: Lock-free MPMC Ring Buffer · WAL · Feed Handlers│
├─────────────────────────────────────────────────────────────┤
│  Storage: Arena Allocator · Column Store · Tiered (→S3)     │
├─────────────────────────────────────────────────────────────┤
│  Cluster: Consistent Hashing · RF=2 · Auto Failover         │
├─────────────────────────────────────────────────────────────┤
│  Security: TLS · JWT/OIDC · RBAC · Audit (SOC2/MiFID II)   │
└─────────────────────────────────────────────────────────────┘
```

---

## 🔍 Why ZeptoDB?

| | **ZeptoDB** | **kdb+** | **ClickHouse** | **TimescaleDB** | **QuestDB** |
|---|:---:|:---:|:---:|:---:|:---:|
| **Ingestion** | 5.52M evt/s | ~5M evt/s | ~100K evt/s | ~50K evt/s | ~1M evt/s |
| **Filter 1M rows** | 272μs | ~300μs | ~10ms | ~50ms | ~2ms |
| **ASOF JOIN** | ✅ Native | ✅ Native | ❌ UDF | ❌ Manual | ✅ Native |
| **SQL** | ✅ Standard | ❌ q lang | ✅ Dialect | ✅ PostgreSQL | ✅ Dialect |
| **Python zero-copy** | 522ns | ❌ IPC only | ❌ | ❌ | ❌ |
| **License cost** | Free (BUSL-1.1) | $100K–500K/yr | Free (Apache 2.0) | Free (Apache 2.0) | Free (Apache 2.0) |

**TL;DR:** kdb+ class time-series performance, standard SQL, zero-copy Python, and a native Agent Memory layer for context retrieval, prompt cache, and AgentOps telemetry.

---

## 🚀 Quick Start

| Method | Command |
|--------|---------|
| **Binary** | Download from [GitHub Releases](https://github.com/ZeptoDB/ZeptoDB/releases) |
| **Homebrew** | `brew install ZeptoDB/tap/zeptodb` |
| **Docker** | `docker run -p 8123:8123 zeptodb/zeptodb:0.0.1` |
| **PyPI** | `pip install zeptodb` |
| **Source** | [Build instructions below](#build-from-source) |

### Binary

```bash
# amd64
curl -LO https://github.com/ZeptoDB/ZeptoDB/releases/latest/download/zeptodb-linux-amd64-0.0.1.tar.gz
tar xzf zeptodb-linux-amd64-0.0.1.tar.gz
./zeptodb-linux-amd64-0.0.1/zepto_http_server --port 8123

# arm64 (AWS Graviton)
curl -LO https://github.com/ZeptoDB/ZeptoDB/releases/latest/download/zeptodb-linux-arm64-0.0.1.tar.gz
```

> **Note:** Prebuilt binaries require runtime libraries (LLVM 19, Arrow, etc.). See the [Binary Installation Guide](docs/getting-started/BINARY_INSTALL.md) for prerequisites and troubleshooting.

### Docker

```bash
docker run -p 8123:8123 zeptodb/zeptodb:0.0.1

# Insert data
curl -X POST http://localhost:8123/ \
  -d "INSERT INTO trades VALUES (1, 1714000000000000000, 185.50, 100)"

# Query
curl -X POST http://localhost:8123/ \
  -d "SELECT vwap(price, volume), count(*) FROM trades WHERE symbol = 'AAPL'"
```

### Python

```python
import zeptodb

db = zeptodb.Pipeline()
db.start()
db.ingest(symbol=1, price=185.50, volume=100)
db.drain()

# Zero-copy numpy access (522ns)
prices = db.get_column(symbol=1, name="price")
```

### Build from Source

```bash
# Dependencies (Amazon Linux 2023 / Fedora)
sudo dnf install -y clang19 clang19-devel llvm19-devel \
  highway-devel numactl-devel ucx-devel ninja-build lz4-devel

mkdir -p build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang-19 -DCMAKE_CXX_COMPILER=clang++-19
ninja -j$(nproc)

./zepto_http_server --port 8123
```

📖 Full guide: [Quick Start](https://zeptodb.com/getting-started/quick_start/) · [Agent Memory](https://zeptodb.com/use-cases/agent-memory/) · [Python Reference](https://zeptodb.com/api/python_reference/) · [SQL Reference](https://zeptodb.com/api/sql_reference/)

---

## 📊 Performance

Single node. End-to-end latencies including SQL parsing. No cherry-picking.

| Operation | Latency | Notes |
|-----------|---------|-------|
| **Ingestion throughput** | **5.52M events/sec** | Lock-free MPMC ring buffer |
| Filter 1M rows | **272μs** | Highway SIMD vectorized scan |
| VWAP 1M rows | **532μs** | Fused price×volume aggregation |
| GROUP BY (8 threads) | **248μs** | Partition-parallel scatter/gather |
| EMA 1M rows | **2.2ms** | Streaming exponential moving average |
| Window SUM 1M rows | **1.36ms** | Prefix-sum O(n) algorithm |
| xbar (1M → 3,334 bars) | **11ms** | Time-bucketed OHLCV |
| SQL parse | **1.5–4.5μs** | Recursive descent, zero allocation |
| Python column access | **522ns** | Zero-copy shared memory |
| Indexed lookup (g#/p#) | **3.3μs** | 274× faster than full scan |
| HDB flush to disk | **4.8 GB/s** | LZ4 compressed |
| Partition routing | **2ns** | Consistent hash ring |

---

## 💡 SQL Examples

```sql
-- 5-minute OHLCV candlestick bars
SELECT xbar(timestamp, 300000000000) AS bar,
       first(price) AS open, max(price) AS high,
       min(price) AS low, last(price) AS close,
       sum(volume) AS volume
FROM trades WHERE symbol = 'AAPL'
GROUP BY xbar(timestamp, 300000000000)

-- ASOF JOIN (point-in-time lookup)
SELECT t.price, q.bid, q.ask
FROM trades t
ASOF JOIN quotes q
ON t.symbol = q.symbol AND t.timestamp >= q.timestamp

-- EMA with delta
SELECT symbol, price,
       EMA(price, 20) OVER (PARTITION BY symbol ORDER BY timestamp) AS ema20,
       DELTA(price) OVER (ORDER BY timestamp) AS price_change
FROM trades

-- Window JOIN (time-range aggregation)
SELECT t.price, wj_avg(q.bid) AS avg_bid
FROM trades t
WINDOW JOIN quotes q ON t.symbol = q.symbol
AND q.timestamp BETWEEN t.timestamp - 5000000000 AND t.timestamp + 5000000000

-- Materialized view (incremental, updated on ingest)
CREATE MATERIALIZED VIEW ohlcv_5min AS
  SELECT symbol, xbar(timestamp, 300000000000) AS bar,
         first(price) AS open, max(price) AS high,
         min(price) AS low, last(price) AS close,
         sum(volume) AS vol
  FROM trades
  GROUP BY symbol, xbar(timestamp, 300000000000)

-- Storage tiering
ALTER TABLE trades SET STORAGE POLICY
  HOT 1 HOURS WARM 24 HOURS COLD 30 DAYS DROP 365 DAYS
```

Full SQL reference: [SQL_REFERENCE.md](docs/api/SQL_REFERENCE.md) — INSERT, UPDATE, DELETE, CASE WHEN, LIKE, UNION, CTE, subqueries, and more.

---

## 🏗️ Use Cases

| Domain | Why ZeptoDB | Key Features |
|--------|------------|--------------|
| **Finance / HFT** | Sub-ms tick processing, kdb+-class perf | ASOF JOIN, xbar, EMA, VWAP |
| **Quant Research** | Backtest in Python, execute in C++ | Zero-copy numpy, Polars DSL |
| **Agentic AI** | Add agents to time-series workflows without losing operational context | Agent-scoped memory, context assembly, exact/semantic cache |
| **Crypto / DeFi** | 24/7 multi-exchange streaming | Binance feed handler, real-time agg |
| **IoT / Manufacturing** | High-frequency sensor ingestion and OPC-UA integration | OPC-UA client/server, Telegraf output, Historical Access, DELTA/RATIO, time-bar agg, LZ4 |
| **Physical AI / Robotics** | Typed ROS 2 telemetry, rosbag2 replay, feature store | [ROS 2 roadmap](docs/design/ros2_physical_ai_roadmap.md), [ROS 2 setup](docs/operations/ROS2_SETUP.md), [edge guide](docs/operations/ROS2_EDGE_DEPLOYMENT.md), [use cases](docs/usecases/physical_ai.md), spatial SQL |
| **Autonomous Vehicles** | Sensor fusion, driving log replay | ASOF JOIN, Parquet HDB, parallel scan |
| **Observability** | High-cardinality metrics | SQL + Grafana, TTL + S3 tiering |

---

## ⚙️ Optimization Stack

<table>
<tr><th>Hardware</th><th>Software</th></tr>
<tr><td>

- **Highway SIMD** — 256/512-bit vectorized scans
- **NUMA-aware** — memory pinned to local node
- **UCX / RDMA** — kernel-bypass networking
- **Arena allocator** — zero GC, zero fragmentation

</td><td>

- **LLVM JIT** — runtime expression compilation
- **Lock-free MPMC** — zero-contention ingestion
- **Columnar storage** — cache-friendly sequential access
- **Partition-parallel** — 3.48× scaling at 8 threads

</td></tr>
</table>

---

## 🔒 Enterprise Security

| Feature | Details |
|---------|---------|
| TLS/HTTPS | OpenSSL 3.2, cert/key PEM |
| Authentication | API Key (SHA256) + JWT/OIDC (HS256/RS256, JWKS auto-fetch) |
| Authorization | RBAC: 5 roles + symbol-level ACL + multi-tenancy |
| Rate Limiting | Token bucket per-identity + per-IP |
| Secrets | Vault KV v2 → K8s secrets → env var (priority chain) |
| Audit Log | 7-year retention, SOC2/EMIR/MiFID II compliant |

---

## 🚢 Deployment

```bash
# Docker
docker run -p 8123:8123 zeptodb/zeptodb

# Helm
helm install zeptodb ./deploy/helm/zeptodb

# Bare-metal (systemd)
./deploy/scripts/install_service.sh
```

Guides: [Production Deployment](https://zeptodb.com/deployment/production_deployment/) · [Kubernetes](https://zeptodb.com/operations/kubernetes_operations/) · [Bare-metal Tuning](https://zeptodb.com/deployment/bare_metal_tuning/)

---

## 🔄 Migration

Migrate from existing databases with built-in tooling:

```bash
./zepto-migrate --source kdb+ --hdb-path /data/hdb --target localhost:8123
```

Supported: **kdb+** (HDB loader, q→SQL) · **ClickHouse** (DDL/query conversion) · **DuckDB** (Parquet) · **TimescaleDB** (hypertable conversion)

---

## 💬 Community

- [Website](https://zeptodb.com) — docs, benchmarks, comparisons, and Agent Memory guides
- [Discord](https://discord.gg/zeptodb) — questions, discussions, real-time help
- [GitHub Discussions](https://github.com/ZeptoDB/ZeptoDB/discussions) — design proposals, Q&A, ideas
- [Twitter/X](https://twitter.com/zeptodb) — release announcements, benchmarks

---

## 🛠️ Built with Kiro

ZeptoDB is developed using **[Kiro](https://kiro.dev)** (AI coding assistant, IDE + CLI) on top of a large existing C++20 codebase. Three Kiro features do most of the heavy lifting:

1. **Multi-agent orchestration** — an orchestrator (`zeptodb-dev`) delegates to `zepto-developer`, `zepto-reviewer`, and `zepto-qa` subagents. Reviewer is tool-restricted to read-only; QA runs builds on x86_64 and aarch64 in parallel with review. See `.kiro/agents/`.
2. **MCP + Skills** — six keyword-activated `SKILL.md` knowledge packs (design-doc index, layer patterns, edge-case catalog, doc-update checklist, review checklist, cross-arch verification) plus a Playwright MCP server for Web UI smoke tests. See `.kiro/skills/` and `.kiro/settings/mcp.json`.
3. **Persistent context** — `.kiro/KIRO.md` (project rulebook) + per-agent `.kiro/context/*.md` are auto-loaded on every session via `.kiro/settings.json`. The repo itself becomes the prompt — no re-pasting rules, no doc-code drift.

Full case study, workflow diagrams, and learnings: **[`docs/KIRO_USAGE.md`](docs/KIRO_USAGE.md)**.

---

## 🤝 Contributing

We welcome contributions of all sizes — from typo fixes to new features.

- 🏷️ Check out issues labeled [`good-first-issue`](https://github.com/ZeptoDB/ZeptoDB/labels/good-first-issue) for easy starting points
- 📖 Read [CONTRIBUTING.md](CONTRIBUTING.md) for build instructions and guidelines
- 💬 Join [Discord](https://discord.gg/zeptodb) to discuss ideas before starting large changes

---

<div align="center">

If ZeptoDB is useful to you, consider giving it a ⭐ — it helps others discover the project.

</div>

---

## 📄 License

[Business Source License 1.1 (BUSL-1.1)](LICENSE) — You can use, modify, and distribute ZeptoDB freely, including in production. The only restriction: you cannot offer it as a commercial database-as-a-service (DBaaS).

- **Change Date:** 2030-04-01 — on this date, the license automatically converts to Apache 2.0
- **SPDX:** `BUSL-1.1`

For commercial licensing inquiries: skswlsaks@gmail.com
