Metadata-Version: 2.4
Name: zleap-sag
Version: 0.7.0
Summary: Local-first memory & knowledge engine for AI agents
Project-URL: Homepage, https://github.com/Zleap-AI/zleap
Project-URL: Repository, https://github.com/Zleap-AI/zleap
Project-URL: Documentation, https://github.com/Zleap-AI/zleap/tree/main/packages/sag#readme
Project-URL: Changelog, https://github.com/Zleap-AI/zleap/blob/main/packages/sag/CHANGELOG.md
Project-URL: Issues, https://github.com/Zleap-AI/zleap/issues
Author-email: Zleap Team <contact@zleap.ai>
License-Expression: MIT
License-File: LICENSE
Keywords: ai-agents,knowledge-graph,llm,memory,rag,retrieval,vector-database
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: aiohttp>=3.9
Requires-Dist: aiosqlite>=0.19
Requires-Dist: json-repair>=0.58
Requires-Dist: jsonschema>=4
Requires-Dist: lancedb>=0.16
Requires-Dist: numpy>=1.26
Requires-Dist: openai>=1.6
Requires-Dist: pydantic-settings>=2.1
Requires-Dist: pydantic>=2.5
Requires-Dist: pyyaml>=6
Requires-Dist: sqlalchemy[asyncio]>=2.0
Requires-Dist: tiktoken>=0.5
Requires-Dist: tokenizers>=0.22
Requires-Dist: tzdata
Provides-Extra: all
Requires-Dist: aiomysql>=0.2; extra == 'all'
Requires-Dist: alembic>=1.17; extra == 'all'
Requires-Dist: asyncpg>=0.29; extra == 'all'
Requires-Dist: cryptography>=41; extra == 'all'
Requires-Dist: elasticsearch-dsl<9,>=8; extra == 'all'
Requires-Dist: elasticsearch<9,>=8; extra == 'all'
Requires-Dist: litellm>=1.40; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
Provides-Extra: dev
Requires-Dist: aiomysql>=0.2; extra == 'dev'
Requires-Dist: alembic>=1.17; extra == 'dev'
Requires-Dist: asyncpg>=0.29; extra == 'dev'
Requires-Dist: cryptography>=41; extra == 'dev'
Requires-Dist: elasticsearch-dsl<9,>=8; extra == 'dev'
Requires-Dist: elasticsearch<9,>=8; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pre-commit>=3.6; extra == 'dev'
Requires-Dist: psycopg2-binary>=2.9; extra == 'dev'
Requires-Dist: pytest-asyncio>=1; extra == 'dev'
Requires-Dist: pytest-cov>=7; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: es
Requires-Dist: elasticsearch-dsl<9,>=8; extra == 'es'
Requires-Dist: elasticsearch<9,>=8; extra == 'es'
Provides-Extra: litellm
Requires-Dist: litellm>=1.40; extra == 'litellm'
Provides-Extra: migrations
Requires-Dist: alembic>=1.17; extra == 'migrations'
Provides-Extra: mysql
Requires-Dist: aiomysql>=0.2; extra == 'mysql'
Requires-Dist: cryptography>=41; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == 'postgres'
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: sqlite
Description-Content-Type: text/markdown

# zleap-sag

[![PyPI](https://img.shields.io/pypi/v/zleap-sag)](https://pypi.org/project/zleap-sag/)
[![Python](https://img.shields.io/pypi/pyversions/zleap-sag)](https://pypi.org/project/zleap-sag/)
[![License](https://img.shields.io/pypi/l/zleap-sag)](https://github.com/Zleap-AI/zleap/blob/main/LICENSE)

**Local-first memory & knowledge engine for AI agents.** Ingest documents, extract an
event/entity graph with an LLM, and retrieve over it — fully local by default, progressive
to production databases.

> Distribution `zleap-sag` · import `zleap.sag` · Python ≥ 3.11 · MIT

## Highlights

- **Zero-infra by default** — embedded SQLite + LanceDB (built-in BM25), no services to run. Data lives in `./.zleap/`.
- **Progressive to production** — swap in MySQL / PostgreSQL / OceanBase (relational) and Elasticsearch / pgvector (vector) by changing config only; pipeline code is unchanged.
- **Single-database option** — run the whole engine on one PostgreSQL (pgvector) or one OceanBase instance (SQL + vector).
- **Tiny API** — `DataEngine` orchestrates ingest → extract → search; injected `EngineConfig`; typed results; one `SagError` base.
- **Multiple retrieval strategies** — `vector`, `atomic`, and multi-hop (`multi`) over the event-entity graph.

## Quick Start

### 1. Install

```bash
pip install zleap-sag        # runs as-is: embedded SQLite + LanceDB, no services needed
```

Add an extra only when you switch that backend on (quote the brackets in shells):

```bash
pip install "zleap-sag[es]"        # Elasticsearch            (vector_provider="es")
pip install "zleap-sag[mysql]"     # MySQL or OceanBase       (aiomysql driver)
pip install "zleap-sag[postgres]"  # PostgreSQL, incl. pgvector single-DB
pip install "zleap-sag[all]"       # all backends + Alembic migrations + litellm
```

### 2. Run

Point it at any OpenAI-compatible LLM + embedding endpoint. Storage defaults to `./.zleap/`
(SQLite + LanceDB); the schema is created automatically on first run.

```python
import asyncio
from zleap.sag import DataEngine, EngineConfig
from zleap.sag.config import LLMConfig, EmbeddingConfig


async def main():
    config = EngineConfig(
        llm=LLMConfig(api_key="sk-...", base_url="https://your-gateway/v1", model="qwen3.6-flash"),
        embedding=EmbeddingConfig(model="bge-large-en-v1.5"),
    )
    async with DataEngine(config) as engine:      # start(): create local tables on first run
        await engine.ingest("your_document.md")   # parse → chunk → embed
        await engine.extract()                     # LLM → event/entity graph
        result = await engine.search("Who founded Acme?", strategy="multi", top_k=5)
        for section in result.sections:
            print(section.get("content", "")[:200])


asyncio.run(main())
```

Storage lands in `./.zleap/` — add it to your `.gitignore`. Runnable scripts:
[`examples/`](https://github.com/Zleap-AI/zleap/tree/main/packages/sag/examples).

## Configuration

### Two ways to configure — pick one

zleap-sag always builds a single `EngineConfig`. Provide it by **parameter injection** *or*
**environment variables** — they are alternatives, not layered.

| Aspect | Parameter injection | Environment variables |
| --- | --- | --- |
| Call | `EngineConfig(llm=LLMConfig(...), ...)` | `EngineConfig.from_env()` |
| Values come from | explicit Python arguments | `OPENAI_API_KEY`, `LLM_MODEL`, … (or a `.env` file) |
| Pass keys in code? | **yes** — every value | **no** — read from the environment |
| Best for | notebooks, embedding in an app | containers / 12-factor deployments |

> **Does setting env vars remove the need to pass keys?** Only if you call
> `EngineConfig.from_env()`. Plain `EngineConfig(...)` **never reads the environment** — you
> pass keys there explicitly. Don't mix the two: use `from_env()` *or* inject params.

Environment-variable path, minimal set (zero-infra):

```bash
export OPENAI_API_KEY=sk-...
export OPENAI_BASE_URL=https://your-gateway/v1     # optional; defaults to OpenAI
export LLM_MODEL=qwen3.6-flash
export EMBEDDING_MODEL=bge-large-en-v1.5
```

```python
config = EngineConfig.from_env()          # or EngineConfig.from_env(env_file=".env")
```

### Required vs optional

- **Required:** `llm` and `embedding` — each an OpenAI-compatible endpoint.
- **Optional (all default to the local stack):** storage backend, `data_dir` (`./.zleap`), `rerank`, `language`, `log_level`.

### Two gotchas

- **Separate embedding endpoint?** Set `EmbeddingConfig(base_url=..., api_key=...)` (or
  `EMBEDDING_BASE_URL` / `EMBEDDING_API_KEY`). If omitted, embedding **reuses the LLM's** key and URL.
- **Don't** set `EmbeddingConfig(dimensions=...)` unless your model supports it — many
  embedding models reject a dimension override.

Full variable reference (storage, backends, rerank):
[`.env.example`](https://github.com/Zleap-AI/zleap/blob/main/.env.example).

## Storage backends

Switch backends by changing `EngineConfig` only — the ingest/extract/search code is
identical. Spin up local backends for testing with `make up` (docker compose).

| Deployment | Relational | Vector | Extra | Schema init |
| --- | --- | --- | --- | --- |
| **Local (default)** | SQLite | LanceDB | — | automatic on `start()` |
| Production | MySQL / PostgreSQL / OceanBase | Elasticsearch | `[mysql]` / `[postgres]` / `[es]` | `init_schema()` once |
| Single DB | PostgreSQL | pgvector | `[postgres]` | `init_schema()` once |
| Single DB | OceanBase ≥ 4.3.3 | oceanbase | `[mysql]` | `init_schema()` once |

### Initialization

- **Local SQLite** — `start()` creates the schema and seeds default entity types
  automatically. Nothing to call.
- **Production backends** — call `await engine.init_schema()` once (idempotent: adds missing
  tables, never drops). For versioned migrations use the `[migrations]` extra (Alembic).

### Example — MySQL + Elasticsearch

```python
from zleap.sag import DataEngine
from zleap.sag.config import EngineConfig, RelationalConfig, ESConfig, LLMConfig, EmbeddingConfig

config = EngineConfig(
    relational=RelationalConfig(
        provider="mysql", host="localhost", user="sag2", password="sag2", database="sag2"
    ),
    vector_provider="es",
    es=ESConfig(hosts=["http://localhost:9200"]),
    llm=LLMConfig(api_key="sk-...", base_url="https://your-gateway/v1", model="qwen3.6-flash"),
    embedding=EmbeddingConfig(model="bge-large-en-v1.5"),
)

async with DataEngine(config) as engine:
    await engine.init_schema()          # once, for production backends
    await engine.ingest("your_document.md")
    await engine.extract()
    result = await engine.search("Who founded Acme?", strategy="multi", top_k=5)
```

### Example — single database

```python
# One PostgreSQL for relational + vector (pip install "zleap-sag[postgres]")
config = EngineConfig(
    relational=RelationalConfig(provider="postgres", host="localhost", port=5432,
                                user="sag2", password="sag2", database="sag2"),
    vector_provider="pgvector",
    llm=LLMConfig(...), embedding=EmbeddingConfig(...),
)

# One OceanBase for SQL + vector (pip install "zleap-sag[mysql]", OceanBase ≥ 4.3.3)
config = EngineConfig(
    relational=RelationalConfig(provider="oceanbase", host="localhost", port=2881,
                                user="root", password="", database="sag2"),
    vector_provider="oceanbase",
    llm=LLMConfig(...), embedding=EmbeddingConfig(...),
)
```

> OceanBase ANN indexes need the tenant setting `ob_vector_memory_limit_percentage > 0`; if
> unset, the engine falls back to exact vector search automatically.

## How it works

One `DataEngine` instance binds one data source. The three steps chain automatically — the
`source_config_id` returned by `ingest()` is reused by `extract()` and `search()`:

- **`ingest(path)`** — parse a document → chunk → embed → persist chunks and vectors. Takes a **file path** (single document).
- **`extract()`** — an LLM extracts an event/entity graph from the last ingest's chunks.
- **`search(query, strategy=, top_k=)`** — retrieve. Strategies: `vector`, `atomic`, `multi` (default), `multi1`, `hopllm`, `multi_es`.

`DataEngine` is an async context manager: `async with DataEngine(config) as engine` runs
`start()` on enter and `aclose()` on exit. Result types:
`from zleap.sag.results import IngestResult, ExtractResult, SearchResult, ChunkResult`.
To chunk a raw string without persisting (no `start()` / DB needed), use `DataEngine.chunk()`.

## Notes

- **One config per process** for now — core connections are process-global; run one
  `EngineConfig` per process.
- **`multi_es`** needs a real-BM25 vector backend (LanceDB or Elasticsearch); on pgvector it
  raises `MissingCapabilityError` rather than degrading silently.
- All engine errors derive from `SagError` — catch it at the boundary.

## Links

[Examples](https://github.com/Zleap-AI/zleap/tree/main/packages/sag/examples) ·
[Changelog](https://github.com/Zleap-AI/zleap/blob/main/packages/sag/CHANGELOG.md) ·
[Contributing](https://github.com/Zleap-AI/zleap/blob/main/CONTRIBUTING.md) ·
[Config reference](https://github.com/Zleap-AI/zleap/blob/main/.env.example)
