Metadata-Version: 2.4
Name: civec
Version: 0.1.0
Summary: Generic vector infrastructure components
Requires-Python: >=3.10
Requires-Dist: pydantic
Provides-Extra: clients
Requires-Dist: httpx; extra == 'clients'
Provides-Extra: detections
Requires-Dist: psycopg[binary]>=3.2.10; extra == 'detections'
Requires-Dist: sqlalchemy>=2.0.45; extra == 'detections'
Provides-Extra: dev
Requires-Dist: pyright>=1.1.403; extra == 'dev'
Requires-Dist: pytest>=8.4.2; extra == 'dev'
Provides-Extra: fastapi
Requires-Dist: fastapi; extra == 'fastapi'
Provides-Extra: gateway
Requires-Dist: pydantic-settings; extra == 'gateway'
Provides-Extra: gateway-ui
Requires-Dist: streamlit<1.55,>=1.54; extra == 'gateway-ui'
Provides-Extra: ingestion
Requires-Dist: numpy; extra == 'ingestion'
Provides-Extra: lancedb
Requires-Dist: lancedb; extra == 'lancedb'
Requires-Dist: pyarrow; extra == 'lancedb'
Provides-Extra: query
Requires-Dist: numpy; extra == 'query'
Requires-Dist: pandas; extra == 'query'
Requires-Dist: scikit-learn; extra == 'query'
Provides-Extra: rabbitmq
Requires-Dist: pika>=1.3.2; extra == 'rabbitmq'
Provides-Extra: rules-geometry
Requires-Dist: shapely>=2.1.2; extra == 'rules-geometry'
Provides-Extra: tenants-sql
Requires-Dist: psycopg[binary]>=3.2.10; extra == 'tenants-sql'
Requires-Dist: sqlalchemy>=2.0.45; extra == 'tenants-sql'
Description-Content-Type: text/markdown

# Civec

Package-ready vector infrastructure for building vector database pipelines.

The base install contains typed records, namespace policies, and protocol seams. Optional extras add storage, query, transport, HTTP, and geometry integrations without forcing those dependencies into the minimal install.

## Install

```bash
uv add civec
uv add "civec[query,lancedb,rules-geometry]"
```

## Develop

```bash
uv sync --all-extras
uv run python -m pytest
uv run pyright
```

## Core Concepts

- `VectorRecord`: generic stored vector with `id`, `namespace_id`, `vector`, `source_timestamp`, `ingested_at`, and metadata.
- `NamespaceRecord`: generic tenant-like namespace model with shard, index, retention, and metadata policies.
- `RecordTransformer`: converts raw app messages into normalized `VectorRecord` objects.
- `QueryVectorProvider`: converts text, image, or custom query payloads into query vectors.
- `ShardRouter`: maps records and queries to storage shards by namespace.
- `GenericQueryService`: searches LanceDB-like shards and returns JSON-safe result rows.
- `civec.extensions.reranking`: exact reranking helpers for retriever flows.
- `civec.extensions.detection_queues`: generic top-K review queue scoring and rule helpers.
- `civec.extensions.gateway`: generic namespace routing and message mutation service.
- `civec.clients`: typed HTTP client helpers for service adapters.
- `IngestionEngine`: transforms, partitions, batches, and writes records through injected app adapters.
- `rules`: reusable field access, condition models, and route evaluation used by gateways and review queues.

## Ingestion

```python
from dataclasses import dataclass
from datetime import datetime, timezone

from civec.ingestion import IngestionEngine, IngestionEngineConfig
from civec.models import VectorRecord


@dataclass(frozen=True)
class Event:
    event_id: str
    namespace_id: str
    vector: list[float]


class EventTransformer:
    def parse_records(self, message: Event, now: datetime) -> list[VectorRecord]:
        return [
            VectorRecord(
                id=message.event_id,
                namespace_id=message.namespace_id,
                vector=message.vector,
                source_timestamp=None,
                ingested_at=now,
                metadata={},
            )
        ]
```

Provide a partitioner and writer for your app. The library handles transformation and batching; your adapter owns transport acknowledgements, environment variables, and concrete storage wiring.

## Query And Reranking

`GenericQueryService` expects a shard registry and LanceDB-like connection. Configure column names for your schema:

```python
from civec.query import GenericQueryService, QueryColumnConfig
from civec.extensions.reranking import attach_cosine_similarity

service = GenericQueryService(
    db_conn,
    registry,
    logger,
    columns=QueryColumnConfig(
        namespace_column="namespace_id",
        timestamp_column="ingested_at",
        vector_column="vector",
    ),
)
```

App layers can keep legacy names like `tenant_id` or `ts_ingested` by setting `QueryColumnConfig` instead of changing external APIs.

## Rules

The `civec.rules` package evaluates generic JSON documents using typed field definitions, condition configs, and route definitions. It is suitable for routing gateways and rule-filtered review queues. Install `rules-geometry` when using WKT or GeoJSON intersection operators.

## Dependency Extras

- `clients`: HTTP client helper dependencies.
- `lancedb`: LanceDB and PyArrow helpers.
- `query`: NumPy/Pandas query helpers.
- `rabbitmq`: RabbitMQ transport adapter dependencies.
- `fastapi`: FastAPI adapter dependencies.
- `tenants-sql`: SQL tenant/shard registry dependencies.
- `rules-geometry`: Shapely-backed geometry rule operators.
- `detections`: Detection queue storage dependencies.
- `gateway`: Gateway support dependencies.
- `gateway-ui`: Streamlit UI dependencies.
- `dev`: test and type-check tooling.
