Metadata-Version: 2.4
Name: agentflow-runtime
Version: 2.0.0
Summary: Real-time data platform serving context to AI agents
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: argon2-cffi<26,>=23
Requires-Dist: bcrypt<6,>=5
Requires-Dist: confluent-kafka<3,>=2.5
Requires-Dist: dagster<2,>=1.13.1
Requires-Dist: duckdb<2,>=1.1
Requires-Dist: fastapi<1,>=0.111
Requires-Dist: httpx<1,>=0.27
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<2,>=1.41
Requires-Dist: opentelemetry-instrumentation-fastapi<1,>=0.62b0
Requires-Dist: opentelemetry-instrumentation-httpx<1,>=0.62b0
Requires-Dist: opentelemetry-sdk<2,>=1.41
Requires-Dist: pandera<1,>=0.20
Requires-Dist: prometheus-client<1,>=0.21
Requires-Dist: pyarrow<25,>=17
Requires-Dist: pydantic-settings<3,>=2.5
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: pyyaml<7,>=6
Requires-Dist: sqlglot<31,>=30
Requires-Dist: structlog<27,>=24.4
Requires-Dist: uvicorn[standard]<1,>=0.30
Provides-Extra: cloud
Requires-Dist: boto3<2,>=1.35; extra == 'cloud'
Requires-Dist: pyiceberg[pyiceberg-core]<1,>=0.7; extra == 'cloud'
Provides-Extra: contract
Requires-Dist: schemathesis==4.21.10; extra == 'contract'
Provides-Extra: dev
Requires-Dist: bandit<2,>=1.9; extra == 'dev'
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: hatchling<2,>=1.25; extra == 'dev'
Requires-Dist: hypothesis<7,>=6; extra == 'dev'
Requires-Dist: jsonschema<5,>=4; extra == 'dev'
Requires-Dist: mypy<3,>=1.11; extra == 'dev'
Requires-Dist: pandas<4,>=2.2; extra == 'dev'
Requires-Dist: pytest-asyncio<2,>=0.24; extra == 'dev'
Requires-Dist: pytest-cov<8,>=5; extra == 'dev'
Requires-Dist: pytest<10,>=8.3; extra == 'dev'
Requires-Dist: ruff<1,>=0.6; extra == 'dev'
Requires-Dist: testcontainers[kafka]<5,>=4.9; extra == 'dev'
Requires-Dist: types-pyyaml<7,>=6; extra == 'dev'
Requires-Dist: types-redis<6,>=4.6; extra == 'dev'
Provides-Extra: flink
Requires-Dist: apache-flink==2.3.0; extra == 'flink'
Provides-Extra: integrations
Requires-Dist: langchain-core<2,>=1.2.22; extra == 'integrations'
Requires-Dist: langchain-text-splitters<2,>=1.1.2; extra == 'integrations'
Requires-Dist: langchain<2,>=0.3.30; extra == 'integrations'
Requires-Dist: langgraph<2,>=1; extra == 'integrations'
Requires-Dist: langsmith<1,>=0.7.31; extra == 'integrations'
Requires-Dist: llama-index-core<1,>=0.12; extra == 'integrations'
Provides-Extra: load
Requires-Dist: fakeredis<3,>=2.21; extra == 'load'
Requires-Dist: locust<3,>=2.29; extra == 'load'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]<4,>=3.2; extra == 'postgres'
Description-Content-Type: text/markdown

# AgentFlow

> Event-native metrics layer: business metrics that move when events happen — measured **1.1 s p50** event-to-metric freshness on production defaults. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery for people, dashboards, services, and AI agents alike.

[![Release gate](https://img.shields.io/badge/release_gate-v1.6_published-brightgreen)](docs/dv2-multi-branch/RELEASE_STATUS.md)
[![codecov](https://codecov.io/gh/brownjuly2003-code/agentflow/branch/main/graph/badge.svg)](https://codecov.io/gh/brownjuly2003-code/agentflow)
[![Python](https://img.shields.io/badge/python-3.11+-blue)](pyproject.toml)
[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

## Why this exists

BI on a replica answers yesterday's questions. Support, ops, and merch workflows need *current* orders, metrics, and health signals at the moment of decision — not a stale warehouse snapshot, not a pile of one-off service adapters, and not a cache that quietly serves 30-second-old numbers.

AgentFlow's axis is **event → live metric**: every metric declares which events move it (a contract-tested lineage graph), and the serving layer keeps reads fresh by invalidating its cache when events arrive — a measured behavior, not a slogan ([docs/freshness-benchmark.md](docs/freshness-benchmark.md)). One serving boundary on top of that axis:

- streaming ingestion for operational events (validated, enriched, journaled)
- a semantic layer that exposes entities, metrics, lineage, and query endpoints
- typed, versioned contracts — each metric ships with its source events and a staleness budget
- Python and TypeScript clients that speak the same API surface

Consumers are whoever needs the number now: humans, dashboards, downstream services, and AI agents — agents are one consumer, not the product.

## Highlights

- **Measured event-to-metric freshness** — an event entering the pipeline is reflected in `GET /v1/metrics/*` in **1.06 s p50 / 1.99 s p95** on production defaults (event-driven cache invalidation, no webhook registration), tunable to **238 ms p50**; a plain TTL cache on the same pipeline sits at ~15 s. Reproducible via `python scripts/benchmark_freshness.py` → [freshness benchmark](docs/freshness-benchmark.md)
- **Lineage as a contract** — all six metrics declare their source events, serving table, and a 2.5 s p95 staleness budget in versioned contracts, exposed through `/v1/catalog` and `/v1/contracts` and pinned by tests against the actual write path
- **Published release line through `v1.6.0`** on PyPI (`agentflow-runtime`, `agentflow-client`) and npm (`@yuliaedomskikh/agentflow-client`) via OIDC Trusted Publishers with SLSA provenance on every artifact
- **Tested and gated** — 1,500+ unit tests plus a broad Windows no-Docker suite; CI enforces 13 required status checks (lint, schema, unit, integration, helm, perf, terraform, bandit, safety, npm-audit, trivy, contract, build-smoke) through branch protection
- **Dual SDK parity** across Python and TypeScript — retries, circuit breakers, batching, pagination, contract pinning, idempotency keys, `as_of` historical reads — over sub-second entity lookups (p50 `38–55 ms`, p99 `167 ms` on local hardware)
- **Security in the hot path** — tenant isolation on every read surface, parameterized queries, `sqlglot` AST validation for NL-to-SQL, fail-closed auth, secret scrubbing, and a Bandit gate for new findings
- **Production-shaped extras** — two CDC paths (hardened Debezium/Kafka Connect + a ClickHouse per-branch fan-out), on-call [runbooks](docs/runbooks/README.md), and a [narrated demo](docs/dv2-multi-branch/) of the DV2 multi-branch warehouse

## Quick start

> **Upgrading from v1.0.x?** See the [v1.1 migration guide](docs/migration/v1.1.md) before installing.

Prerequisites:

- Python `3.11+`
- `make`
- Docker Compose (`make demo` starts Redis and the ClickHouse serving store)

PowerShell 7+:

```powershell
git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
. .\scripts\setup.ps1
make demo
```

macOS / Linux:

```bash
git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
source ./scripts/setup.sh
make demo
```

`make demo` starts Redis and ClickHouse, seeds demo data through the full pipeline (validated events land in the ClickHouse serving store), and serves the API on `http://localhost:8000`. Swagger UI is available at `http://localhost:8000/docs`.

Try it:

```bash
curl http://localhost:8000/v1/entity/order/ORD-20260404-1001

curl -X POST http://localhost:8000/v1/query \
  -H "Content-Type: application/json" \
  -d '{"question":"Show me top 3 products"}'
```

Local demo runs without API-key enforcement unless you explicitly configure `AGENTFLOW_API_KEYS_FILE`.

## Architecture

```text
Event sources -> Kafka -> Flink -> Iceberg --------\
                                                    -> Semantic layer -> FastAPI -> Agent / SDK
Local demo   -> local_pipeline -> ClickHouse ------/
                       (DuckDB stays the local lake / test store)
```

Stack:

- **Ingestion**: Kafka producers, Debezium/Kafka Connect CDC, and a local synthetic pipeline
- **Processing**: Flink plus validation and enrichment stages
- **Storage**: Iceberg for production-shaped tables; **ClickHouse is the serving store** (ADR 0006 — ReplacingMergeTree upserts, `final=1` reads), DuckDB the local-dev / test store
- **Serving**: FastAPI, contract registry, lineage, search, and operational endpoints
- **Orchestration**: Dagster
- **IaC**: Terraform, Helm, Docker Compose, and a Fly.io demo config

See [docs/architecture.md](docs/architecture.md) for the detailed design, trade-offs, and deployment topologies.

CDC source capture is standardized on Debezium/Kafka Connect; downstream consumers use the canonical AgentFlow CDC contract defined in [ADR 0005](docs/decisions/0005-cdc-ingestion-strategy.md).

## What's inside

| Area | Files |
|------|-------|
| API core | `src/serving/api/` |
| Semantic layer | `src/serving/semantic_layer/` |
| Python SDK | `sdk/agentflow/` |
| TypeScript SDK | `sdk-ts/src/` |
| Agent integrations | `integrations/agentflow_integrations/` (LangChain, LlamaIndex, CrewAI, MCP) |
| Flink jobs | `src/processing/flink_jobs/` |
| Test suites | `tests/` |
| Design decisions | `docs/decisions/` (ADRs) |
| Public site | `site/` |
| IaC | `infrastructure/terraform/`, `infrastructure/dv2/`, `helm/`, `k8s/` |
| DV2.0 warehouse | `warehouse/agentflow/dv2/` (hubs / links / satellites + real-dataset loader) |

## Documentation

**Core**
- [Architecture](docs/architecture.md) — system context, data flow, failure modes
- [API Reference](docs/api-reference.md) — endpoint-by-endpoint curl / Python / TypeScript examples
- [Operational Runbook](docs/runbook.md) + [On-Call Runbooks](docs/runbooks/README.md) — local stack, CDC capture, and production-incident playbooks
- [Security Audit](docs/security-audit.md) — threat model, controls, and evidence
- [Glossary](docs/glossary.md) — interview-ready explanations of the core technical terms
- [Interactive Technical Walkthrough](docs/index.md) — MkDocs Material guide (Mermaid architecture, SDK, deployment, observability)

**Deep dives**
- [DV2.0 Multi-Branch Extension](docs/dv2-multi-branch/architecture.md) — Data Vault 2.0 model for mid-market e-com (5 locations / 3 jurisdictions): [schema](docs/dv2-multi-branch/schema_dv2.md), [end-to-end flow](docs/dv2-multi-branch/architecture.md), [demo evidence](docs/dv2-multi-branch/demo_evidence.md)
- [CDC Deployment Plan](docs/plans/2026-04-debezium-kafka-connect-deployment-plan.md) — Debezium/Kafka Connect rollout
- [Competitive Analysis](docs/competitive-analysis.md) · [Release Readiness](docs/release-readiness.md) · [Cost Analysis](docs/cost-analysis.md)
- [Fly.io Demo Deploy](deploy/fly/README.md) — minimal hosted demo
- [Contributing](CONTRIBUTING.md) · [Changelog](CHANGELOG.md)

## Development

```bash
# verified release slice
python -m pytest tests/unit tests/integration tests/sdk -q

# benchmark and regression gate
python scripts/run_benchmark.py
python scripts/check_performance.py --baseline docs/benchmark-baseline.json --current .artifacts/load/results.json --max-regress 20

# benchmark trend: [.github/perf-history.json](.github/perf-history.json) is appended on every main push;
# render the history locally with `make perf-plot` (writes docs/perf/history.html).

# contracts and security
python scripts/generate_contracts.py --check
bandit -r src sdk --ini .bandit --severity-level medium -f json -o .tmp/bandit-current.json
python scripts/bandit_diff.py .bandit-baseline.json .tmp/bandit-current.json
```

## Status

**`v1.6.0` is the current release line** — PyPI `agentflow-runtime` /
`agentflow-client` and npm `@yuliaedomskikh/agentflow-client`, all
published via OIDC Trusted Publishers with SLSA provenance attestations.
CI on `main` is green across all 13 required checks.

The `v1.1.0` → `v1.6.0` arc landed in six increments on top of a security
audit-closure sprint:

- **`v1.1.0`** — audit closure: tenant isolation across every read
  surface, SQL guard centralized on `sqlglot`, entity allowlist
  enforcement, fail-closed auth, secret rotation, Helm hardening,
  OpenAPI drift gate, and the required status checks.
- **`v1.2.0`** — DV2 multi-branch warehouse: 38 Data Vault 2.0 tables
  (8 hubs / 8 links / 22+ satellites), an Argo Workflows `dv2-refresh`
  template, a dbt project (3 mart models + 12 tests), and per-branch CDC
  fan-out via ClickHouse `MaterializedPostgreSQL`.
- **`v1.3.0`** — `helm/kafka-connect` hardening matched to `helm/agentflow`
  (NetworkPolicy + PDB + securityContext), live Helm validation across both
  charts, and the narrated DV2 demo (terminal + web-UI + dbt docs).
- **`v1.4.0`** — maintenance: on-call runbooks, `SECURITY.md`, issue/PR
  templates, contract/DORA CI hardening, repo hygiene, and a dependency
  wave (`mypy`, Terraform AWS provider, TypeScript, GitHub Actions,
  Vitest). No runtime API changes from `v1.3.0`.
- **`v1.5.0`** — security & correctness hardening: argon2id key hashing
  with an O(1) peppered lookup index (M-C4), an NL→SQL guard bypass fix
  (typed `read_csv` / `read_parquet` scan functions now denied in
  projection position), `sqlglot` control-byte and mutation-target
  repairs, and a strict-`mypy` expansion across the orchestration and
  freshness slices. No public API changes.
- **`v1.6.0`** — the architecture-fixing release: ClickHouse becomes the
  shipped serving engine (pipeline sink, `ReplacingMergeTree` row versions,
  backend-routed event scan, a dedicated CI E2E lane against a real
  ClickHouse), PII protection moves from the removed app-level string-parse
  gate to engine-enforced vault governance (fail-closed column grants,
  per-jurisdiction officer roles, row policies, `SQL SECURITY DEFINER`
  views — every live adversarial probe green), plus the vendored NL→SQL
  generation engine (LangGraph, routed through GraceKelly), the DV2 raw
  vault on PostgreSQL with `LISTEN`/`NOTIFY` freshness, the MinIO-backed
  PyIceberg catalog, and the OpenSSF Scorecard channel (5.8 → 7.0).

The tagged line and `main` are in sync as of `v1.6.0`. See the
[changelog](CHANGELOG.md) for full detail.

### Scope

This is a reference data-engineering project. The streaming, warehouse, and
deployment artifacts (Flink, Iceberg, Helm, Terraform, k8s) are exercised
against a local pipeline and a kind cluster in CI rather than a managed
cloud. Wiring it to a live production source needs inputs that live outside
the repo — CDC source onboarding (runbook ready in
[docs/operations/cdc-production-onboarding.md](docs/operations/cdc-production-onboarding.md)),
a public benchmark on production-grade hardware, and an external pen-test
attestation.

## Screenshots

| Admin UI | API docs |
|----------|----------|
| <img src="docs/screenshots/admin-ui.png" alt="AgentFlow admin UI" width="420"> | <img src="docs/screenshots/swagger-docs.png" alt="AgentFlow API docs" width="420"> |

| Landing page | Benchmark run |
|--------------|---------------|
| <img src="docs/screenshots/landing-page.png" alt="AgentFlow landing page" width="420"> | <img src="docs/screenshots/benchmark-terminal.png" alt="AgentFlow benchmark terminal" width="420"> |

Capture notes and publish-time checks are listed in [docs/publication-checklist.md](docs/publication-checklist.md).

## License

MIT. See [LICENSE](LICENSE).

## Credits

Built as a data-engineering reference project. Initial release cycle
`2026-04-10` → `2026-04-20`, with post-audit hardening and the DV2
extension landing through `v1.4.0`. Architecture decisions are recorded as
ADRs in [docs/decisions/](docs/decisions/).
