# Requirements

> Generated from `docs/arch/requirements.yaml`. Do not hand-edit.

Each requirement is anchored — link to one directly with `#REQ-NNN` (e.g. [REQ-001](#REQ-001)). Inline `REQ-NNN` mentions deep-link to their section, and the docs search box indexes every requirement by id, category, and description.

## 1. Access Governance & Security

### REQ-001 · Query Governance {#REQ-001}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Any authenticated identity can query using any supported language (GraphQL, SQL, pgwire, Arrow Flight). Data returned is governed solely by table/column visibility, RLS, and masking — there is no capability gate on querying itself. Relationship join enforcement applies unless the role holds the ignore_relationships capability. There is no query registry and no pre-approval concept.

**Use case:** Removing the query-execution capability gate means governance is expressed entirely through data-layer controls (visibility, RLS, masking), not through access lists.

**Code:** `provisa/security/`, `provisa/compiler/`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_governance.py`, `tests/unit/test_registry_removed.py`, `provisa-ui/e2e/governance-core.spec.ts`

### REQ-002 · Query Governance {#REQ-002}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Rights and Stage 2 governance enforcement is platform-level — applied to every query at compile time. No client path can bypass it without bypassing the server ([REQ-266](#REQ-266)).

**Use case:** Platform-level rights enforcement prevents any client path from bypassing governance controls.

**Code:** `provisa/security/`, `provisa/compiler/`

**Tests:** `tests/unit/test_governance.py`, `tests/integration/test_governance_integration.py`

### REQ-003 · Query Governance {#REQ-003}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

All queries and mutations are governed by user rights alone — table/view rights plus relationship rights. No registry membership or query approval is required for any operation.

**Use case:** Uniform rights-based governance lets users run any query their rights permit without approval overhead.

**Code:** `provisa/core/`

**Tests:** `tests/unit/test_governance.py`, `tests/integration/test_governance_integration.py`

### REQ-004 · Query Governance {#REQ-004}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Test endpoint accepts arbitrary queries against registered schema with full guards; MUST NOT be exposed in production.

**Use case:** Isolated test endpoint lets developers validate queries safely without risking production exposure.

**Code:** `provisa/core/`

**Tests:** `tests/unit/test_governance.py`, `tests/unit/test_test_endpoint_guard.py`

### REQ-005 · Query Governance {#REQ-005}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Result-size ceilings are defined per role/table in config (`max_rows`); Stage 2 injects LIMIT when a query would exceed the role's ceiling for any referenced table. Clients may always narrow further (fewer columns, additional filters).

**Use case:** Per-role/table row ceilings bound result size without an approved-query registry.

**Code:** `provisa/core/`

**Tests:** `tests/unit/test_governance.py`, `tests/integration/test_governance_integration.py`

### REQ-006 · Query Governance {#REQ-006}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Large-result redirect and Arrow output are available to any query the user's rights permit, subject to configured thresholds ([REQ-029](#REQ-029), [REQ-137](#REQ-137)).

**Use case:** Large-result and Arrow output available to any rights-permitted query, governed by configured thresholds.

**Code:** `provisa/core/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/unit/test_governance.py`, `tests/unit/test_redirect_s3.py`

### REQ-038 · Security {#REQ-038}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Two independent enforcement layers: schema visibility and SQL enforcement (Stage 2). Both derive from the user's table/view and relationship rights.

**Use case:** Independent visibility and SQL-enforcement layers ensure governance cannot be bypassed by compromising one layer.

**Code:** `provisa/security/`

**Tests:** `tests/e2e/test_security.py`, `tests/integration/test_governance_integration.py`, `tests/unit/test_masking.py`, `tests/unit/test_rls.py`, `tests/unit/test_visibility.py`

### REQ-039 · Security {#REQ-039}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Schema visibility layer: unauthorized tables/columns do not appear in SDL or query builder; compiler rejects at parse time.

**Use case:** Schema visibility enforcement ensures unauthorized tables and columns are invisible to clients at design time.

**Code:** `provisa/security/`

**Tests:** `tests/e2e/test_security.py`, `tests/integration/test_governance_integration.py`, `tests/unit/test_masking.py`, `tests/unit/test_rls.py`, `tests/unit/test_visibility.py`, `provisa-ui/e2e/governance-core.spec.ts`

### REQ-040 · Security {#REQ-040}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

SQL enforcement layer: executor injects RLS WHERE clauses and strips unauthorized columns before execution, every request.

**Use case:** SQL-level RLS and column stripping ensure governance is enforced at execution time, not just at schema time.

**Code:** `provisa/security/`

**Tests:** `tests/e2e/test_security.py`, `tests/integration/test_governance_integration.py`, `tests/integration/test_rls_execution.py`, `tests/unit/test_inherited_roles.py`, `tests/unit/test_layer_contracts.py`, `tests/unit/test_masking.py`, `tests/unit/test_rls.py`, `tests/unit/test_sampling_rls.py`, `tests/unit/test_visibility.py`

### REQ-041 · Security {#REQ-041}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

RLS rules defined at table registration as PG-style SQL filter expressions mapped to user roles.

**Use case:** Declarative RLS rules let stewards define row-level access without writing database triggers or views.

**Code:** `provisa/security/`

**Tests:** `tests/e2e/test_security.py`, `tests/integration/test_governance_integration.py`, `tests/integration/test_rls_execution.py`, `tests/unit/test_layer_contracts.py`, `tests/unit/test_masking.py`, `tests/unit/test_rls.py`, `tests/unit/test_visibility.py`

### REQ-042 · Security {#REQ-042}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Source registration, table registration, relationship definition, security configuration, query development, query authorization, and ignore-relationships rights are distinct and independently configured.

**Use case:** Fine-grained, independently assignable rights prevent privilege creep across registration and execution roles.

**Code:** `provisa/security/`

**Tests:** `tests/e2e/test_security.py`, `tests/integration/test_governance_integration.py`, `tests/unit/test_governance.py`, `tests/unit/test_masking.py`, `tests/unit/test_rls.py`, `tests/unit/test_visibility.py`

### REQ-203 · ABAC Approval Hook {#REQ-203}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Pluggable operation approval hook for enterprises with complex ABAC that can't be expressed as static RLS rules. Evaluated at query time. Request: user_id, roles, session_vars, tables, columns, operation. Response: approved/denied + optional additional filter. Position: after RLS injection, before execution.

**Use case:** ABAC approval hook lets enterprises enforce attribute-based policies too complex for static RLS rules.

**Code:** `provisa/security/`

**Tests:** `tests/integration/test_abac_hook_endpoint.py`, `tests/integration/test_governance_integration.py`, `tests/unit/test_abac_hook.py`, `tests/unit/test_abac_hook_extended.py`, `tests/unit/test_approval_hook.py`

### REQ-204 · ABAC Approval Hook {#REQ-204}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Approval hook scoping — per-table (`approval_hook: true`), per-source (`approval_hook: true` on source config), or global (`auth.approval_hook`). Compiler checks at query time: if no table in the query has approval_hook enabled (directly, via source, or global), skip the call entirely. Zero overhead for unscoped tables.

**Use case:** Scoped hook evaluation skips the approval call entirely for unscoped tables, keeping overhead at zero.

**Code:** `provisa/security/`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_abac_hook.py`, `tests/unit/test_abac_hook_extended.py`, `tests/unit/test_approval_hook.py`

### REQ-246 · ABAC Approval Hook {#REQ-246}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Approval hook protocols — three transport options, configured via `auth.approval_hook.type` (Part of Phase AE): - `webhook` (default): HTTP POST to URL. ~5-50ms. Simplest, works with any external ABAC service. - `grpc`: gRPC with persistent connection + multiplexing. ~1-5ms. Binary protocol for high-volume same-datacenter deployments. Proto service definition shipped with Provisa. - `unix_socket`: Unix domain socket for same-machine sidecars (e.g., OPA). <0.5ms. Path configured via `auth.approval_hook.socket_path`.

**Use case:** Three approval hook transports (HTTP/gRPC/Unix socket) let enterprises match latency requirements to ABAC infra.

**Code:** `provisa/security/`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_abac_hook.py`, `tests/unit/test_abac_hook_extended.py`, `tests/unit/test_approval_hook.py`

### REQ-247 · ABAC Approval Hook {#REQ-247}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Approval hook config (Part of Phase AE): auth.approval_hook.type: webhook | grpc | unix_socket; url for webhook/grpc; socket_path for unix_socket; timeout_ms: 500; fallback: deny | allow on timeout; scope: all or omit for per-table/per-source scoping. Per-table: tables[].approval_hook: true. Per-source: sources[].approval_hook: true. Global: auth.approval_hook.scope: all.

**Use case:** Granular approval hook scoping (global/source/table) avoids adding governance overhead to unscoped tables.

**Code:** `provisa/security/`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_abac_hook.py`, `tests/unit/test_abac_hook_extended.py`, `tests/unit/test_approval_hook.py`

### REQ-262 · Two-Stage Compiler (Governed SQL) {#REQ-262}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Compiler refactored into two explicit stages. Stage 1: GraphQL → plain PG-style SQL (semantic compilation only — resolves aliases, domains, relationships, emits physical column names and table names with AS aliases). Stage 2: plain SQL → governed SQL (AST rewrite applying RLS, masking, visibility, and the row cap). Stage 2 is a standalone transformer that accepts SQL + role and returns governed SQL, with no knowledge of GraphQL or the semantic model.

**Use case:** Two-stage compiler separates semantic translation from governance so Stage 2 can be reused by any SQL client.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_graphql_alias.py`, `tests/unit/test_stage2.py`, `tests/unit/test_two_stage_compiler.py`

### REQ-263 · Two-Stage Compiler (Governed SQL) {#REQ-263}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Stage 2 governance transformer applies four governance concerns: (1) RLS — inject WHERE predicate per table reference per role (SQLGlot AST rewrite); (2) column masking — wrap masked column expressions with masking function (AST rewrite); (3) column visibility — remove or NULL-out columns invisible to the role (AST rewrite); (4) row cap — inject/cap LIMIT to the most restrictive per-role/table `max_rows` ceiling for any referenced table. A role without the FULL_RESULTS capability receives the configured default cap. The row cap is the single mechanism for bounding result size; statistical sampling is a user query feature, not a governance concern ([REQ-478](#REQ-478)).

**Use case:** Stage 2 governance transformer enforces RLS, masking, visibility, and the row cap uniformly on every SQL path and transport.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_mask_inject.py`, `tests/unit/test_stage2.py`, `tests/unit/test_stage2_complex.py`, `tests/unit/test_two_stage_compiler.py`

### REQ-264 · Two-Stage Compiler (Governed SQL) {#REQ-264}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Stage 2 must handle all SQL structural patterns: subqueries, CTEs, JOINs, `SELECT *` (expand via schema introspection then apply visibility), UNION, and nested expressions. RLS and masking must be injected at every table reference in the full AST, not only the outermost SELECT.

**Use case:** Stage 2 handling of CTEs and subqueries ensures governance is applied at every table reference, not just outermost.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/e2e/test_sql_endpoint.py`, `tests/integration/test_governance_integration.py`, `tests/unit/test_mask_inject.py`, `tests/unit/test_sql_endpoint_governance.py`, `tests/unit/test_stage2.py`, `tests/unit/test_two_stage_compiler.py`

### REQ-265 · Two-Stage Compiler (Governed SQL) {#REQ-265}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Stage 2 operates on physical column names (left side of AS alias pairs). Stage 1 guarantees all aliases are explicit in emitted SQL, so Stage 2 never needs to resolve semantic model names.

**Use case:** Stage 2 operating on physical column names makes it independent of the semantic model and alias resolution.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_stage2.py`, `tests/unit/test_two_stage_compiler.py`, `tests/integration/test_governance_integration.py`

### REQ-266 · Two-Stage Compiler (Governed SQL) {#REQ-266}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

With Stage 2 as a standalone SQL transformer, any SQL may enter the pipeline from any client — GraphQL, raw SQL via `/data/sql`, JDBC, DB-API, SQLAlchemy — and governance is uniformly enforced. No client can bypass governance without bypassing the server.

**Use case:** Stage 2 as standalone SQL transformer ensures no client can bypass governance without bypassing the server.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_sql_endpoint_governance.py`, `tests/unit/test_stage2.py`, `tests/unit/test_two_stage_compiler.py`, `provisa-ui/e2e/governance-core.spec.ts`

### REQ-267 · Two-Stage Compiler (Governed SQL) {#REQ-267}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A `/data/sql` REST endpoint accepts raw PG-compatible SQL + role (via auth), passes it through Stage 2 governance, routes and executes identically to the GraphQL path. Subject to the user's rights: any table or view the user's rights permit may be queried freely; Stage 2 governance is applied uniformly.

**Use case:** /data/sql endpoint lets raw SQL tools submit queries that still pass through full governance enforcement.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_governance_integration.py`, `tests/unit/test_sql_endpoint_governance.py`, `tests/unit/test_stage2.py`, `tests/unit/test_two_stage_compiler.py`, `provisa-ui/e2e/governance-core.spec.ts`

### REQ-369 · Rate Limiting {#REQ-369}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Per-role rate limits configurable in `provisa.yaml`: max requests per second, max concurrent SSE subscriptions, max concurrent Arrow Flight streams. Limits enforced at the API layer before compilation or execution. Requests exceeding limits rejected with HTTP 429 and a `Retry-After` header.

**Use case:** Per-role rate limits prevent any single role from exhausting server resources or upstream connection pools.

**Code:** `provisa/api/`

**Tests:** `tests/unit/test_rate_limiting.py`, `tests/integration/test_rate_limiting_integration.py`, `provisa-ui/e2e/security-rate-limiting.spec.ts`

### REQ-370 · Rate Limiting {#REQ-370}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The NL query service (`POST /query/nl`) has an independent rate limit configurable via `nl.rate_limit` (requests per minute per role). LLM API cost is the primary driver — unbounded NL requests are a cost risk regardless of server capacity. Requests exceeding the limit are rejected before any LLM call is made.

**Use case:** NL rate limit caps LLM API cost exposure without restricting other query paths.

**Code:** `provisa/nl/`

**Tests:** `tests/unit/test_rate_limiting.py`, `tests/integration/test_rate_limiting_integration.py`, `provisa-ui/e2e/security-rate-limiting.spec.ts`

### REQ-371 · Rate Limiting {#REQ-371}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Rate limit state stored in Redis (existing `cache.redis_url`) using a sliding window counter. No rate limit state is held in-process — stateless Provisa containers ([REQ-057](#REQ-057)) share limit state across all instances.

**Use case:** Redis-backed rate limit state ensures limits are enforced correctly across all horizontal Provisa instances.

**Code:** `provisa/api/`

**Tests:** `tests/integration/test_cache_store.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_rate_limiting.py`

### REQ-402 · Security {#REQ-402}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

RLS rules can be scoped to a domain instead of a single table by setting `domain_id` (nullable FK to domains) instead of `table_id` (now also nullable); a single domain-level rule applies to all tables in that domain unless overridden by a table-specific rule.

**Use case:** Domain-level RLS rules reduce configuration burden when the same row filters apply across many tables in a semantic domain.

**Code:** `provisa/core/schema.sql`, `provisa/core/repositories/rls.py`, `provisa/compiler/`

**Tests:** `tests/e2e/test_security.py`, `tests/integration/test_governance_integration.py`, `tests/unit/test_rls.py`

### REQ-531 · Security {#REQ-531}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Masked columns are rejected from WHERE and HAVING clauses at query parse time, before execution, preventing inference of the unmasked value through binary search filtering. Enforced via V005 validation in the SQL validator (provisa/compiler/sql_validator.py). Applies across all query interfaces (GraphQL, SQL, Cypher).

**Use case:** Predicate guard closes the filtering side-channel so callers cannot infer unmasked values by submitting successive WHERE comparisons against a masked column.

**Code:** `provisa/compiler/sql_validator.py`

**Tests:** `tests/e2e/test_security.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_stage2.py`, `tests/integration/test_security_integration_extra.py`

### REQ-554 · Security {#REQ-554}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

All roles receive sampled results capped at DEFAULT_SAMPLE_SIZE (default: 100 rows) unless the role holds the `full_results` capability. The cap is controlled by the PROVISA_SAMPLE_SIZE environment variable and enforced via the Stage 2 row cap mechanism.

**Use case:** Default row cap prevents accidental large result delivery to unprivileged roles without requiring per-query LIMIT clauses.

**Code:** `provisa/compiler/sampling.py`, `provisa/compiler/stage2.py`

**Tests:** `tests/e2e/test_security.py`, `tests/unit/test_stage2.py`, `tests/integration/test_security_integration_extra.py`

### REQ-555 · ABAC Approval Hook {#REQ-555}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The gRPC approval hook maintains a single persistent channel per Provisa instance (one grpc.aio channel reused across all calls to that endpoint), reducing connection overhead for high-throughput co-located policy services.

**Use case:** Persistent gRPC channel eliminates per-call connection overhead for high-volume approval checks against a co-located policy service.

**Code:** `provisa/auth/approval_hook.py`

**Tests:** `tests/integration/test_audit_auth_integration.py`, `tests/unit/test_abac_hook_extended.py`, `tests/unit/test_approval_hook.py`, `provisa-ui/e2e/security-rate-limiting.spec.ts`

### REQ-556 · ABAC Approval Hook {#REQ-556}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The approval hook client implements a circuit breaker that opens after 5 consecutive failures (configurable via circuit_breaker_threshold) and enters half-open state after 30 seconds (configurable via circuit_breaker_cooldown_s), preventing cascading failures from a slow or unavailable hook endpoint.

**Use case:** Circuit breaker prevents a slow or unavailable approval hook from blocking all query execution when the fallback policy is 'allow'.

**Code:** `provisa/auth/approval_hook.py`

**Tests:** `tests/unit/test_abac_hook_extended.py`, `tests/unit/test_approval_hook.py`, `tests/integration/test_security_integration_extra.py`

### REQ-557 · Security {#REQ-557}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Data source credentials use ${env:VAR_NAME} syntax in the Provisa config YAML; values are resolved at runtime from environment variables via the secrets provider in provisa/core/secrets.py. Plaintext passwords are never stored in the config database.

**Use case:** Environment-variable secret references keep credentials out of config files and the config database, satisfying standard secrets hygiene requirements.

**Code:** `provisa/core/secrets.py`

**Tests:** `tests/e2e/test_security.py`, `tests/unit/test_secrets.py`

### REQ-591 · Security {#REQ-591}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

SET LOCAL scopes app.tenant_id to the current database transaction. When the transaction commits or rolls back, the variable resets, preventing tenant context leakage across transactions.

**Use case:** Transaction-scoped tenant context prevents cross-tenant data leakage if middleware is bypassed or connection is reused across requests.

**Code:** `provisa/core/db.py`

**Tests:** `tests/e2e/test_security.py`, `tests/unit/test_core_registration.py`, `tests/integration/test_security_integration_extra.py`

### REQ-594 · Security {#REQ-594}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

TenantMiddleware defines a skip-path set `{/billing/signup, /billing/webhook, /health, /docs, /openapi.json}`. Requests to these paths bypass tenant resolution entirely and do not require a JWT with a tenant_id claim.

**Use case:** Skip-path exemptions allow Lemon Squeezy billing webhooks, health checks, and OpenAPI docs to function without tenant JWTs while all other paths enforce tenant isolation.

**Code:** `provisa/api/middleware/tenant_middleware.py`

**Tests:** `tests/e2e/test_security.py`, `tests/unit/test_tenancy_requirements.py`, `tests/integration/test_security_integration_extra.py`, `provisa-ui/e2e/security-rate-limiting.spec.ts`

### REQ-596 · Audit Logging {#REQ-596}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Every query is recorded in `query_audit_log` with tenant_id, user_id, role_id, SHA-256 query hash, table_ids, source, status_code, duration_ms, and logged_at. The table is append-only: PostgreSQL rules block DELETE and UPDATE at the database level. Query text is never stored verbatim — only its SHA-256 hash. Two indexes support tenant-scoped and per-user time-range queries.

**Use case:** Append-only audit log with hash-only query storage satisfies data minimization requirements while providing immutable evidence of query activity for compliance and investigation.

**Code:** `provisa/audit/query_log.py`

**Tests:** `tests/unit/test_audit_requirements.py`, `tests/integration/test_audit_auth_integration.py`

### REQ-603 · Query Governance {#REQ-603}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

V002 relationship governance: every JOIN ON condition in SQL and Cypher queries must match an approved, registered relationship (source_col = target_col). SQL and Cypher queries that traverse a join not backed by a registered relationship are rejected at compile time. GraphQL queries that traverse relationships defined in the SDL are always pre-approved and exempt from the V002 check.

**Use case:** V002 enforces approved navigation paths so joins between sensitive tables require explicit steward registration and approval, providing an audit surface for cross-table data access.

**Code:** `provisa/compiler/sql_validator.py`

**Tests:** `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_stage2.py`, `tests/integration/test_security_integration_extra.py`

### REQ-613 · Query Governance {#REQ-613}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Every query that touches a domain asset is logged in an append-only audit log (query_audit_log). The log captures: user_id, role_id, query_hash, table_ids, source, status_code, duration_ms, and logged_at. The log is protected by PostgreSQL rules that prevent DELETE and UPDATE operations (SOC2 append-only requirement). Indexed by (tenant_id, logged_at) and (user_id, logged_at) for efficient compliance reporting.

**Use case:** Append-only query audit log provides SOC2-compliant evidence of who queried what data and when.

**Code:** `provisa/audit/query_log.py`, `provisa/audit/compliance_reporter.py`

**Tests:** `tests/unit/test_audit_requirements.py`, `tests/integration/test_security_integration_extra.py`

## 2. Authentication & Identity

### REQ-120 · Authentication {#REQ-120}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Pluggable auth provider interface — abstract AuthProvider producing AuthIdentity (user_id, email, roles, claims) mapped to Provisa roles. One provider at a time configured in YAML.

**Use case:** Pluggable auth provider interface lets organizations connect any identity system without code changes.

**Code:** `provisa/auth/`

**Tests:** `tests/unit/test_auth_providers.py`, `tests/unit/test_auth_middleware.py`, `tests/integration/test_auth_integration.py`

### REQ-121 · Authentication {#REQ-121}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Firebase Authentication — validates Firebase ID tokens via firebase-admin SDK. Supports all Firebase auth methods (email/password, Google, Apple, GitHub, phone, anonymous, SAML, OIDC).

**Use case:** Firebase auth integration supports all Firebase sign-in methods including SSO and anonymous users.

**Code:** `provisa/auth/`

**Tests:** `tests/unit/test_auth_providers.py`, `tests/unit/test_auth_middleware.py`, `tests/integration/test_auth_integration.py`, `provisa-ui/e2e/auth-providers.spec.ts`

### REQ-122 · Authentication {#REQ-122}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Keycloak OIDC — validates JWT access tokens from Keycloak via OIDC discovery + JWKS. Realm roles + client roles → Provisa role mapping.

**Use case:** Keycloak OIDC integration maps realm and client roles to Provisa roles for enterprise SSO deployments.

**Code:** `provisa/auth/`

**Tests:** `tests/unit/test_auth_providers.py`, `tests/unit/test_auth_middleware.py`, `tests/integration/test_auth_integration.py`, `provisa-ui/e2e/auth-providers.spec.ts`

### REQ-123 · Authentication {#REQ-123}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Generic OAuth 2.0 / OIDC — works with any OIDC-compliant provider (PingFederate, Okta, Azure AD, Auth0). OIDC discovery URL → JWKS → JWT validation. Configurable role claim mapping.

**Use case:** Generic OIDC support lets any compliant provider (Okta, Azure AD, Auth0) integrate without custom code.

**Code:** `provisa/auth/`

**Tests:** `tests/integration/test_auth_integration.py`, `tests/unit/test_auth_middleware.py`, `tests/unit/test_auth_providers.py`, `tests/unit/test_config_validation_edge_cases.py`, `provisa-ui/e2e/auth-providers.spec.ts`

### REQ-124 · Authentication {#REQ-124}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Simple username/password auth for testing — users defined in config YAML with bcrypt hashed passwords. Issues short-lived JWT. NOT for production (requires allow_simple_auth: true flag).

**Use case:** Simple username/password auth lets developers test the platform locally without configuring an IdP.

**Code:** `provisa/auth/`

**Tests:** `tests/unit/test_auth_providers.py`, `tests/unit/test_auth_middleware.py`, `tests/integration/test_auth_integration.py`

### REQ-125 · Authentication {#REQ-125}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Superuser bootstrap access — superuser credentials in config (username + password from env secret). Always admin role + all capabilities regardless of auth provider. For initial setup.

**Use case:** Superuser bootstrap ensures admins can always access the platform for initial setup regardless of auth provider.

**Code:** `provisa/auth/`

**Tests:** `tests/unit/test_auth_providers.py`, `tests/unit/test_auth_middleware.py`, `tests/integration/test_auth_integration.py`

### REQ-535 · Authentication {#REQ-535}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When no auth provider is configured (dev mode), any request is treated as a role identity — the username IS the role, taken from the `x-provisa-role` header or defaulting to `admin`. This identity maps to all configured roles with wildcard domain access, enabling unrestricted local development without configuring an IdP.

**Use case:** Dev mode role identity lets developers test without an IdP while still exercising role-based code paths.

**Code:** `provisa/auth/middleware.py`

**Tests:** `tests/unit/test_auth_middleware.py`, `tests/integration/test_audit_auth_integration.py`

### REQ-551 · Authentication {#REQ-551}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The `assignments_source` config field controls how role assignments are resolved: `claims` reads role assignments directly from JWT token claims; `provisa` reads role assignments from Provisa's own internal assignment store. Defaults to `claims`.

**Use case:** Two assignment resolution modes let organizations choose between IdP-managed roles (claims) and Provisa-managed roles (provisa store) without changing auth provider.

**Code:** `provisa/core/models.py`, `provisa/auth/`

**Tests:** `tests/unit/test_auth_providers.py`, `tests/integration/test_auth_integration.py`

## 3. Source Registration & Data Modeling

### REQ-012 · Registration & Governance {#REQ-012}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Source registration is privileged; validates connection, calls Trino dynamic catalog API, no restart required, available within seconds.

**Use case:** Privileged, live source registration lets stewards connect new data sources in seconds without downtime.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_introspect.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_source_registration_validation.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-013 · Registration & Governance {#REQ-013}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Source registration does not expose data — no table queryable until explicitly registered by a steward.

**Use case:** Data is invisible until explicitly registered, ensuring no accidental exposure of raw database tables.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-014 · Registration & Governance {#REQ-014}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Unregistered tables do not exist — cannot be referenced in queries, do not appear in schema browser, cannot be mutation targets.

**Use case:** Unregistered tables are completely absent from schema and runtime, eliminating enumeration attacks.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-015 · Registration & Governance {#REQ-015}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

There is no per-table governance mode. Every table and view is queryable directly under the user's rights (table/view rights + relationship rights) with Stage 2 governance applied uniformly. No registry-required mode exists.

**Use case:** Uniform rights-based access removes per-table governance modes; every table follows the same model.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-016 · Registration & Governance {#REQ-016}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Table publication triggers schema generation pass; table immediately available in query builder.

**Use case:** Immediate schema availability after table publication removes friction for query developers.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-017 · Registration & Governance {#REQ-017}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

NoSQL/non-relational sources are exposed read-only through their native Trino connector (e.g. the MongoDB connector), driven by the type-specific mapping DSL ([REQ-251](#REQ-251)); no mutations. (Revised 2026-06-18: the original "automatic Parquet materialization" wording is superseded by the connector approach, which is the implementation.)

**Use case:** Native Trino connectors make NoSQL sources queryable without manual ETL pipelines.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-018 · Registration & Governance {#REQ-018}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Trino FK metadata used to infer candidate intra-source relationships for steward confirmation/rejection.

**Use case:** FK-inferred relationship suggestions reduce manual steward work when registering related tables.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-019 · Registration & Governance {#REQ-019}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cross-source relationships defined manually by steward with cardinality (many-to-one, one-to-many). (Revised 2026-06-18: one-to-one removed — the relationship-field model is a strict binary, single object vs list, so a 1:1 collapses to many-to-one; model a true 1:1 as a many-to-one in each direction.)

**Use case:** Manual cross-source relationship definitions let stewards federate data across different databases with full control.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-020 · Registration & Governance {#REQ-020}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Relationships owned by defining steward, versioned, flagged for re-review on schema changes affecting join fields.

**Use case:** Versioned, owner-tracked relationships flag stale joins when schema changes, preventing silent data errors.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_introspect.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_relationship_metadata.py`

### REQ-021 · Registration & Governance {#REQ-021}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

GraphQL schema reflects registration model (business intent), not raw database structure.

**Use case:** Business-intent schema means the GraphQL API reflects concepts consumers understand, not raw table structures.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-119 · JSONB & API Sources {#REQ-119}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Stewards can promote specific nested fields from JSONB columns into native PostgreSQL generated columns (GENERATED ALWAYS AS ... STORED). Promoted columns are filterable, indexable, relationship-eligible, and auto-maintained by PostgreSQL. Supports dot-path extraction for nested fields.

**Use case:** JSONB field promotion to generated columns makes nested document fields indexable and relationship-eligible.

**Code:** `provisa/api_source/`

**Tests:** `tests/unit/test_api_promotions.py`, `tests/integration/test_schema_compiler_integration.py`

### REQ-133 · Views (Governed Computed Datasets) {#REQ-133}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Views are SQL-defined computed datasets registered in the Provisa config with full column-level governance (visibility, masking, descriptions, aliases).

**Use case:** SQL-defined governed views let stewards expose computed datasets with full column-level governance applied.

**Code:** `provisa/compiler/`, `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/e2e/test_mv_optimization.py`

### REQ-134 · Views (Governed Computed Datasets) {#REQ-134}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Views go through the same governance pipeline as tables — RLS, masking, sampling, role-based schema visibility, approval workflow.

**Use case:** Views through the same governance pipeline as tables ensures computed datasets receive identical security treatment.

**Code:** `provisa/compiler/`, `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/e2e/test_mv_optimization.py`, `tests/integration/test_schema_compiler_integration.py`

### REQ-135 · Views (Governed Computed Datasets) {#REQ-135}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Views with `materialize: true` are backed by a periodically refreshed MV (CTAS). Views without materialization run as live subqueries via Trino.

**Use case:** Materialized views enable pre-computed result caching, while live views support real-time freshness needs.

**Code:** `provisa/compiler/`, `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/e2e/test_mv_optimization.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-136 · Views (Governed Computed Datasets) {#REQ-136}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Views are the governed mechanism for adding computed semantics (aggregations, transformations) to the platform. This preserves the GraphQL constraint that no new semantics can be added outside the platform.

**Use case:** Views as the governed mechanism for computed semantics ensures all aggregations and transforms are auditable.

**Code:** `provisa/compiler/`, `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/e2e/test_mv_optimization.py`, `tests/integration/test_schema_compiler_integration.py`

### REQ-151 · Column Path Extraction {#REQ-151}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Columns with `path` extract values from JSON source columns using PG `>>` syntax. SQLGlot transpiles to `json_extract_scalar` for Trino.

**Use case:** Column path extraction from JSON columns makes nested document fields queryable as first-class columns.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_sql_gen.py`, `tests/integration/test_schema_compiler_integration.py`

### REQ-152 · Column Path Extraction {#REQ-152}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Path columns on PostgreSQL sources route direct. Non-PG sources force Trino routing.

**Use case:** Path column routing rules ensure JSON extraction uses the correct engine for each source type.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_sql_gen.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-153 · Column Path Extraction {#REQ-153}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Path columns are read-only computed fields — mutations unaffected.

**Use case:** Path columns are read-only, preventing mutations from targeting computed derived fields.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_sql_gen.py`

### REQ-154 · Naming & Schema {#REQ-154}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Optional `domain_prefix` prepends `domain_id__` (double underscore) to all GraphQL names.

**Use case:** Domain prefix with double underscore lets stewards namespace all GraphQL names to avoid type conflicts.

**Code:** `provisa/compiler/`

**Tests:** `tests/integration/test_schema_compiler_integration.py`, `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-155 · Naming & Schema {#REQ-155}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Table and column `alias` fields override GraphQL names.

**Use case:** Column and table aliases let stewards expose business-friendly names without altering underlying schema.

**Code:** `provisa/compiler/`

**Tests:** `tests/integration/test_schema_compiler_integration.py`, `tests/unit/test_graphql_alias.py`, `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-156 · Naming & Schema {#REQ-156}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Table and column `description` fields included in GraphQL SDL.

**Use case:** Description fields in SDL give query developers in-schema documentation without leaving the GraphQL tooling.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-157 · Naming & Schema {#REQ-157}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Order-by enum values preserve original column case (not uppercased).

**Use case:** Preserved original column case in order-by enums prevents case mismatch errors in consumer queries.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-158 · Auto-Materialized Relationships {#REQ-158}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cross-source relationships with `materialize: true` auto-generate MV definitions at startup.

**Use case:** Automatic MV generation for materialized cross-source relationships eliminates manual ETL setup for joins.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/unit/test_mv_refresh.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-159 · Auto-Materialized Relationships {#REQ-159}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

Only cross-source relationships generate MVs. Same-source relationships are already fast via direct routing.

**Use case:** MVs only for cross-source relationships avoids unnecessary materialization overhead for same- source fast joins.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/unit/test_mv_refresh.py`

### REQ-160 · Auto-Materialized Relationships {#REQ-160}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Auto-MVs start STALE and are populated by the background refresh loop.

**Use case:** Auto-MVs starting STALE ensures the refresh loop populates them before they serve live traffic.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_mv_registry.py`, `tests/unit/test_mv_refresh.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-194 · Naming Convention {#REQ-194}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The naming authority is the single source of truth for all client-facing names. Physical backend column names are never exposed to clients. **Alias** (`column.alias` in config): the canonical name. SQL uses it verbatim (no convention applied). GQL applies its configured convention to it. CQL derives from GQL. **No alias**: each query language derives its name from the physical column name via its configured convention. **SQL convention**: configurable, default `snake_case`. Applied via `apply_sql_name()`. **GQL convention**: configurable (`apollo_graphql` default = camelCase, `hasura_graphql` = snake_case). Applied via `apply_gql_name()`. **CQL**: derived from the GQL name.

**Use case:** Consistent multi-language naming with a single authority ensures client-facing names are predictable and independent of physical schema.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_graphql_alias.py`, `tests/unit/test_naming.py`

### REQ-195 · Naming Convention {#REQ-195}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Aligns with Hasura v2 `naming_convention` setting: `hasura-default` maps to `snake_case`, `graphql-default` maps to `camelCase`. DDN `namingConvention: graphql` maps to `camelCase`.

**Use case:** Hasura v2 parity — naming convention alignment lets teams familiar with Hasura conventions feel at home.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_naming.py`

### REQ-250 · Registration & Governance {#REQ-250}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

All Trino catalog configuration must flow through Provisa's config YAML — users never manage Trino .properties files directly. Provisa generates catalog properties, table definition files, and auto-registers tables at runtime following the Kafka source config pattern as exemplar.

**Use case:** Config-driven Trino catalog generation hides Trino properties files from operators entirely.

**Code:** `provisa/core/`

**Tests:** `tests/e2e/test_kafka_flow.py`, `tests/integration/test_elasticsearch_introspect.py`, `tests/integration/test_introspect.py`, `tests/integration/test_nosql_catalogs.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_prometheus_source.py`, `tests/unit/test_redis_source.py`, `tests/unit/test_trino_catalog_files.py`

### REQ-251 · Registration & Governance {#REQ-251}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Each NoSQL/non-relational source type requires a type-specific mapping DSL in the Provisa config to define how its native data structures map to relational tables. Examples: Redis key patterns → rows, hash fields → columns; Elasticsearch index patterns with nested field paths; Prometheus metric names with label-to-column mapping.

**Use case:** Type-specific mapping DSL lets stewards define how Redis keys, Elasticsearch fields, and Prometheus metrics become SQL columns.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_introspect.py`, `tests/integration/test_nosql_catalogs.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_table_search.py`

### REQ-363 · Semantic Layer / Semantic Model {#REQ-363}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The SQLAlchemy dialect introspects table and column metadata via `POST /data/graphql` (GraphQL introspection query) instead of any admin or unfiltered endpoint. This ensures the dialect's `get_table_names()` and `get_columns()` responses are role-governed: the GraphQL server applies the Semantic Layer filter before returning the introspection result, so callers see only tables and columns their role is permitted to access.

**Use case:** SQLAlchemy dialect introspects via governed GraphQL endpoint; role determines table and column visibility.

**Code:** `provisa/compiler/`, `provisa/security/`

**Tests:** `tests/unit/test_visibility.py`, `tests/unit/test_enforcement_metadata.py`, `tests/integration/test_registration_governance_integration.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-366 · Registration & Governance {#REQ-366}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Views require an approval workflow, OR the originator must already hold the rights to the underlying tables and to any joins used within the view. Any join within a view likewise requires approval or originator rights. Convenience views (adding no new semantics) are discouraged — instead grant the relationship rights and query in any form. Creating a view implies new semantics: derived/calculated values, or the view name itself as a new business concept. Approval gates therefore apply to both views (for the semantics they introduce, consistent with [REQ-134](#REQ-134)) and relationships (for navigational intent).

**Use case:** View approval (or originator-rights) plus join approval ensures new semantics and cross-domain imports are governed; convenience views are discouraged in favor of relationship rights.

**Code:** `provisa/core/`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_core_registration.py`

### REQ-367 · Domain Model {#REQ-367}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

There is only one view type — always intradomain (the output belongs to a domain). Two valid reasons to create a view: (1) the source is cross-domain (importing external data into the domain as a named business concept), (2) the source is local (deriving new or calculated data from existing domain assets). Two hard constraints follow: cross-domain data may only enter a domain via a view, and new or derived data may only exist as a view. No other reason to create a view is valid; there is no separate cross-domain view type — just views, always intradomain, used for exactly these two purposes.

**Use case:** Unified view model with two hard constraints ensures cross-domain imports are read-only adapters while computed data is always created within a single domain.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/mv/`

**Tests:** `tests/unit/test_domain_views.py`

### REQ-392 · Domain Model {#REQ-392}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Schema endpoint returns node_labels with a `pk: string | null` field per label, designating the primary key column name. Allows graph UI to construct reliable `WHERE NOT n.<pk> IN [<value>]` exclusion clauses (instead of heuristic-based `id(n)` fallback), enabling "Exclude from query", deep-linking to nodes, "show related to this entity", and inspector PK highlighting.

**Use case:** Explicit PK designation in schema enables semantic graph operations and node-centric navigation without node ID heuristics.

**Code:** `provisa/api/rest/cypher_router.py`, `provisa/cypher/label_map.py`

**Tests:** `tests/unit/test_core_registration.py`, `tests/unit/test_graph_schema.py`, `tests/unit/test_graph_schema_pk.py`, `tests/unit/test_relationship_alias.py`, `tests/unit/test_schema_pk_requirements.py`

### REQ-393 · Domain Model {#REQ-393}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The semantic layer must support user-designated primary key columns per table column via an `is_primary_key: bool` field. Designation is informational only — Provisa does not enforce uniqueness; the underlying datastore is authoritative.

**Use case:** User-designated PKs enable semantic graph operations without requiring Provisa to validate database constraints.

**Code:** `provisa/core/`, `provisa/compiler/`

**Tests:** `tests/unit/test_core_registration.py`, `tests/unit/test_domain_table_uniqueness.py`

### REQ-394 · Domain Model {#REQ-394}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Multiple PK checkboxes on a table infer a composite key; the first designated PK column is used as the canonical `id_column` for Cypher node identity resolution, taking priority over all heuristics.

**Use case:** First PK column as canonical id_column ensures consistent node identity across Cypher graph operations.

**Code:** `provisa/cypher/`, `provisa/compiler/`

**Tests:** `tests/unit/test_core_registration.py`

### REQ-399 · Domain Model {#REQ-399}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

When a Relationship is saved, the source_column on the source table is marked as `is_foreign_key=true` in table_columns. Two new boolean columns `is_foreign_key` and `is_alternate_key` are added to table_columns via `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`.

**Use case:** Foreign key and alternate key tracking on columns enables semantic relationship metadata without external constraints.

**Code:** `provisa/core/schema.sql`, `provisa/core/repositories/`

**Tests:** `tests/unit/test_core_registration.py`

### REQ-400 · Domain Model {#REQ-400}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When a Relationship is saved, the target_column on the target table is marked as `is_primary_key=true` if no other column in that table already has `is_primary_key=true`; otherwise it is marked as `is_alternate_key=true`.

**Use case:** Automatic PK/AK designation on relationship target columns reduces steward manual key configuration work.

**Code:** `provisa/core/repositories/relationship.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_core_registration.py`

### REQ-413 · Registration & Governance {#REQ-413}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Auto-generate GQL relationships from FK constraints in database schema introspection — relationships discoverable from FK metadata in addition to manual steward configuration and AI-assisted hints.

**Use case:** Auto-generated FK relationships reduce manual relationship registration overhead while preserving steward control.

**Code:** `provisa/compiler/introspect.py`

**Tests:** `tests/unit/test_discovery_requirements.py`, `tests/integration/test_registration_governance_integration.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-414 · Registration & Governance {#REQ-414}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Demo/install example schema must include at least one FK relationship to exercise auto-generated relationship discovery.

**Use case:** Demo schema coverage of FK auto-discovery ensures feature is validated during onboarding.

**Code:** `demo/files/create_demo_files.py`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_core_registration.py`, `tests/integration/test_registration_governance_integration.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-415 · Registration & Governance {#REQ-415}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

The `hasura_v2_relationship_style` option controls whether FK-derived relationships use Hasura V2's naming conventions — singular for many-to-one, plural for one-to-many using inflection.

**Use case:** V2-compatible relationship naming enables teams to adopt Hasura conventions without schema migration.

**Code:** `provisa/compiler/introspect.py`

**Tests:** `tests/unit/test_discovery_requirements.py`, `tests/unit/test_fk_relationship_style.py`

### REQ-417 · Registration & Governance {#REQ-417}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Hasura v2 migration tool (`provisa/hasura_v2/mapper.py`) maps Hasura Remote Schemas to Provisa `graphql_remote` source registrations instead of skipping them with "NOT SUPPORTED" warning. Provisa already has GraphQL Remote source capability (`graphql_remote_adapter.py` and `graphql_remote_router.py`); the mapper was a gap. Migration preserves Remote Schema name, URL, headers, and authentication configuration in the registered source.

**Use case:** Hasura Remote Schema migration removes GTM blocker — customers see supported sources instead of "NOT SUPPORTED" during migration path evaluation.

**Code:** `provisa/hasura_v2/mapper.py`, `provisa/source_adapters/graphql_remote_adapter.py`, `provisa/api/admin/graphql_remote_router.py`

**Tests:** `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_hasura_remote_schema.py`, `tests/unit/test_hasura_v2_comprehensive.py`, `tests/unit/test_import_shared.py`

### REQ-418 · Domain Model {#REQ-418}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Report authoring workflow: analysts pull cross-domain data into their own domain via views (data-import adapters), then define calculations and relationships only within that domain. New or derived data exists only as views; no direct cross-domain calculations or relationships.

**Use case:** Unified view model ensures domain logic stays local and views serve exactly two purposes: importing external data and creating derived data within a single domain.

**Code:** `provisa/compiler/`, `provisa/mv/`

**Tests:** `tests/unit/test_domain_views.py`

### REQ-432 · Registration & Governance {#REQ-432}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A registered table must be unique by (domain_id, table_name) — stronger guarantee than the DB constraint UNIQUE(source_id, schema_name, table_name). Table registration (admin registerTable/updateTable mutations) rejects any collision of domain+table against a different physical table. Startup/schema-rebuild validates the entire registry and fails if any duplicate exists.

**Use case:** Domain+table uniqueness prevents ambiguous table references across all query layers (GraphQL, Cypher, view SQL).

**Code:** `provisa/core/`

**Tests:** `tests/unit/test_core_registration.py`, `tests/unit/test_schema_visibility_filters.py`

### REQ-433 · Registration & Governance {#REQ-433}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

A datasource may be associated with multiple domains. Any domain owner may register any unclaimed table from that source. Once a table is claimed by one domain, no other domain may register it — first-come ownership model. Unique constraint enforced on (source_id, normalized_table_name). The UI greys out claimed tables regardless of which domain claimed them.

**Use case:** Multi-domain with first-claim ownership prevents the same physical table from being registered multiple times while allowing flexible domain-to-source associations.

**Code:** `provisa/core/`, `provisa-ui/src/pages/TablesPage`

**Tests:** `tests/unit/test_dataset_uniqueness.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-434 · Registration & Governance {#REQ-434}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Creation-request mechanism: any governed create operation (view, relationship, etc.) attempted by a user lacking the authority to perform it produces a persisted request rather than an error. The request enters a queue ([REQ-063](#REQ-063)) visible to every user holding the rights to execute that create. An authorized user may execute or reject the request; rejection carries a specific, actionable reason. No create is performed until an authorized user executes the request.

**Use case:** Creation-request queue lets users without create authority propose views/relationships for an authorized holder to execute.

**Code:** `provisa/core/`, `provisa-ui/src/`

**Tests:** `tests/unit/test_core_registration.py`, `tests/unit/test_creation_requests.py`

### REQ-542 · Naming & Schema {#REQ-542}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Regex naming rules in config are applied to table names in order when generating GraphQL field names, before uniqueness resolution.

**Use case:** Ordered naming rules let stewards strip common prefixes (e.g., prod_pg_) from auto-generated GraphQL names without manual aliasing.

**Code:** `provisa/compiler/naming.py`

**Tests:** `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-543 · Auto-Materialized Relationships {#REQ-543}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Mutations to source tables of a materialized cross-source relationship mark the corresponding MV as stale for re-refresh. The `refresh_interval` defaults to 300 seconds (5 minutes) when not explicitly set.

**Use case:** Mutation-triggered staleness ensures the MV is refreshed promptly after writes, keeping cached join results consistent.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_mv_refresh.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-604 · Naming & Schema {#REQ-604}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Relationships support an optional `graphql_alias` field that names the SDL field exposed on the parent type. When `graphql_alias` is absent, the SDL field name is derived from the target table's `field_name` and the relationship cardinality via `rel_field_name(target.field_name, cardinality)` (singular for many-to-one, plural for one-to-many).

**Use case:** graphql_alias lets stewards expose relationship traversal fields under business-friendly names independent of the target table's raw field_name.

**Code:** `provisa/compiler/schema_gen.py`

**Tests:** `tests/unit/test_graphql_alias.py`, `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-605 · Registration & Governance {#REQ-605}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When `root_table_ids` is set on a `SchemaInput`, tables whose IDs are absent from that set are excluded from root query fields in the generated SDL but remain present as GraphQL named types reachable via relationship fields. Domain access filtering applies the same mechanism: tables in inaccessible domains become type-defs only. Full removal of a type (including its definition) requires deleting the table registration entirely.

**Use case:** Type-def-only tables let stewards expose navigable object graphs where some types are reachable only via relationship traversal, not by independent client query, without requiring separate registration.

**Code:** `provisa/compiler/schema_gen.py`

**Tests:** `tests/unit/test_visibility.py`

### REQ-609 · Domain Model {#REQ-609}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Every domain must have a designated steward before it can serve governed data. A domain may exist in a pending state until a steward is assigned; a domain without a steward cannot expose tables, views, or relationships to consumers.

**Use case:** Steward requirement ensures accountability is always assigned before governed data is exposed.

**Code:** `provisa/core/domain_policy.py`

**Tests:** `tests/unit/test_domain_policy.py`

### REQ-610 · Domain Model {#REQ-610}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A field access grant belongs to the requesting domain, not to the specific view that prompted it. Any subsequent view in the requesting domain may use the granted fields without additional cross-domain approval. New fields not covered by the existing grant require a new request.

**Use case:** Domain-scoped field grants prevent repeated approval ceremonies for the same cross-domain data access pattern.

**Code:** `provisa/core/`

**Tests:** `tests/unit/test_core_registration.py`

### REQ-611 · Registration & Governance {#REQ-611}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Discovery is structured across five tiers of increasing governance: (1) Registered source schema — raw inventory, admin-level visibility; (2) Unclaimed tables — introspected from registered sources with no domain owner; (3) Domain assets — claimed tables and steward-defined views, fully governed; (4) Relationships — approved traversal paths between Tier 3 assets; (5) Field grants — domain-to-domain field access permissions. Each tier is a prerequisite for the next.

**Use case:** Structured discovery tiers give stewards and analysts a clear mental model of what data is governed and at what level.

**Code:** `provisa/core/`, `provisa/discovery/`

**Tests:** `tests/unit/test_discovery_requirements.py`

### REQ-612 · Registration & Governance {#REQ-612}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Relationship candidates are ranked by a four-level confidence hierarchy: (Highest) Approved catalog relationship validated by both stewards; (High) Intra-source FK constraint — explicit modeling intent; (Medium) Intra-source semantic inference — column name/type similarity; (Low) Cross-source semantic inference — naming conventions diverge, high false-positive risk. Candidates corroborated by multiple evidence types accumulate confidence.

**Use case:** Confidence hierarchy lets stewards prioritize which relationship suggestions to review first, reducing false-positive noise.

**Code:** `provisa/discovery/analyzer.py`, `provisa/discovery/collector.py`

**Tests:** `tests/unit/test_discovery_requirements.py`

### REQ-635 · Registration & Governance {#REQ-635}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The schema name presented to users must be the name the data source itself uses to group datasets. For relational databases this is the native schema (or database for MySQL). For flat/API sources with no native grouping concept, a fixed constant naming the source type is used.

**Use case:** Schema names derived from native source grouping prevent user confusion and keep the semantic layer consistent with the underlying data source.

**Code:** `provisa/api/admin/introspect.py`

**Tests:** `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-636 · Registration & Governance {#REQ-636}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

When a Trino connector is configured for a source type (i.e., the type is in SOURCE_TO_CONNECTOR), Trino is the preferred path for schema and table introspection. Native driver introspection is only used for source types with no Trino connector or those that override via native_schemas/native_tables returning a non-None value.

**Use case:** Trino-first introspection ensures schema discovery uses the same path as query execution, keeping introspection results consistent with what is queryable.

**Code:** `provisa/api/admin/introspect.py`, `provisa/core/models.py`

**Tests:** `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_native_introspect.py`, `tests/unit/test_compiler_introspect.py`

### REQ-637 · Registration & Governance {#REQ-637}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

PG cache schemas, Trino catalog names, and other implementation-layer namespaces must never appear as schema or table names presented to the user. Only native source schema/table names and registered aliases are exposed.

**Use case:** Hiding implementation namespaces prevents users from seeing or depending on internal infrastructure details that may change.

**Code:** `provisa/api/admin/introspect.py`

**Tests:** `tests/integration/test_introspect.py`, `tests/unit/test_catalog_cache.py`, `tests/unit/test_core_registration.py`

### REQ-638 · Registration & Governance {#REQ-638}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The UI calls one availableSchemas endpoint and one availableTables endpoint. Backend routing selects the correct introspection strategy per source type internally, with no source-type-specific endpoints exposed to the UI.

**Use case:** Single introspection endpoints for schemas and tables simplify the UI and centralize routing logic in one place.

**Code:** `provisa/api/admin/schema.py`, `provisa/api/admin/introspect.py`

**Tests:** `tests/integration/test_introspect.py`, `tests/unit/test_available_tables.py`, `tests/unit/test_core_registration.py`, `provisa-ui/e2e/security-rate-limiting.spec.ts`

### REQ-639 · Registration & Governance {#REQ-639}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A source type not explicitly handled by the introspection router must return [] (empty list) for both schemas and tables — never None. Returning None triggers a Trino fallback path that is not appropriate for non-Trino sources.

**Use case:** Returning empty list instead of None for unknown source types prevents broken Trino fallback execution on source types that have no Trino connector.

**Code:** `provisa/api/admin/introspect.py`

**Tests:** `tests/integration/test_introspect.py`, `tests/unit/test_core_registration.py`

### REQ-640 · Naming & Schema {#REQ-640}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Dataset name comparisons must always normalize both sides through the centralized normalization function (toSnakeCase in the UI via naming.ts; provisa.compiler.naming in the backend) before comparing. Inline ad-hoc normalization such as name.toLowerCase() or hardcoded replace(/-/g, '_') is forbidden. This applies to isRegistered checks, alias lookups, duplicate detection, and cross-source name matching.

**Use case:** Centralized normalization for all name comparisons prevents silent mismatches caused by different casing conventions across source types.

**Code:** `provisa-ui/src/naming.ts`, `provisa-ui/src/pages/TablesPage.tsx`, `provisa/compiler/naming.py`

**Tests:** `tests/unit/test_naming.py`, `tests/unit/test_schema_service.py`

### REQ-641 · Registration & Governance {#REQ-641}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Every registered dataset has two identity forms: (1) physical identity — the native name as returned by the source, used by the query engine; (2) semantic identity — domainId + tableName (alias, normalized), used by the API, GraphQL schema, and end users. Resolution between forms uses registered_tables: physical-to-semantic via (sourceId, normalizedPhysicalName); semantic-to-physical via (domainId, tableName).

**Use case:** Dual identity model with centralized resolution prevents ad-hoc string comparisons that break across naming conventions and source types.

**Code:** `provisa/core/repositories/table.py`

**Tests:** `tests/unit/test_graphql_alias.py`, `tests/unit/test_naming.py`

## 4. Source Connectors

### REQ-147 · Kafka Sources {#REQ-147}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Kafka topics queryable via Trino Kafka connector. Routed through Trino (TRINO_ONLY source).

**Use case:** Kafka topic querying via Trino lets analysts query streaming data with the same SQL and governance as relational tables.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_schema.py`, `tests/integration/test_kafka_source.py`

### REQ-148 · Kafka Sources {#REQ-148}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Default time window (`default_window`) auto-injected as WHERE clause on `_timestamp`. Prevents unbounded reads.

**Use case:** Auto-injected time-window filter prevents unbounded Kafka topic scans that would exhaust cluster resources.

**Code:** `provisa/kafka/`

**Tests:** `tests/integration/test_kafka_source.py`, `tests/unit/test_kafka_schema.py`, `tests/unit/test_kafka_window.py`

### REQ-149 · Kafka Sources {#REQ-149}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Discriminator filter for multi-type topics — multiple table configs on the same physical topic, each filtered by a discriminator field/value.

**Use case:** Discriminator filters let a single physical Kafka topic serve multiple logical table definitions safely.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_schema.py`, `tests/integration/test_kafka_source.py`

### REQ-150 · Kafka Sources {#REQ-150}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Manual schema definition for topics without Schema Registry.

**Use case:** Manual schema definition for Schema Registry-less topics lets stewards govern message structure explicitly.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_schema.py`, `tests/integration/test_kafka_source.py`

### REQ-229 · Direct-Route Dialect Expansion {#REQ-229}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

For every source type, three things must be true for direct-route capability: (1) Trino connector is packaged in the Trino deployment, (2) SQLGlot has the dialect for transpilation, (3) `SOURCE_TO_DIALECT` and `SOURCE_TO_CONNECTOR` entries exist in Provisa config. New direct-route sources to add: clickhouse, mariadb, singlestore, redshift, databricks, hive, druid, exasol. File and lake sources (iceberg, hive_s3, delta) are TRINO_ONLY — no direct-route, no SQLGlot dialect entry required; only Trino connector packaging and Provisa config entries needed. Each source must have all applicable items verified.

**Use case:** Verified direct-route support for new RDBMS dialects lets data engineers connect more source types without Trino. File and lake sources require only connector and config entries since they always route through Trino.

**Code:** `provisa/transpiler/`

**Tests:** `tests/unit/test_dialect_expansion.py`

### REQ-295 · Query-API Sources (Neo4j & SPARQL) {#REQ-295}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Neo4j registered as a non-Trino source using the existing API source pipeline. Source config accepts Neo4j HTTP API endpoint URL, target database name, and auth (none/basic/bearer token). Tables defined as steward-authored Cypher SELECT queries. The Neo4j HTTP API v2 response envelope (`data.fields` / `data.values`) is unwrapped by the built-in `neo4j_tabular` response normalizer into flat row dicts before the existing flattener runs. Column names match the aliases declared in the Cypher `RETURN` clause. Does not use Trino.

**Use case:** Neo4j registered as an API source lets stewards expose Cypher query results as governed GraphQL tables.

**Code:** `provisa/neo4j/`, `provisa/sparql/`, `provisa/api_source/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_neo4j_exec.py`, `tests/integration/test_sparql_exec.py`, `tests/unit/test_neo4j_normalizers.py`

### REQ-296 · Query-API Sources (Neo4j & SPARQL) {#REQ-296}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Neo4j table registration includes a query preview step. On submission, Provisa executes the steward's Cypher query against the live endpoint with an implicit `LIMIT 5` appended and displays the resulting rows. If the response contains node or edge objects rather than flat scalar values, registration is blocked with an error directing the steward to rewrite the query using explicit scalar projections (`RETURN n.name AS name, n.age AS age` rather than `RETURN n`). Steward owns the response shape via query authoring; Provisa does not auto-flatten graph objects.

**Use case:** Query preview at Neo4j table registration catches non-scalar projections early before they cause runtime errors.

**Code:** `provisa/neo4j/`, `provisa/sparql/`, `provisa/api_source/`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/integration/test_neo4j_exec.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_neo4j_normalizers.py`, `tests/unit/test_neo4j_preview.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-297 · Query-API Sources (Neo4j & SPARQL) {#REQ-297}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

SPARQL connector supports any SPARQL 1.1 compliant triplestore endpoint (Apache Jena Fuseki, Ontotext GraphDB, Stardog, Blazegraph, Virtuoso). Source config accepts endpoint URL, auth (none/basic/bearer), and optional default graph URI. Tables defined as SPARQL SELECT queries. The built-in `sparql_bindings` response normalizer unwraps the standard SPARQL 1.1 JSON results envelope (`results.bindings[].variable.value`) into flat row dicts before the existing flattener runs. Column names match SPARQL SELECT variable names. Does not use Trino.

**Use case:** SPARQL connector lets stewards expose any SPARQL 1.1 triplestore as governed GraphQL query results.

**Code:** `provisa/neo4j/`, `provisa/sparql/`, `provisa/api_source/`

**Tests:** `tests/unit/test_neo4j_normalizers.py`, `tests/integration/test_neo4j_exec.py`

### REQ-298 · Query-API Sources (Neo4j & SPARQL) {#REQ-298}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

API source caller extended to support HTTP POST with a request body. Required for Neo4j (POST to `/db/{database}/query/v2` with `Content-Type: application/json` body `{"statement": "<cypher>"}`) and SPARQL (POST with `Content-Type: application/x-www-form-urlencoded` body `query=<encoded-sparql>` per SPARQL 1.1 protocol). `ApiEndpoint.method` already accepts arbitrary HTTP verbs; this closes the implementation gap where POST bodies were not transmitted. Existing GET endpoints unaffected.

**Use case:** HTTP POST support for API source caller enables Neo4j and SPARQL endpoints that require POST requests.

**Code:** `provisa/neo4j/`, `provisa/sparql/`, `provisa/api_source/`

**Tests:** `tests/unit/test_neo4j_normalizers.py`, `tests/integration/test_neo4j_exec.py`

### REQ-299 · Query-API Sources (Neo4j & SPARQL) {#REQ-299}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

A `response_normalizer: str | None` field added to `ApiEndpoint`. When set, the named normalizer is applied to the raw API response before `response_root` navigation and the existing flattener. Built-in normalizers: `neo4j_tabular` (zips `data.fields[]` with each `data.values[][]` entry into a flat dict) and `sparql_bindings` (extracts `results.bindings[]` and maps each `{variable: {type, value}}` binding to `{variable: value}`). Normalizer names are validated at source registration time; unknown names are rejected. Extensible for future query-API envelope formats without changes to the core flattener.

**Use case:** Named response normalizers (neo4j_tabular, sparql_bindings) cleanly unwrap API envelopes before flattening.

**Code:** `provisa/neo4j/`, `provisa/sparql/`, `provisa/api_source/`

**Tests:** `tests/integration/test_neo4j_exec.py`, `tests/unit/test_api_flattener.py`, `tests/unit/test_api_normalizer_requirements.py`, `tests/unit/test_misc_requirements.py`, `tests/unit/test_neo4j_normalizers.py`

### REQ-307 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-307}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Provisa supports a "remote GraphQL schema" source type. The steward registers an external GraphQL endpoint URL plus optional auth (none/bearer/basic). Provisa introspects the remote schema via a standard `__schema` introspection query and stores the result.

**Use case:** Remote GraphQL schema source type lets stewards expose external GraphQL APIs as governed Provisa tables.

**Code:** `provisa/graphql_remote/`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-308 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-308}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Each Query field on the remote schema is exposed as a virtual read-only table in Provisa. Column names and types are inferred from the field's return type (object fields → columns). Each Mutation field is exposed as a tracked function with `return_schema` derived from the mutation's return type. This eliminates manual table and function registration for remote GraphQL APIs.

**Use case:** Auto-registering remote Query fields as tables and Mutation fields as functions eliminates manual registration.

**Code:** `provisa/graphql_remote/`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-309 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-309}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

At query execution time, Provisa translates the incoming GraphQL request into a remote GraphQL query and forwards it to the remote endpoint. The response rows are materialized as Parquet in a Trino Iceberg table on S3 (`results.api_cache`, `s3a://provisa-results/api_cache/`). Repeated calls within TTL are served from the Iceberg table via Trino — zero remote hops. The cache table is automatically dropped after TTL expires.

**Use case:** S3/Iceberg materialization of remote schema results eliminates repeated network hops and enables federated SQL (WHERE/ORDER BY/LIMIT) over cached data.

**Code:** `provisa/graphql_remote/`, `provisa/api_source/trino_cache.py`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_api_cache.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-310 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-310}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

RLS, column masking, domain access control, and column visibility are applied to remote schema results in Stage 2 of the governance pipeline, identically to local table results. The remote endpoint has no knowledge of Provisa's governance rules.

**Use case:** Stage 2 governance applied to remote schema results ensures external APIs never bypass RLS or masking.

**Code:** `provisa/graphql_remote/`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-311 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-311}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Remote schema introspection is re-run on demand via an "Refresh Schema" admin mutation. Provisa does not poll for remote schema changes automatically — the steward triggers refresh when the remote API evolves. On refresh, existing table and function registrations derived from the remote schema are updated; custom RLS/masking rules applied on top are preserved.

**Use case:** On-demand schema refresh lets stewards update remote schema registrations when the upstream API evolves.

**Code:** `provisa/graphql_remote/`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-312 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-312}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Remote schema type name conflicts are resolved by prefixing all generated type names with a steward-supplied namespace (e.g. `Shopify__Order` rather than `Order`). Namespace is set at source registration time and cannot be changed without re-registering.

**Use case:** Namespace prefixing prevents remote schema type name collisions with local registered table types.

**Code:** `provisa/graphql_remote/`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-313 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-313}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Relationships between remote schema virtual tables and local registered tables are supported via the standard relationship model. Provisa resolves the local side from cache/DB and the remote side via the cached remote call. Cross-source join behaviour follows the existing federation routing rules.

**Use case:** Relationships between remote and local tables follow standard federation routing so joins work transparently.

**Code:** `provisa/graphql_remote/`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_introspect.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-314 · OpenAPI Auto-Registration Connector {#REQ-314}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

An OpenAPI source is registered by providing an OpenAPI 3.x or Swagger 2.0 spec. The spec path may be a local file path (absolute or relative to the config directory) or a remote URL. Provisa fetches remote specs at registration time and caches them; local specs are read directly.

**Use case:** OpenAPI spec registration auto-discovers all GET endpoints as governed virtual tables without manual config.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`

### REQ-315 · OpenAPI Auto-Registration Connector {#REQ-315}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

If auto-discovery of a spec is not possible (behind auth, no spec endpoint, hand-written API), the steward may manually author an OpenAPI 3.x spec in the admin UI or upload a YAML/JSON file. The manually created spec is stored locally and treated identically to a fetched one.

**Use case:** Manual OpenAPI spec authoring lets stewards govern APIs that have no public spec endpoint.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`

### REQ-316 · OpenAPI Auto-Registration Connector {#REQ-316}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

On registration, Provisa parses the spec and auto-registers all GET operations as virtual query tables. Path parameters and query parameters become GraphQL arguments. The `responses.200` (or `responses.2xx`) schema determines the virtual table's column set.

**Use case:** GET operation auto-registration maps path and query params to GraphQL arguments automatically.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`

### REQ-317 · OpenAPI Auto-Registration Connector {#REQ-317}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

All non-GET operations (POST, PUT, PATCH, DELETE) are auto-registered as tracked functions (mutations). Request body schema properties become GraphQL mutation input arguments. The `responses.200`/`2xx` schema becomes the mutation's `return_schema`.

**Use case:** Non-GET operations auto-registered as mutations expose REST write endpoints as governed GraphQL mutations.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`

### REQ-318 · OpenAPI Auto-Registration Connector {#REQ-318}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

GET operation results are materialized as Parquet in a Trino Iceberg table on S3 (`results.api_cache`, `s3a://provisa-results/api_cache/`). The cache key is a SHA-256 hash of `source_id + operation path + native args`. Repeated calls within TTL hit Trino directly. The cache table is dropped after TTL expires. Mutations are never cached — they are side-effecting.

**Use case:** S3/Iceberg materialization of GET operation results prevents repeated upstream REST calls and enables Trino SQL filtering over cached rows.

**Code:** `provisa/api_source/trino_cache.py`, `provisa/api_source/router_integration.py`

**Tests:** `tests/unit/test_api_cache.py`, `tests/integration/test_openapi_source.py`

### REQ-319 · OpenAPI Auto-Registration Connector {#REQ-319}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Full governance is applied to all OpenAPI source results: RLS, column masking, domain access control, and column visibility are enforced in Stage 2 identically to local tables.

**Use case:** Stage 2 governance applied to OpenAPI results ensures REST API data never bypasses RLS or masking.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`

### REQ-320 · OpenAPI Auto-Registration Connector {#REQ-320}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Auth configuration for the upstream REST API (bearer token, API key header, basic auth, OAuth2 client credentials) is stored per-source in the secrets store and injected into outgoing requests. The caller's Provisa token is not forwarded.

**Use case:** Per-source upstream auth config keeps API credentials in the secrets store and out of client requests.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`

### REQ-321 · OpenAPI Auto-Registration Connector {#REQ-321}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Spec refresh is triggered on demand via an admin mutation. On refresh, existing virtual table and tracked function registrations derived from the spec are updated; governance rules applied on top are preserved.

**Use case:** On-demand spec refresh preserves governance rules while updating virtual table registrations from updated specs.

**Code:** `provisa/openapi/`

**Tests:** `tests/unit/test_openapi_loader.py`, `tests/integration/test_openapi_source.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-322 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-322}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Provisa supports a "grpc_remote" source type. The steward registers an external gRPC server address plus a `.proto` file path or URL. Provisa parses the proto and auto-registers virtual tables and tracked functions. The `provisa/grpc_remote/` module handles all external gRPC connectivity and is entirely distinct from `provisa/grpc/`, which serves Provisa's own gRPC server.

**Use case:** gRPC remote source type lets stewards expose external gRPC services as governed GraphQL tables and mutations.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-323 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-323}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Query vs mutation classification defaults to a name-prefix rule: methods whose names start with `Get`, `List`, `Find`, `Fetch`, `Search`, or `Stream` are classified as queries (virtual tables); all others are classified as mutations (tracked functions). Per-method override precedence is retained in the admin UI. Structural heuristic (server-streaming / repeated-message) is kept as fallback for non-prefixed methods.

**Use case:** Default name-prefix classification removes manual tagging while preserving per-method overrides and structural fallback.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_grpc_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_table_search.py`

### REQ-324 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-324}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Proto scalar types are mapped to SQL/GraphQL types as follows: `string`/`bytes` → `text`; `int32`/`uint32`/`sint32`/`fixed32`/`sfixed32` → `integer`; `int64`/`uint64`/`sint64`/`fixed64`/`sfixed64` → `bigint`; `float` → `real`; `double` → `numeric`; `bool` → `boolean`; `repeated <T>` → `jsonb`; nested message → `jsonb`; enum → `text`.

**Use case:** Proto scalar type mapping to SQL/GraphQL types gives gRPC message fields correct query and filter semantics.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/integration/test_graphql_remote_source.py`, `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-325 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-325}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Each query-classified gRPC method is exposed as a virtual read-only table. Output message fields become columns using the type mapping in [REQ-324](#REQ-324). Input message fields become GraphQL query arguments. Server-streaming methods (`Stream*`) collect all streamed response messages into a list before returning rows.

**Use case:** Query-classified gRPC methods exposed as virtual tables let clients query remote services with GraphQL syntax.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/integration/test_graphql_remote_source.py`, `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-326 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-326}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Each mutation-classified gRPC method is exposed as a tracked function (mutation). Input message fields become GraphQL mutation input arguments. The output message schema becomes the mutation's `return_schema`.

**Use case:** Mutation-classified gRPC methods exposed as tracked functions let clients call remote services via GraphQL mutations.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/integration/test_graphql_remote_source.py`, `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-327 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-327}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Query method results are materialized as Parquet in a Trino Iceberg table on S3 (`results.api_cache`, `s3a://provisa-results/api_cache/`). The cache key is a SHA-256 hash of `source_id + method + native args`. Mutations are never cached — they are side-effecting. One `grpc.aio.Channel` is reused per registered source across requests, stored in `AppState.grpc_remote_channels`. The cache table is dropped after TTL expires.

**Use case:** S3/Iceberg materialization of gRPC query results enables Trino SQL filtering over cached rows and eliminates repeated remote calls. Channel reuse reduces connection overhead.

**Code:** `provisa/grpc_remote/`, `provisa/api_source/trino_cache.py`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/integration/test_cache_store.py`, `tests/integration/test_grpc_execution.py`, `tests/unit/test_api_cache.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_grpc_remote_cache.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-328 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-328}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Full governance is applied to gRPC remote source results: RLS, column masking, domain access control, and column visibility are enforced in Stage 2 identically to local table results. The remote gRPC server has no knowledge of Provisa's governance rules.

**Use case:** Stage 2 governance applied to gRPC results ensures remote service data never bypasses RLS or masking.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-329 · gRPC Remote Schema Connector (REQ-322–329) {#REQ-329}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Proto schema refresh is triggered on demand via an admin mutation. On refresh, Provisa re-parses the proto, updates virtual table and tracked function registrations, and preserves any RLS/masking rules applied on top. Proto import paths (for well-known types such as `google/protobuf/timestamp.proto`) are stored at registration time and re-used on refresh.

**Use case:** On-demand proto refresh preserves governance rules while updating registrations when the remote proto changes.

**Code:** `provisa/grpc_remote/`

**Tests:** `tests/integration/test_grpc_execution.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-331 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-331}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A new `ingest` source type allows external services (OTEL Collector, Fluentd, custom webhooks, etc.) to POST JSON events to Provisa via `POST /events/ingest/{source_id}/{table}`. Provisa owns the write path and persists events to a steward-configured backing relational store.

**Use case:** Ingest source type lets external push producers (OTEL Collector, Fluentd) write events to governed tables.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`, `tests/integration/test_ingest_push_integration.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-332 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-332}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Each ingest source specifies a SQLAlchemy-compatible connection (dialect, host, port, database, username/password). Provisa creates one `AsyncEngine` per source at startup and reuses it across all requests. The `dialect` field on the `sources` table carries the SQLAlchemy driver string (e.g. `postgresql+asyncpg`, `mysql+aiomysql`).

**Use case:** Per-source SQLAlchemy AsyncEngine lets ingest tables target any supported relational store asynchronously.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`

### REQ-333 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-333}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Stewards define the schema for each ingest table as a list of columns with `column_name`, `data_type` (SQL type, allowlisted), and `path` (dot-notation extraction path into the POST payload). Provisa auto-generates and executes `CREATE TABLE IF NOT EXISTS` DDL at startup using these definitions. System columns `_received_at` and `_updated_at` (TIMESTAMPTZ) are always injected.

**Use case:** Auto-generated DDL from column definitions creates ingest tables at startup with system audit columns included.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`, `tests/integration/test_ingest_push_integration.py`

### REQ-334 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-334}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The `path` field on `table_columns` uses dot-notation to walk nested JSON payloads. Array index segments are supported (e.g. `resourceLogs.0.resource.attributes`). If `path` is absent, the column name is used as the top-level key. Missing paths yield NULL.

**Use case:** Dot-notation path extraction lets stewards map deeply nested JSON payloads to flat relational columns.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`

### REQ-335 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-335}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The ingest endpoint accepts a single JSON object or a JSON array of objects per request. Each event in the array is extracted and written as a separate row. Returns HTTP 202 with a count of inserted rows. Source or table not found returns 404; engine unavailable returns 503.

**Use case:** Batch ingest endpoint accepts arrays of events and returns inserted count so producers confirm delivery.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`, `tests/integration/test_ingest_push_integration.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-336 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-336}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Ingest tables are subscribable via the standard SSE endpoint (`GET /data/subscribe/{table}`). The subscription provider polls `_updated_at` watermark at a configurable interval (default: 5 s). Full governance is applied: RLS row filtering and column masking are enforced identically to local table subscriptions.

**Use case:** Ingest tables subscribable via SSE lets consumers receive pushed events as a live stream with full governance.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`, `tests/integration/test_ingest_push_integration.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-337 · Ingest Sources — Governed HTTP Push Receiver (REQ-331–337) {#REQ-337}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

`table_columns.data_type TEXT` stores the steward-declared SQL type for ingest columns. This field is also used by future schema introspection tooling for sources where Trino is not available (ingest, API sources).

**Use case:** data_type column on table_columns stores the steward-declared SQL type needed for non-Trino source introspection.

**Code:** `provisa/ingest/`

**Tests:** `tests/unit/test_ingest_ddl.py`, `tests/unit/test_ingest_router.py`

### REQ-338 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-338}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`websocket` is a supported source type. Connects to a WebSocket server, optionally sends a JSON subscribe payload on connect, receives JSON messages as a stream of ChangeEvents.

**Use case:** WebSocket sources feed real-time events (market data, IoT, CDC streams) into the governed data fabric.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`, `tests/integration/test_websocket_rss_integration.py`

### REQ-339 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-339}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

WebSocket provider auto-reconnects on disconnect or error with configurable `reconnect_interval` (default: 5 s). The reconnect loop runs until `close()` is called.

**Use case:** Auto-reconnect ensures continuous event ingestion without operator intervention on transient disconnects.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`, `tests/integration/test_websocket_rss_integration.py`

### REQ-340 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-340}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

WebSocket message fields: `op` (insert/update/delete, default insert), `_ts`/`timestamp` (ISO 8601 or Unix epoch float, default now), all other fields become the row. `event_path` dot-notation extracts a sub-object from the message before mapping.

**Use case:** Standardized message field mapping lets stewards define schema once regardless of upstream message shape.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`

### REQ-341 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-341}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

WebSocket source connection URL is constructed from `host`, `port`, `path`, and `use_ssl` (federation_hints). `subscribe_payload` (JSON string in federation_hints) is sent on connect. `event_path` (federation_hints) selects a sub-field of each message.

**Use case:** URL construction from config allows WebSocket sources to be registered via standard Provisa YAML config.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`

### REQ-342 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-342}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`rss` is a supported source type. Polls an RSS 2.0 or Atom feed URL at a configurable interval (default: 300 s = 5 min). Only items with a publication date newer than the last-seen watermark are emitted as ChangeEvents with `operation="insert"`.

**Use case:** RSS/Atom feeds become governed polling sources for news, alerts, and changelog data.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`, `tests/integration/test_websocket_rss_integration.py`

### REQ-343 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-343}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

RSS parser handles both RSS 2.0 (`<item>`) and Atom (`<entry>`) formats. Fields extracted: `title`, `link`, `description`/`summary`, `published` (pubDate/updated), `id` (guid/id). Dates accept RFC 2822 and ISO 8601 formats; unparseable dates fall back to a sentinel minimum date (`datetime.min`, UTC) to explicitly mark them unparseable rather than silently substituting the current time.

**Use case:** RSS 2.0 and Atom format parity ensures any feed can be registered without format-specific handling.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`, `tests/integration/test_websocket_rss_integration.py`

### REQ-344 · Phase AT — WebSocket & RSS Sources (REQ-338–344) {#REQ-344}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

RSS source URL is resolved from `federation_hints.feed_url` if present, otherwise built from `host`, `port`, `path`, and `use_ssl`. `poll_interval` (federation_hints) overrides the default.

**Use case:** Feed URL resolution from config enables RSS sources to be registered the same way as other API sources.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/unit/test_websocket_provider.py`, `tests/unit/test_rss_provider.py`

### REQ-372 · File & Lake Sources {#REQ-372}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Iceberg and Delta Lake sources support time-travel queries via an optional `as_of` argument on root query fields (snapshot ID, version number, or ISO 8601 timestamp). The compiler translates `as_of` to Trino `FOR TIMESTAMP AS OF` / `FOR VERSION AS OF` syntax. Time-travel is read-only and only available on source types whose Trino connector supports it. Queries against non-time-travel-capable sources that supply `as_of` are rejected at compile time.

**Use case:** Time-travel queries let consumers audit historical snapshots of Iceberg and Delta tables without separate table copies.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_time_travel.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-419 · Vector Search {#REQ-419}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

A model registry must be declared in Provisa config listing available embedding models with id, provider, dimensions, API key env var or base URL, and enabled flag. Models not in the registry cannot be used for embedding generation or query vectorization.

**Use case:** Explicit model registry enables air-gapped deployments and audit trails for embedding model usage.

**Code:** `provisa/vector/registry.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-420 · Vector Search {#REQ-420}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Model registry must support multiple provider types: OpenAI-compatible APIs, Ollama (local), and local HuggingFace model paths. This enables air-gapped and regulated deployments to use on-premise models throughout.

**Use case:** Multi-provider support removes vendor lock-in and enables regulated deployments without external API calls.

**Code:** `provisa/vector/providers.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-421 · Vector Search {#REQ-421}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

A column may be declared as an embedding vector by setting `embedding: true` with attributes: model id (must reference model registry), dimensions, and optional source_column mapping to the physical column name in the source.

**Use case:** Schema-level embedding declarations enable Provisa to manage vector columns as first-class citizens.

**Code:** `provisa/compiler/introspect.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-422 · Vector Search {#REQ-422}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Source capability auto-detection at registration time must identify native vector support: pgvector extension for PostgreSQL, Atlas Vector Search for MongoDB, Cortex for Snowflake. Sources without detected capability are flagged as requiring fallback.

**Use case:** Early capability detection allows fallback materialization strategy to be planned at source registration.

**Code:** `provisa/source_adapters/introspect.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-423 · Vector Search {#REQ-423}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A `cosine_similarity(column, query_vector)` UDF must be available in Provisa SQL. It translates to the native vector operator for sources with native capability (pgvector `<=>`, Snowflake `VECTOR_COSINE_SIMILARITY()`, etc.) and to the pgvector fallback cache for sources without.

**Use case:** Unified UDF hides native vs. fallback complexity from analysts.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-424 · Vector Search {#REQ-424}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

For sources without native vector capability, Provisa must transparently materialize the embedding column to an internal pgvector-enabled PostgreSQL cache table, build an HNSW index, rewrite the query against the cache joining PKs back to the source, and return unified results. The caller must not be aware of the fallback path.

**Use case:** Transparent fallback enables vector search on any source without source-side changes.

**Code:** `provisa/vector/fallback_cache.py`

**Tests:** `tests/unit/test_table_search.py`, `tests/unit/test_vector.py`

### REQ-425 · Vector Search {#REQ-425}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The pgvector fallback cache must support TTL-based invalidation, mutation-triggered invalidation, manual refresh via admin API, and row count drift detection. Stale cache must not silently serve results beyond configured TTL.

**Use case:** Multiple invalidation strategies with drift detection prevent silent data staleness.

**Code:** `provisa/vector/cache_invalidation.py`

**Tests:** `tests/unit/test_config_validation_edge_cases.py`, `tests/unit/test_vector.py`

### REQ-426 · Vector Search {#REQ-426}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Embedding columns are subject to the full Provisa governance stack: RLS, column masking, domain boundary rules, and sensitivity tiers. A `restricted` sensitivity tier on an embedding column must prevent cosine similarity search for unauthorized roles.

**Use case:** Embedding vectors inherit standard governance, preventing privacy bypass through vector search.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-427 · Vector Search {#REQ-427}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A virtual embedding column may be declared on a table with a `generated_from` subquery that must return exactly one text value per row. The subquery may reference multiple columns, apply transformations, and include joins to related tables. Provisa validates the subquery against a sample row at declaration time.

**Use case:** Generated embedding columns enable text-to-vector transformations without storing raw text.

**Code:** `provisa/vector/generation.py`

**Tests:** `tests/unit/test_table_search.py`, `tests/unit/test_vector.py`

### REQ-428 · Vector Search {#REQ-428}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Generated embedding columns must support scheduled incremental refresh — only rows where the source data has changed since last embedding are re-embedded. Full rebuild is triggered on model change or schema change affecting the generating subquery.

**Use case:** Incremental refresh avoids unnecessary embedding API calls and reduces operational cost.

**Code:** `provisa/vector/scheduled_refresh.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-429 · Vector Search {#REQ-429}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Once an embedding column is generated with a declared model, that model is locked for that column. Provisa must reject queries that attempt to use a query vector from a different model or incompatible dimensions, with a clear error identifying the conflict.

**Use case:** Model locking prevents silent semantic errors from mismatched embedding spaces.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/unit/test_vector.py`

### REQ-430 · Vector Search {#REQ-430}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Query-time vectorization must be supported: when a similarity search is expressed with a text string rather than a raw vector, Provisa calls the declared embedding model to generate the query vector before executing the search. Both text input and raw vector input must be supported interfaces.

**Use case:** Query-time vectorization enables natural text-based search without pre-vectorization ceremony.

**Code:** `provisa/vector/query_vectorization.py`

**Tests:** `tests/unit/test_schema_visibility_filters.py`, `tests/unit/test_vector.py`

### REQ-431 · Vector Search {#REQ-431}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Vector search features are organized as three independent phases: (1) native vector search with model registry, embedding column declaration, and cosine_similarity UDF translation; (2) non-native fallback via pgvector cache materialization; (3) declarative embedding generation from source text columns. Each phase is independently deployable and delivers standalone value.

**Use case:** Phased decomposition enables incremental feature adoption and reduces delivery risk.

**Code:** `provisa/vector/`

**Tests:** `tests/unit/test_schema_visibility_filters.py`, `tests/unit/test_vector.py`

### REQ-673 · Cost Statistics {#REQ-673}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Connector cardinality capability — cardinality(source, table) -> Estimate{value, exact, method}. A source may expose a CHEAP volume metric so a caller can size a table without a full scan. Resolution order: (1) a cheap native statistic when the source has one (GraphQL totalCount, OpenAPI X-Total-Count, PG pg_class.reltuples, Trino SHOW STATS, Parquet/Iceberg row-group metadata); (2) an exact COUNT(*) only when the connector reports counting is cheap; (3) UNKNOWN when sizing is expensive and no native estimate exists — fail-open, never a hidden full scan. The Estimate carries `exact` (true for a native count(*)/totalCount, false for reltuples-style estimates) and `method`, so a consumer knows what the number is and what it may be used for. The producer is SPARSE and OPT-IN: a native count is authored only for expensive-count sources — the API sources (openapi/graphql_remote/grpc), where a count today means materializing the full pull, and huge un-indexed row-stores. Cheap-count sources contribute nothing and carry no maintenance. Generalizes and supersedes the GQL-specific count_query helper (provisa/sources/counts.py) under the connector abstraction ([REQ-840](#REQ-840)-843). DONE (2026-07): the estimate type + resolution order are implemented — provisa/federation/ cardinality.py::Estimate(value, exact, method: CardinalityMethod) and resolve_cardinality(native_stat, exact_count_cheap, exact_count) returning, in order, a source's native statistic (exact totalCount or approximate reltuples), else an exact COUNT(*) only when the connector says counting is cheap, else UNKNOWN (fail-open, never a hidden full scan). REMAINING (kept in-progress): the per-source native-stat PRODUCERS wired onto the connectors (graphql totalCount, openapi X-Total-Count, PG reltuples, Trino SHOW STATS, Parquet/Iceberg row-group metadata) — sparse/opt-in for expensive-count sources.

**Use case:** One estimate serves two consumer classes. The AUTOMATED PLANNER reads it to choose a federation strategy (materialize vs virtual, [REQ-826](#REQ-826)), gate hot-table promotion ([REQ-236](#REQ-236)) without a blind COUNT(*), and set redirect/probe thresholds. HUMAN PLANNERS read the same estimate — in the catalog, in plan/EXPLAIN output, and on the cold graph-counts path — to size a source before committing to an expensive federated query, with exact/estimate/unknown surfaced honestly. An "unknown, sizing requires a full pull" answer is itself actionable, unlike today's bare 0.

**Code:** `provisa/federation/cardinality.py`

**Tests:** `tests/unit/test_cardinality.py`

### REQ-540 · GovData Sources {#REQ-540}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Sources of type `govdata` expose U.S. government open data partitioned by subject grouping. Configuring a govdata source with a subject exposes all schemas for that subject automatically.

**Use case:** Subject-grouped GovData sources let stewards selectively grant access to government data by domain area.

**Code:** `provisa/core/models.py`

**Tests:** `tests/integration/test_govdata_source.py`, `tests/unit/test_core_registration.py`

### REQ-541 · GovData Sources {#REQ-541}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The `ref` and `geo` schemas are always included as linker schemas in every GovData source — they are not configurable and not tied to any subject grouping.

**Use case:** Always-included linker schemas ensure GovData joins on reference and geography data are always available without explicit configuration.

**Code:** `provisa/core/models.py`

**Tests:** `tests/integration/test_govdata_source.py`, `tests/unit/test_core_registration.py`

### REQ-550 · Registration & Governance {#REQ-550}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Sources are classified into three execution categories: Direct-capable (has both native driver and federated connector; executes single-source queries via native driver), Federation-only (no direct driver; always routed through Trino), and Materialize (no federated connector; data fetched and cached as Parquet in S3 or PostgreSQL before being queryable by the federation engine).

**Use case:** Tripartite source categorization gives stewards a clear mental model of how each source type executes queries and when the federation engine is involved.

**Code:** `provisa/core/models.py`, `provisa/transpiler/router.py`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/unit/test_routing.py`

### REQ-553 · Registration & Governance {#REQ-553}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

File-based sources (`csv`, `parquet`, `sqlite`) use the `path` config field instead of `host`/`port` connection parameters. The `path` may be a local filesystem path or an fsspec-supported remote object-storage URL (`s3://`, `ftp://`, `sftp://`); the crawler detects fsspec URIs via `fsspec.core.url_to_fs` and walks them the same way as local directories.

**Use case:** File path config for file-based sources lets stewards register local files and remote object-storage files as governed tables using the same config structure as network-connected sources.

**Code:** `provisa/core/models.py`, `provisa/file_source/source.py`, `provisa/file_source/crawler.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_core_registration.py`

### REQ-597 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-597}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

GraphQL remote source registration accepts a `field_overrides` map (`{fieldName: "query" | "mutation"}`) that is applied after introspection and takes priority over structural classification. Only query-type fields can be reclassified as mutations via overrides; mutation-type fields have no override path in GraphQL.

**Use case:** field_overrides lets stewards correct mis-classified remote fields (e.g. a mutation named like a query) at registration time without modifying the upstream schema.

**Code:** `provisa/graphql_remote/mapper.py`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_graphql_remote_overrides_requirements.py`, `tests/unit/test_misc_requirements.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-598 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-598}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Remote schema source registrations (GraphQL, gRPC, OpenAPI) accept a `relationships` list that stores FK/PK join paths as manually declared relationships (no `remote_managed` flag). Relationships auto-detected during introspection are stored with `remote_managed: True`. On schema refresh, auto-detected (`remote_managed: True`) relationships are re-run and may change; manually declared relationships (no flag) are preserved unchanged.

**Use case:** Distinguishing manual from auto-detected relationships ensures steward-declared join paths survive schema refreshes without being overwritten by introspection results.

**Code:** `provisa/graphql_remote/mapper.py`, `provisa/api/admin/graphql_remote_router.py`, `provisa/api/admin/grpc_remote_router.py`, `provisa/api/admin/openapi_router.py`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-599 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-599}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

For all remote schema source types (GraphQL, gRPC, OpenAPI), required input parameters that are not already response fields are registered as `_nf_`-prefixed native filter columns. The `native_filter_type` is `query_param` for GQL and OpenAPI query parameters, `path_param` for OpenAPI path parameters, and `grpc_input` for gRPC. When a parameter name matches a response field name, parameter metadata is merged into the existing column rather than creating a duplicate.

**Use case:** Native filter columns let consumers filter remote-source queries using standard `where` arguments that are passed through to the upstream API rather than applied as post-fetch SQL predicates.

**Code:** `provisa/graphql_remote/mapper.py`, `provisa/api/admin/grpc_remote_router.py`, `provisa/openapi/register.py`, `provisa/api/app.py`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_source.py`, `tests/integration/test_grpc_execution.py`, `tests/integration/test_openapi_source.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-600 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-600}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Remote schema sources enforce depth limits on nested object expansion: `graphql_remote.max_object_depth` (default: 5) caps GQL OBJECT nesting; `graphql_remote.max_list_depth` (default: 2) caps LIST-typed nested OBJECT inclusion; `graphql_remote.max_list_items` (default: 100) injects `first: N` to cap array size. For gRPC and OpenAPI, nested message/object sub-field resolution is one level deep only. Fields beyond these limits are excluded from the fetch.

**Use case:** Depth limits prevent unbounded data expansion and runaway remote API calls when remote schemas contain deeply recursive or list-heavy structures.

**Code:** `provisa/graphql_remote/mapper.py`, `provisa/grpc_remote/mapper.py`, `provisa/openapi/register.py`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_grpc_remote_loader.py`, `tests/unit/test_openapi_loader.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-601 · OpenAPI Auto-Registration Connector {#REQ-601}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

OpenAPI virtual table names are derived from the operation's `operationId`. If no `operationId` is defined, Provisa slugifies `{method}_{path}`. An alias is derived by stripping the leading verb segment and singularizing the noun (e.g. `findPetsByStatus` → `pet_by_status`). The alias is used as the consumer-facing name in GraphQL and other query interfaces.

**Use case:** operationId-based naming and verb-stripped aliases give OpenAPI tables predictable, consumer-friendly names without manual configuration.

**Code:** `provisa/openapi/register.py`

**Tests:** `tests/integration/test_openapi_pet_by_status.py`, `tests/unit/test_graphql_alias.py`, `tests/unit/test_openapi_loader.py`

### REQ-602 · GraphQL Remote Schema Connector (REQ-307–313) {#REQ-602}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Schema generation synthesizes `ColumnMetadata` for remote schema tables (GraphQL remote, gRPC remote, OpenAPI) at schema build time since these sources have no Trino catalog to introspect. Type mapping (GQL scalar → Provisa type, proto scalar → SQL type, JSON Schema → Provisa type) is applied during synthesis to produce typed column metadata equivalent to what catalog introspection produces for local relational tables.

**Use case:** Synthesized ColumnMetadata lets remote schema tables participate in schema generation and query compilation identically to locally connected relational tables.

**Code:** `provisa/compiler/schema_gen.py`, `provisa/api/app.py`

**Tests:** `tests/integration/test_graphql_execution.py`, `tests/integration/test_graphql_remote_source.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_graphql_remote_mapper.py`, `tests/unit/test_remote_adapter_contract.py`, `tests/unit/test_schema_service.py`

### REQ-652 · Vector Search {#REQ-652}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Column masking does not apply to embedding columns — a vector cannot be partially masked and remain semantically meaningful. Access to an embedding column is enforced via visibility (hide entirely), RLS (filter rows), sensitivity tier (block search), and domain boundary rules; the column is either visible and searchable for a given role, or absent from the schema entirely.

**Use case:** Preventing partial-masking of vectors ensures access control is all-or-nothing for embedding columns, which cannot be meaningfully redacted at the field level.

**Code:** `provisa/vector/governance.py`

**Tests:** `tests/unit/test_vector.py`

## 5. Query Languages, Compilation & Operations

### REQ-007 · Compiler & Schema {#REQ-007}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Compiler is purpose-built with no dependency on PostGraphile, DuckDB, or any third-party GraphQL server framework.

**Use case:** Purpose-built compiler eliminates third-party framework constraints and reduces attack surface.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-008 · Compiler & Schema {#REQ-008}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Schema generation pass runs at registration time; queries Trino INFORMATION_SCHEMA, applies per-role column visibility, incorporates relationships, produces GraphQL SDL.

**Use case:** Schema generation at registration time means the GraphQL schema always reflects current data and role visibility.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-009 · Compiler & Schema {#REQ-009}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Query compilation produces PG-style SQL from validated GraphQL AST — single SQL statement, no resolver chain, no N+1. Exception: action query fields ([REQ-361](#REQ-361)) and remote-schema relationships ([REQ-313](#REQ-313)) resolve nested relationships via batched secondary fetches, which is outside this single-statement guarantee.

**Use case:** Single-statement SQL compilation eliminates N+1 query patterns and improves query execution performance.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_graphql_remote_source.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_layer_contracts.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-010 · Compiler & Schema {#REQ-010}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Trino type mapping: VARCHAR→String, INTEGER→Int, BOOLEAN→Boolean, TIMESTAMP→DateTime, JSONB→JSON. Nullability preserved.

**Use case:** Consistent Trino type mapping ensures GraphQL consumers receive predictable, well-typed field values.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-011 · Compiler & Schema {#REQ-011}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

References to unregistered tables, excluded columns, undefined relationships, or type mismatches rejected at compile time with precise errors.

**Use case:** Compile-time rejection of invalid references catches developer errors early before they reach the data layer.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-032 · Mutation Execution {#REQ-032}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

DB mutations are single-source by definition, bypass Trino, no routing decision, no registry approval — but always require write authority on the target table ([REQ-033](#REQ-033)). Webhook mutations are governed separately and require specific steward approval for use ([REQ-209](#REQ-209)).

**Use case:** DB mutations skip the registry approval path but still require table write authority; webhook mutations require explicit approval.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/executor/`

**Tests:** `tests/unit/test_mutation_sql.py`, `tests/integration/test_mutations.py`

### REQ-033 · Mutation Execution {#REQ-033}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

User must have write rights to target table for any mutation.

**Use case:** Write-rights enforcement ensures users can only mutate tables they are explicitly authorized to change.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/executor/`

**Tests:** `tests/unit/test_mutation_sql.py`, `tests/integration/test_mutations.py`

### REQ-034 · Mutation Execution {#REQ-034}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Mutation input types reflect only columns user's role is permitted to write; excluded column references rejected at parse time.

**Use case:** Role-filtered mutation input prevents users from writing to columns their role cannot access.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/executor/`

**Tests:** `tests/unit/test_mutation_sql.py`, `tests/integration/test_mutations.py`

### REQ-035 · Mutation Execution {#REQ-035}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

RLS WHERE clauses injected into UPDATE and DELETE before execution.

**Use case:** RLS injection into UPDATE and DELETE ensures row-level security is enforced on all write operations.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/executor/`

**Tests:** `tests/unit/test_mutation_sql.py`, `tests/integration/test_mutations.py`

### REQ-036 · Mutation Execution {#REQ-036}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Mutations can only target registered tables — compiler does not generate mutation types for unregistered tables.

**Use case:** Mutations only target registered tables, preventing writes to unmanaged or unknown database objects.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/executor/`

**Tests:** `tests/unit/test_mutation_sql.py`, `tests/integration/test_mutations.py`

### REQ-037 · Mutation Execution {#REQ-037}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Cross-source transactions not supported. NoSQL sources do not support mutations.

**Use case:** Restricting mutations to single-source and relational targets prevents partial cross-source transaction failures.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/executor/`

**Tests:** `tests/unit/test_mutation_sql.py`, `tests/integration/test_mutations.py`

### REQ-066 · SQLGlot Transpilation {#REQ-066}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Compiler emits PG-style SQL as canonical output; SQLGlot translates to Trino SQL or target RDBMS dialect.

**Use case:** PG-style canonical SQL output gives a single, well-understood intermediate representation for all dialects.

**Code:** `provisa/transpiler/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_layer_contracts.py`, `tests/unit/test_transpile.py`, `tests/unit/test_transpiler.py`

### REQ-067 · SQLGlot Transpilation {#REQ-067}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Target dialect determined by source type captured at table registration time.

**Use case:** Dialect detection at table registration time means transpilation is automatic and requires no per-query config.

**Code:** `provisa/transpiler/`

**Tests:** `tests/unit/test_transpile.py`, `tests/unit/test_transpiler.py`, `tests/integration/test_compiler_integration.py`

### REQ-068 · SQLGlot Transpilation {#REQ-068}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Supported dialects: PostgreSQL, MySQL, SQL Server, Trino, DuckDB, Snowflake, BigQuery.

**Use case:** Multi-dialect SQLGlot support lets Provisa federate across PostgreSQL, MySQL, SQL Server, Trino, and more.

**Code:** `provisa/transpiler/`

**Tests:** `tests/unit/test_transpile.py`, `tests/unit/test_transpiler.py`, `tests/integration/test_compiler_integration.py`

### REQ-196 · Aggregates {#REQ-196}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Auto-generated aggregate queries following Hasura v2 pattern. Every table gets a `<table>_aggregate` root field. Numeric columns get sum/avg/stddev/variance. All comparable columns get min/max. All columns get count. No configuration required for default behavior.

**Use case:** Hasura v2 parity — auto-generated aggregate fields expose sum/avg/min/max without any steward configuration.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/mv/`

**Tests:** `tests/unit/test_sql_gen_aggregate.py`, `tests/integration/test_compiler_integration.py`

### REQ-197 · Aggregates {#REQ-197}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Per-role aggregate gating via `allow_aggregations` (matching v2) or per-table `aggregates` config section for explicit override of auto-detected functions and role visibility.

**Use case:** Hasura v2 parity — per-role aggregate gating prevents unauthorised users from deriving statistical insights.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/mv/`

**Tests:** `tests/unit/test_sql_gen_aggregate.py`, `tests/integration/test_compiler_integration.py`

### REQ-198 · Aggregates {#REQ-198}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Aggregate MV routing — when a query requests aggregates over a pattern already materialized in an MV, the compiler rewrites the query to use the MV. Requires aggregate catalog + query rewriter.

**Use case:** Aggregate MV routing transparently accelerates aggregate queries without requiring query rewrites.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/mv/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_aggregate_mv_catalog.py`, `tests/unit/test_aggregate_mv_routing.py`, `tests/unit/test_sql_gen_aggregate.py`

### REQ-199 · Aggregates {#REQ-199}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

View auto-materialization for aggregate optimization — expensive views auto-materialized and registered in aggregate catalog instead of requiring bespoke `materialized_views` entries. Default TTL configurable globally via `materialized_views.default_ttl` (seconds, default: 3600). Individual views can override with `refresh_interval`. Stale MVs refreshed by background loop; queries against stale MVs fall back to live execution. (Explicit, defined exception to [REQ-064](#REQ-064) fail-fast — stale-MV→live is designed behavior, not silent error handling.)

**Use case:** View auto-materialization avoids repeated expensive subquery execution for high-traffic computed datasets.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/mv/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_aggregate_mv_catalog.py`, `tests/unit/test_aggregate_mv_routing.py`, `tests/unit/test_sql_gen_aggregate.py`

### REQ-200 · OrderBy Alignment {#REQ-200}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

GraphQL order_by schema must follow Hasura v2 convention: column-keyed input type `{column_name: direction}` instead of current `{field: ENUM, direction: ENUM}` struct.

**Use case:** Hasura v2 parity — column-keyed order_by schema lets Hasura-trained developers sort results intuitively.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_orderby_alignment.py`, `tests/integration/test_compiler_integration.py`

### REQ-201 · OrderBy Alignment {#REQ-201}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

OrderBy direction enum must include 6 values: `asc`, `asc_nulls_first`, `asc_nulls_last`, `desc`, `desc_nulls_first`, `desc_nulls_last`. SQL compiler maps to `ORDER BY col ASC NULLS FIRST`, etc.

**Use case:** Hasura v2 parity — six-value direction enum gives precise NULL-ordering control matching standard SQL.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_orderby_alignment.py`, `tests/integration/test_compiler_integration.py`

### REQ-202 · OrderBy Alignment {#REQ-202}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Relationship ordering — order by related object fields (e.g., `order_by: {author: {name: asc}}`). Matches Hasura v2 default behavior.

**Use case:** Hasura v2 parity — relationship ordering lets clients sort parent rows by child attributes in one request.

**Code:** `provisa/compiler/`

**Tests:** `tests/unit/test_orderby_alignment.py`, `tests/integration/test_compiler_integration.py`

### REQ-205 · Tracked Functions & Custom Mutations {#REQ-205}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Database functions (stored procedures, UDFs) registered in Provisa config and exposed as GraphQL mutations or queries. VOLATILE functions exposed as mutations; STABLE/IMMUTABLE as queries. Follows Hasura v2's `pg_track_function` pattern.

**Use case:** Tracked DB functions expose stored procedures as GraphQL mutations so clients use one API for all writes.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-206 · Tracked Functions & Custom Mutations {#REQ-206}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Function config section in `provisa.yaml` (|) provides structured registration of stored procedures with governance metadata.

**Use case:** Function config section provides a structured way to register stored procedures with governance metadata.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-207 · Tracked Functions & Custom Mutations {#REQ-207}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Function return type MUST reference a registered table. The result set maps back to GraphQL using that table's type definition, column governance (visibility, masking), and RLS rules. This ensures function results go through the same security pipeline as direct queries.

**Use case:** Requiring functions to return a registered table ensures their results go through the same security pipeline.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/e2e/test_query_pipeline.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-208 · Tracked Functions & Custom Mutations {#REQ-208}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Functions execute via direct DB connection (same as mutations). Never routed through Trino.

**Use case:** Direct DB execution for functions avoids Trino round-trips on mutation-class operations.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/integration/test_direct_exec.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-209 · Tracked Functions & Custom Mutations {#REQ-209}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Webhook-backed mutations — external HTTP endpoint called as a GraphQL mutation (Part of Phase AC). Config includes name, url, method, domain_id, governance (requires_approval), arguments with name and type, returns (registered table or inline type), visible_to, and timeout_ms. Webhook mutations are NOT DB mutations ([REQ-031](#REQ-031)/[REQ-032](#REQ-032)) and require specific steward approval for use; `governance: registry-required` gates them.

**Use case:** Webhook mutations let GraphQL clients trigger external services without learning a second API.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-210 · Tracked Functions & Custom Mutations {#REQ-210}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Webhook mutations support inline return type definitions (not backed by a registered table) for cases where the webhook returns a custom shape. Inline types define fields with names and GraphQL types.

**Use case:** Inline return types for webhooks let custom response shapes be exposed without creating a registered table.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-211 · Tracked Functions & Custom Mutations {#REQ-211}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Function and webhook argument types map to GraphQL input types. Arguments are validated at parse time. SQL injection prevented via parameterized calls for DB functions and JSON serialization for webhooks.

**Use case:** Argument validation at parse time prevents SQL injection and type errors before any DB or HTTP call is made.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-252 · Compiler & Schema {#REQ-252}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Schema inference must be supported where the Trino connector provides auto-discovery (MongoDB, Cassandra, Elasticsearch). Sources with inference support should offer a `discover: true` flag that introspects and generates a starting column list. Explicit column definitions always take precedence over inferred schemas. Sources without inference capability (Redis) require explicit column definitions.

**Use case:** Schema inference for supported connectors (MongoDB, Cassandra) reduces manual column definition work.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_discover_inference.py`, `tests/unit/test_mongodb_discovery.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-253 · Compiler & Schema {#REQ-253}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Naming convention changes (global, source, or table level) reflected immediately in GraphQL schema. Admin mutations trigger _rebuild_schemas() which regenerates in-memory graphql-core schema objects. GraphiQL and Voyager fetch fresh introspection on each page load. No client-side schema caching.

**Use case:** Immediate schema rebuild on naming convention changes ensures GraphQL clients always see consistent names.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/e2e/test_naming_reload.py`, `tests/integration/test_schema_gen.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`, `provisa-ui/e2e/registration-admin.spec.ts`

### REQ-259 · Compiler & Schema {#REQ-259}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Apollo Federation v2 subgraph support — when enabled, Provisa generates a Federation v2 compliant schema with @key directives on entity types (derived from primary keys), _service and _entities root fields, and batch entity resolution. Provisa can be composed into an Apollo Gateway/Router supergraph. Entity resolution respects RLS, masking, and role-based visibility. Disabled by default.

**Use case:** Apollo Federation v2 subgraph support lets Provisa participate in a federated supergraph alongside other APIs.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-300 · GraphQL Variable Defaults {#REQ-300}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

GraphQL operations may declare variable default values (e.g. `query Q($limit: Int = 10)`). The compiler MUST apply those defaults for any variable not present in the request `variables` dict, per GraphQL spec §6.4.1. Missing a declared default must not produce a 500 — the default value is used as if the caller had supplied it.

**Use case:** GraphQL variable defaults applied by compiler prevent 500 errors when callers omit optional parameters.

**Code:** `provisa/compiler/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/integration/test_graphql_execution.py`, `tests/unit/test_params.py`

### REQ-301 · GraphQL Variable Defaults {#REQ-301}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

LIMIT and OFFSET values MUST be emitted as positional parameters (`$N`) in compiled SQL, never interpolated as literals. This ensures consistent parameterization regardless of whether the value originated from a GraphQL variable or an inline literal argument.

**Use case:** Parameterized LIMIT/OFFSET prevents SQL injection and ensures consistent query plans regardless of value source.

**Code:** `provisa/compiler/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/integration/test_graphql_execution.py`, `tests/unit/test_params.py`

### REQ-304 · Tracked DB Functions — Custom Return Schema (REQ-304–306) {#REQ-304}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Tracked DB functions MAY declare a `return_schema` (JSON Schema) in place of a `returns` (registered table reference). When `return_schema` is present and `returns` is empty, the schema compiler converts the JSON Schema definition to a `GraphQLObjectType` at schema-build time. `type: array` yields `GraphQLList(GraphQLNonNull(obj))`; `type: object` yields a nullable single object. This enables functions that return custom shapes not backed by any registered table.

**Use case:** Custom return_schema for tracked functions lets stored procedures returning non-table shapes be exposed safely.

**Code:** `provisa/compiler/`, `provisa/api/admin/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_function_gen.py`, `tests/unit/test_schema_service.py`

### REQ-305 · Tracked DB Functions — Custom Return Schema (REQ-304–306) {#REQ-305}

**Status:** ✅ complete · **Priority:** MAY · **Type:** ui

The admin UI for tracked DB functions provides a "Return Type" mode toggle: "Registered Table" (existing `schema.table` select) or "Custom Schema". In custom schema mode, the steward pastes a sample JSON response (single object or array); Provisa infers a JSON Schema from it client-side (field names + primitive types). The steward may edit the inferred schema before saving. The inferred+edited JSON Schema is stored as `return_schema JSONB` on `tracked_functions`.

**Use case:** Admin UI return type toggle lets stewards define custom function return schemas by pasting a sample response.

**Code:** `provisa/compiler/`, `provisa/api/admin/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_function_gen.py`, `tests/unit/test_schema_service.py`

### REQ-306 · Tracked DB Functions — Custom Return Schema (REQ-304–306) {#REQ-306}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

JSON Schema → GraphQL type mapping for `return_schema`: `"string"` → `GraphQLString`, `"integer"` → `GraphQLInt`, `"number"` → `GraphQLFloat`, `"boolean"` → `GraphQLBoolean`. Unknown or missing `type` defaults to `GraphQLString`. Properties with nested objects are not recursively expanded in V1 — only top-level scalar fields are mapped.

**Use case:** JSON Schema to GraphQL type mapping for return_schema ensures functions with custom shapes have typed fields.

**Code:** `provisa/compiler/`, `provisa/api/admin/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_function_gen.py`, `tests/unit/test_schema_service.py`

### REQ-345 · Cypher Query Frontend (Phase AU) {#REQ-345}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa exposes a `POST /query/cypher` endpoint that accepts a Cypher SELECT query and optional named parameters (`$param`). The query is compiled to SQL and executed via Trino. All existing governance (RLS, column masking, domain visibility, row ceiling) applies via Stage 2 identically to GraphQL-compiled queries.

**Use case:** Graph-native users write Cypher against Provisa without learning GraphQL or SQL.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`, `provisa-ui/e2e/cypher-api.spec.ts`

### REQ-346 · Cypher Query Frontend (Phase AU) {#REQ-346}

**Status:** ✗ rejected · **Priority:** MUST · **Type:** constraint

SUPERSEDED BY [REQ-818](#REQ-818). Originally: the Cypher compiler is strictly read-only, rejecting any write clause (CREATE/MERGE/SET/DELETE/DETACH/REMOVE) or APOC reference at parse time. Cypher now supports governed CREATE/DELETE/SET writes ([REQ-818](#REQ-818)); only MERGE/DETACH/REMOVE and APOC remain rejected. Use [REQ-818](#REQ-818) and [REQ-671](#REQ-671) instead.

**Use case:** Read-only enforcement prevents graph users from accidentally mutating data via Cypher.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-347 · Cypher Query Frontend (Phase AU) {#REQ-347}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher clauses map to SQL as follows: `MATCH` → `JOIN`; `OPTIONAL MATCH` → `LEFT JOIN`; `WHERE` → `WHERE`; `RETURN` → `SELECT`; `ORDER BY` → `ORDER BY`; `SKIP`/`LIMIT` → `OFFSET`/`LIMIT`; `WITH` (pipeline) → CTE or subquery. Node label `:Label` and relationship type `:TYPE` are resolved via the steward-declared label mapping ([REQ-351](#REQ-351)) to physical table and join references.

**Use case:** Cypher clause mapping enables graph queries to execute as governed SQL via Trino.

**Code:** `provisa/cypher/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-348 · Cypher Query Frontend (Phase AU) {#REQ-348}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Path queries — `shortestPath(...)`, `allShortestPaths(...)`, and variable-length relationship patterns `[*1..n]` — translate to Trino recursive CTEs (`WITH RECURSIVE`) against the adjacency relation defined in the label mapping. Maximum hop depth is enforced at compile time; unbounded `[*]` is rejected.

**Use case:** Path queries and variable-length traversals translate to recursive CTEs for graph algorithms over relational data.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-349 · Cypher Query Frontend (Phase AU) {#REQ-349}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When a `RETURN` clause references a whole node variable, relationship variable, or path variable (not a scalar property), a Stage 3 SQLGlot rewrite pass wraps the projected columns for that variable into a single JSON object column using `CAST(ROW(...) AS JSON)`. This rewrite runs after Stage 2 governance and before execution. It does not modify scalar property projections.

**Use case:** Node/edge/path return types give graph clients structured output matching their mental model.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-350 · Cypher Query Frontend (Phase AU) {#REQ-350}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Three graph output types are defined in the GraphQL schema and returned when Stage 3 wrapping is applied: `Node { id: ID!, label: String!, properties: JSON }`, `Edge { id: ID!, type: String!, startNode: Node!, endNode: Node!, properties: JSON }`, `Path { nodes: [Node!]!, edges: [Edge!]! }`. Scalar property projections return plain column types as normal.

**Use case:** Typed graph output (Node, Edge, Path) lets graph clients parse results without custom deserialization.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-351 · Cypher Query Frontend (Phase AU) {#REQ-351}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The Cypher label map is derived entirely from the existing Provisa `CompilationContext` — no separate config. `TableMeta.type_name` (PascalCase GraphQL type name) becomes the Cypher node label; registered `JoinMeta` entries become relationship types. Consumers write Cypher using the same type names already visible in the Provisa GraphQL schema. No additional registration step is required.

**Use case:** Cypher label map derived from existing compilation context — no extra registration step for Cypher users.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-352 · Cypher Query Frontend (Phase AU) {#REQ-352}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher named parameters (`$param`) are translated to Trino positional parameters at compile time. Parameter types are inferred from the label mapping schema. Missing parameters with no default are rejected at compile time.

**Use case:** Named parameters in Cypher translate to parameterized SQL; prevents injection and enables query reuse.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-353 · Cypher Query Frontend (Phase AU) {#REQ-353}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

WITHDRAWN (2026-06-19). Cross-source Cypher queries are allowed — Trino joins across catalogs natively, so a query whose labels resolve to tables on different sources translates and executes normally. No cross-source restriction is enforced. (Supersedes REQ-481.)

**Use case:** Cross-source graph traversal is a core capability, not an error.

**Code:** `provisa/cypher/`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-354 · Natural Language Query Service (Phase AV) {#REQ-354}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa exposes a `POST /query/nl` endpoint accepting a natural language question. The service submits an async job and returns a `job_id` immediately. The consumer polls `GET /query/nl/{job_id}` or receives the result via SSE when complete.

**Use case:** Natural language entry point for non-technical users who cannot write SQL, GraphQL, or Cypher.

**Code:** `provisa/nl/`

**Tests:** `tests/unit/test_nl_loop.py`, `tests/unit/test_nl_runner.py`, `tests/integration/test_nl_endpoint.py`, `provisa-ui/e2e/cypher-api.spec.ts`

### REQ-355 · Natural Language Query Service (Phase AV) {#REQ-355}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

For each submitted NL query the service runs three independent parallel generation loops — one targeting Cypher, one GraphQL, and one SQL. Each loop uses an LLM to generate a candidate query, validates it through the existing Provisa compiler pipeline (parse + compile, no execution), feeds any compiler error back to the LLM as a correction signal, and retries until the query is valid or a max iteration limit is reached. The three loops are independent — failure in one does not block the others.

**Use case:** Parallel three-target generation maximizes answer quality; one valid result is enough to respond.

**Code:** `provisa/nl/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_nl_endpoint.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_nl_loop.py`, `tests/unit/test_nl_runner.py`, `tests/steps/steps_natural_language_query_service_phase_av.py`

### REQ-356 · Natural Language Query Service (Phase AV) {#REQ-356}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The LLM prompt for each generation loop is scoped to the consumer's role. The schema context supplied to the LLM is the GraphQL SDL visible to that role — tables, fields, and relationships the consumer is not permitted to see are absent from the prompt. Generated queries referencing invisible schema elements are rejected by the compiler, triggering a refinement loop iteration. Governance is enforced at two layers: prompt scoping and compiler validation.

**Use case:** Role-scoped LLM prompt prevents the AI from generating queries referencing data the user cannot see.

**Code:** `provisa/nl/`

**Tests:** `tests/unit/test_nl_loop.py`, `tests/unit/test_nl_runner.py`, `tests/integration/test_nl_endpoint.py`

### REQ-357 · Natural Language Query Service (Phase AV) {#REQ-357}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Once all three generation loops produce valid queries, all three are executed in parallel via the standard Provisa pipeline (Stage 2 governance applies to each). The response includes all three query texts and their results: `{ cypher: { query, result }, graphql: { query, result }, sql: { query, result } }`. A branch that exhausts its iteration limit returns `{ query: null, result: null, error: "<last compiler error>" }` for that branch without blocking the response.

**Use case:** All three query forms returned lets power users choose the most legible or efficient representation.

**Code:** `provisa/nl/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_nl_endpoint.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_nl_loop.py`, `tests/unit/test_nl_runner.py`, `tests/steps/steps_natural_language_query_service_phase_av.py`

### REQ-358 · Natural Language Query Service (Phase AV) {#REQ-358}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The NL query service is differentiated from commodity text-to-SQL tools by: (1) three-target generation (SQL, GraphQL, Cypher) — Cypher output for graph-shaped questions is not available in any comparable tool; (2) role-scoped schema context — the LLM cannot generate queries referencing data the consumer is not permitted to see; (3) compiler-driven refinement loop — deterministic validation replaces heuristic result quality checks.

**Use case:** Differentiator: only platform offering role-scoped, compiler-validated, multi-target NL query generation.

**Code:** `provisa/nl/`

**Tests:** `tests/integration/test_nl_endpoint.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_nl_loop.py`, `tests/unit/test_nl_runner.py`, `tests/steps/steps_natural_language_query_service_phase_av.py`

### REQ-359 · Natural Language Query Service (Phase AV) {#REQ-359}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Any valid query produced by the NL service is executed directly under the consumer's rights, and may be saved by the consumer for reuse. Generated queries are governed by the same rights + Stage 2 enforcement as hand-written queries — the NL service never bypasses governance.

**Use case:** NL-generated queries run under the consumer's rights with full Stage 2 governance — never bypassed.

**Code:** `provisa/nl/`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/integration/test_nl_endpoint.py`, `tests/unit/test_nl_loop.py`, `tests/unit/test_nl_runner.py`

### REQ-360 · Tracked Functions & Custom Mutations {#REQ-360}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Action query fields (tracked functions and webhooks with `exposed_as: query`) must support standard filter/sort/pagination GraphQL arguments: `where`, `order_by`, `limit`, `offset` — applied as Python post-processing after the function executes and results are materialized.

**Use case:** Filter/sort/pagination on action query fields gives function-backed results the same query ergonomics as tables.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-361 · Tracked Functions & Custom Mutations {#REQ-361}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Action query fields returning a known table type must resolve nested relationship fields by batching lookups against the related source table and merging results back onto each row. Relationship resolution applies the same governance rules (column visibility, RLS, column masking) as direct table queries.

**Use case:** Governed relationship resolution on action results ensures nested fields respect RLS, masking, and visibility like direct queries.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-362 · Tracked Functions & Custom Mutations {#REQ-362}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

One-to-many relationships on action result rows must return an array field; many-to-one relationships must return an object field or null. Relationship cardinality is sourced from the JoinMeta declarations that define the table relationships.

**Use case:** Cardinality-correct relationship fields (array vs object) on action results match the JoinMeta declarations.

**Code:** `provisa/compiler/`, `provisa/executor/`, `provisa/webhooks/`

**Tests:** `tests/e2e/test_mutations.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_actions.py`, `tests/unit/test_function_gen.py`

### REQ-403 · Compiler & Schema {#REQ-403}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

RLS injection via `inject_rls()` checks for table-specific rules first, then falls back to domain-level rules; table-level rules take precedence.

**Use case:** Table-level RLS precedence lets stewards override domain rules selectively for exceptions without duplicating configuration.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_rls_compiler_fallback.py`, `tests/unit/test_schema_service.py`

### REQ-409 · Compiler & Schema {#REQ-409}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher translator detects ISO 8601 datetime string literals in WHERE clauses and wraps them as `TIMESTAMP '...'` before SQLGlot parsing, fixing Trino type mismatch errors when filtering on timestamp primary key columns.

**Use case:** Automatic timestamp coercion eliminates manual casting for timestamp PK comparisons in Cypher queries.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_schema_service.py`

### REQ-411 · Compiler & Schema {#REQ-411}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Naming convention option `hasura-default` provides snake_case mutation names (insert_orders, update_orders, delete_orders) and snake_case field names, matching Hasura V2's default naming style.

**Use case:** Alternative naming convention supports teams migrating from Hasura by providing compatible mutation and field name generation.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-412 · Compiler & Schema {#REQ-412}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Naming convention option `graphql-default` provides camelCase field names, PascalCase types, and camelCase mutations (insertOrders, updateOrders), matching Hasura V2's graphql-default and GraphQL best practices. This is now the Provisa default.

**Use case:** GraphQL best-practice naming convention improves developer experience and aligns with ecosystem defaults.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-416 · Compiler & Schema {#REQ-416}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Naming convention now uses three preset enums (`snake`, `hasura_graphql`, `apollo_graphql` default) with `domain_prefix: bool` as orthogonal option. `snake` (PascalCase types, snake_case fields/mutations), `hasura_graphql` (PascalCase types, camelCase fields, snake_case mutations matching Hasura V2), `apollo_graphql` (PascalCase types, camelCase fields/mutations per Apollo/GraphQL idiom). Old free-form strings deprecated.

**Use case:** Enum-based preset naming eliminates ambiguous string values and standardizes on clear, documented conventions compatible with Hasura and Apollo ecosystems.

**Code:** `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_naming_convention_validation.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-478 · Compiler & Schema {#REQ-478}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Statistical row sampling is a user query feature, not governance. Root query fields accept an optional `sample: Float` argument (percentage in (0, 100]); the compiler emits `TABLESAMPLE BERNOULLI (<pct>)` on the base table, which SQLGlot transpiles per dialect (direct PG and Trino both supported). Out-of-range values are rejected at compile time. Not combinable with `as_of` (time-travel) or op-relationship lateral joins. Raw-SQL clients use native `TABLESAMPLE` directly. Distinct from the [REQ-263](#REQ-263) row cap, which bounds result size.

**Use case:** Statistical sampling lets analysts request a random subset for exploration, separate from the governance row cap.

**Code:** `provisa/compiler/sql_gen.py`, `provisa/compiler/schema_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_schema_service.py`, `tests/unit/test_sql_gen.py`

### REQ-525 · Compiler & Schema {#REQ-525}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Auto-generated `.proto` from the data schema is generated per role. Each role receives a proto definition reflecting only the tables and columns visible to that role.

**Use case:** Per-role proto generation ensures gRPC clients only see schema elements they are authorized to access.

**Code:** `provisa/grpc/proto_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_grpc_requirements.py`, `tests/unit/test_schema_service.py`

### REQ-526 · Compiler & Schema {#REQ-526}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

gRPC server reflection is enabled on the Provisa Protobuf gRPC server (port 50051), allowing clients to discover available RPCs and message schemas at runtime without a pre-shared proto file.

**Use case:** Server reflection enables tooling (grpcurl, grpcui, Postman) to discover gRPC services without distributing proto files.

**Code:** `provisa/grpc/reflection.py`, `provisa/grpc/server.py`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_grpc_reflection.py`, `tests/unit/test_grpc_server.py`, `tests/unit/test_schema_service.py`

### REQ-534 · Compiler & Schema {#REQ-534}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

GraphQL queries with multiple root fields are compiled into separate SQL queries and executed independently. Results are merged into a single response: fields below the redirect threshold are returned inline in `data`; fields above the threshold are redirected with per-field entries in `redirects`. Binary formats (Parquet, Arrow) are only supported for single-root queries.

**Use case:** Multi-root query execution lets GraphQL clients fetch multiple independent datasets in a single request while applying per-field redirect thresholds.

**Code:** `provisa/api/data/endpoint.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/integration/test_direct_exec.py`, `tests/unit/test_compiler_requirements.py`, `tests/unit/test_multi_root_queries.py`, `tests/unit/test_schema_service.py`

### REQ-537 · Compiler & Schema {#REQ-537}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`GET /data/schema-version` returns a string combining a per-boot UUID nonce with a monotonically incrementing rebuild counter in the format `<boot-id>-<counter>`. Clients use this value to detect schema changes and invalidate local schema caches.

**Use case:** Schema version endpoint lets clients detect server-side schema changes and re-fetch SDL or introspection without polling.

**Code:** `provisa/api/data/sdl.py`

**Tests:** `tests/integration/test_schema_gen.py`, `tests/unit/test_compiler_requirements.py`, `tests/unit/test_schema_service.py`, `provisa-ui/e2e/cypher-api.spec.ts`

### REQ-571 · Cypher Query Frontend (Phase AU) {#REQ-571}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The Cypher parser is a custom recursive-descent implementation with no dependency on any external Cypher library. It produces a CypherAST from read-only Cypher queries.

**Use case:** Custom parser with no external library dependency gives full control over the parse grammar and error messages for Provisa-specific Cypher extensions.

**Code:** `provisa/cypher/parser.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-572 · Cypher Query Frontend (Phase AU) {#REQ-572}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa handles CALL db.labels(), CALL db.relationshipTypes(), and CALL db.propertyKeys() as introspection procedures that return data from the in-memory semantic layer (CypherLabelMap) without generating or executing any SQL.

**Use case:** Returning schema introspection data from the semantic layer (not the database) lets Neo4j-compatible clients discover the graph schema via standard procedure calls.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-573 · Cypher Query Frontend (Phase AU) {#REQ-573}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Correlated CALL subqueries of the form CALL { WITH x MATCH (x)-[:R]->(n) RETURN n.prop AS alias } are translated to CROSS JOIN LATERAL expressions. The outer-scope variable must appear in WITH; multiple imported variables are supported. Non-correlated top-level CALL blocks (without WITH) are handled by cypher_calls_to_sql_list.

**Use case:** CROSS JOIN LATERAL translation of correlated CALL subqueries lets graph clients use subquery-pattern idioms that are common in openCypher workloads.

**Code:** `provisa/cypher/correlated_call.py`, `provisa/cypher/translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-574 · Cypher Query Frontend (Phase AU) {#REQ-574}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Relationships in Provisa exist solely as join metadata in the semantic layer (JoinMeta). They carry no stored attributes. Relationship property access (e.g. WHERE r.since > 2020 or RETURN r.weight) has no meaning and is not supported.

**Use case:** Restricting relationships to join metadata avoids the complexity of relationship property storage while covering the vast majority of graph traversal use cases.

**Code:** `provisa/cypher/label_map.py`, `provisa/cypher/translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-575 · Cypher Query Frontend (Phase AU) {#REQ-575}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Bidirectional traversal syntax (a)-[]-(b) is rewritten at compile time to a UNION ALL of all matching directed forward and backward relationships from the semantic layer. Every relationship in the semantic layer is directional; bidirectional syntax is sugar that expands to both directions.

**Use case:** Bidirectional sugar covers common graph traversal patterns without requiring stewards to register both directions of every relationship.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-576 · Cypher Query Frontend (Phase AU) {#REQ-576}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When shortestPath endpoints have different node types and no self-referential relationship exists in the schema, the translator emits a flat JOIN chain (structurally shortest schema path) rather than a recursive CTE with ORDER BY hops. Hops are not tracked in this code path, so the result is the structurally shortest schema path, not the data-shortest path across multiple rows.

**Use case:** Flat JOIN for heterogeneous shortestPath avoids unnecessary recursive CTE overhead when the schema topology determines the path uniquely.

**Code:** `provisa/cypher/path_translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_path_translator.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-577 · Cypher Query Frontend (Phase AU) {#REQ-577}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When multiple schema paths of equal hop count connect the same start and end node types, all matching paths are emitted as UNION ALL branches. Row-level deduplication across branches is not performed.

**Use case:** Emitting all schema paths as UNION ALL branches ensures no valid traversal is silently omitted when multiple relationship types connect the same pair of node types.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-578 · Cypher Query Frontend (Phase AU) {#REQ-578}

**Status:** ✅ complete · **Priority:** MAY · **Type:** constraint

WITH clause CTEs are named _w0, _w1, ... using a positional index assigned within a single translation call. Composing multiple independently translated queries (e.g. in a batch) can produce colliding CTE names if the results are concatenated without renaming.

**Use case:** Positional CTE naming is deterministic within one translation call; the collision risk on batch composition is a known limitation that callers must handle.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-642 · Graph Analytics Pipeline {#REQ-642}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A POST /data/graph-analytics endpoint accepts a Cypher query and algorithm name, executes the query via the existing cypher_router pipeline, builds an in-memory NetworkX DiGraph from the resulting nodes and edges, runs the named algorithm, merges a `_analytics` dict into each node/edge, and returns the augmented nodes and edges as JSON with an `elapsed_ms` field.

**Use case:** Server-side graph analytics endpoint lets the browser request algorithm output on any Cypher-defined subgraph without building the graph client-side.

**Code:** `provisa/api/rest/graph_analytics_router.py`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_graph_analytics_requirements.py`, `provisa-ui/e2e/cypher-api.spec.ts`

### REQ-643 · Graph Analytics Pipeline {#REQ-643}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The graph analytics response merges a `_analytics` dict into every node and edge in the result. The keys present in `_analytics` vary by algorithm: centrality algorithms produce `score`; community detection produces `cluster`; k-core produces `core_number`; degree centrality also produces `in_degree` and `out_degree`.

**Use case:** Uniform _analytics key convention lets the UI apply visual encodings without algorithm-specific branching.

**Code:** `provisa/api/rest/graph_analytics_router.py`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_graph_analytics_requirements.py`

### REQ-650 · Graph Analytics Pipeline {#REQ-650}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The graph analytics endpoint enforces a configurable maximum graph size. When the input graph exceeds the configured limit (default: 10,000 nodes or 50,000 edges), the endpoint returns HTTP 413 before running any algorithm.

**Use case:** Graph size cap prevents unbounded memory allocation from oversized graphs passed to in-process algorithm libraries.

**Code:** `provisa/api/rest/graph_analytics_router.py`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_graph_analytics_requirements.py`

### REQ-651 · Graph Analytics Pipeline {#REQ-651}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The Girvan-Newman community detection algorithm is restricted to graphs with fewer than 500 nodes. Requests for Girvan-Newman on larger graphs are rejected unless the caller supplies `force=true` in the params, making the computational risk explicit.

**Use case:** Girvan-Newman restriction prevents O(n³) runtime from making large-graph analytics requests impractical.

**Code:** `provisa/api/rest/graph_analytics_router.py`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_compiler_integration.py`, `tests/unit/test_graph_analytics_requirements.py`

### REQ-653 · Compiler & Schema {#REQ-653}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Table-level `enable_aggregates` and `enable_group_by` flags in provisa.yaml (both default `false`). `enable_aggregates` replaces the role-level `no_aggregations` capability as the primary gate for `{table}_aggregate` root fields. `enable_group_by` gates `{table}_group_by` root fields. Stewards opt in per table, controlling SDL noise at the table level rather than globally per role.

**Use case:** Steward can expose aggregate and group-by fields only on tables where they are meaningful, keeping the SDL small and navigable for API consumers.

**Code:** `provisa/compiler/schema_gen.py`, `provisa/core/models.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_core_registration.py`, `tests/unit/test_schema_service.py`, `provisa-ui/e2e/tables-enable-aggregates.spec.ts`

### REQ-654 · Compiler & Schema {#REQ-654}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`{table}_group_by` root query field. Requires `by: [SelectColumn!]!`; accepts same args as the base table query (`where`, `order_by`, `limit`, `offset`, `distinct_on`). Returns `[{TypeName}GroupByRow!]!` where `{TypeName}GroupByRow` has `groupKey: JSON!` (object mapping group-by column names to their values) and `aggregates: {TypeName}AggregateFields!`. Reuses existing `{TypeName}AggregateFields` type unchanged. Maps to `SELECT <by_cols>, <agg_funcs> FROM <table> WHERE <where> GROUP BY <by_cols> ORDER BY <order_by> LIMIT <limit> OFFSET <offset>`.

**Use case:** Exposes grouped analytics (e.g. count per user, sum per category) as a first-class query without requiring a database view.

**Code:** `provisa/compiler/schema_gen.py`, `provisa/compiler/aggregate_gen.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_group_by.py`, `tests/unit/test_schema_service.py`

### REQ-655 · Compiler & Schema {#REQ-655}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`{table}_group_by` supports two additional filter clauses beyond the base `where`: (1) `aggregates(where: {TypeName}BoolExp)` — a `where` arg on the `aggregates` field of `{TypeName}GroupByRow` that applies conditional aggregation within each group (SQL `FILTER (WHERE ...)`); (2) `having: {TypeName}HavingExp` arg on the root field that filters groups post-aggregation (SQL `HAVING`). `{TypeName}HavingExp` is a new input type with the same numeric/comparable fields as `{TypeName}AggregateFields` but with comparison operators. Three filters compose orthogonally: `_group_by(where:)` → SQL WHERE, `aggregates(where:)` → SQL FILTER, `having:` → SQL HAVING.

**Use case:** Enables queries like "users with more than 3 active inquiries" without client-side filtering or custom views.

**Code:** `provisa/compiler/aggregate_gen.py`, `provisa/compiler/sql_gen.py`

**Tests:** `tests/integration/test_compiler_integration.py`, `tests/unit/test_group_by.py`, `tests/unit/test_schema_service.py`

### REQ-661 · Cypher Mutations {#REQ-661}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Labels in Cypher write clauses (CREATE, SET, DELETE) must resolve to exactly one NodeMeta via the semantic label map. Ambiguous or unknown labels are hard errors at translation time; no fuzzy matching or implicit selection is permitted.

**Use case:** Single-label resolution ensures write operations are unambiguous and cannot accidentally target the wrong table.

**Code:** `provisa/cypher/write_translator.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-662 · Cypher Mutations {#REQ-662}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Labels in write operations must already exist in the governed semantic label map. New labels cannot be created via Cypher write clauses — only pre-registered tables are writable.

**Use case:** Restricting writes to pre-registered tables ensures schema integrity and prevents untracked table creation.

**Code:** `provisa/cypher/write_translator.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-663 · Cypher Mutations {#REQ-663}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

All Cypher write operations (CREATE, SET, DELETE) are gated on `writable_by` ACL defined on the target table's NodeMeta. If a role lacks write permission, the query is rejected at compile time. The check applies uniformly to CREATE, SET, and DELETE.

**Use case:** Consistent write access control across all mutation types prevents unauthorized writes to sensitive tables.

**Code:** `provisa/cypher/write_translator.py`, `provisa/api/rest/cypher_router.py`, `provisa/api/data/endpoint.py`, `provisa/compiler/stage2.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_mutations.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-664 · Cypher Mutations {#REQ-664}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A source connector must explicitly support DML (INSERT, UPDATE, DELETE) to be writable. Read-only connectors (e.g., Trino-federated, iceberg catalog without Delta connector) reject write operations at translation time with a clear error message.

**Use case:** Restricting writes to DML-capable sources prevents silent failures on read-only systems.

**Code:** `provisa/cypher/write_translator.py`, `provisa/core/models.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-665 · Cypher Mutations {#REQ-665}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Relationships in Cypher cannot be written — they are derived from foreign-key joins in the semantic layer, not stored edges. CREATE, SET, and DELETE cannot target relationship entities. Attempting to write a relationship is a hard error at translation time.

**Use case:** Relationship immutability prevents accidental FK manipulation and keeps relationship metadata consistent with the declared join paths.

**Code:** `provisa/cypher/write_translator.py`, `provisa/cypher/label_map.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_mutations.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-666 · Cypher Mutations {#REQ-666}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`CREATE (n:Label {props})` is translated to `INSERT INTO catalog.schema.table (columns) VALUES (values)`. Property names in the Cypher object are mapped to table columns via domain-prefix stripping and alias resolution. Type coercion aligns Cypher scalar values with target column types.

**Use case:** Property-to-column mapping with type coercion enables straightforward graph-idiomatic CREATE statements without explicit SQL casting.

**Code:** `provisa/cypher/write_translator.py`, `provisa/cypher/select_builder.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`, `tests/integration/test_cypher_endpoint.py`

### REQ-667 · Cypher Mutations {#REQ-667}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`MATCH (n:Label) WHERE ... DELETE n` is translated to `DELETE FROM catalog.schema.table WHERE ...`. The WHERE clause is reused from the existing MATCH translator; pattern conditions must compile to standard SQL WHERE predicates before deletion.

**Use case:** WHERE clause reuse avoids duplicating filter translation logic between read and write paths.

**Code:** `provisa/cypher/write_translator.py`, `provisa/cypher/select_builder.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`, `tests/integration/test_cypher_endpoint.py`

### REQ-668 · Cypher Mutations {#REQ-668}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`MATCH (n:Label) WHERE ... SET n.prop = val` is translated to `UPDATE catalog.schema.table SET column = value WHERE ...`. Property-to-column mapping applies domain-prefix stripping; assignment expressions must be valid SQL expressions. Multiple SET clauses compose as comma-separated column updates.

**Use case:** Property-to-column mapping with multi-assignment composition lets graph clients issue idiomatic UPDATE patterns.

**Code:** `provisa/cypher/write_translator.py`, `provisa/cypher/select_builder.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_label_map.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`, `tests/integration/test_cypher_endpoint.py`

### REQ-669 · Cypher Mutations {#REQ-669}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Write operations use a separate parser entry point or translation flag to avoid breaking the read-only query path. Existing read-only parser must remain unchanged. The WriteTranslator class encapsulates CREATE, SET, and DELETE translation logic separately from QueryTranslator.

**Use case:** Separate write translation path prevents mutation syntax from breaking existing read-only query parsing and execution.

**Code:** `provisa/cypher/write_translator.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_path_translator.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

### REQ-670 · Cypher Mutations {#REQ-670}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cypher write endpoints return the number of rows affected (rows inserted for CREATE, rows updated for SET, rows deleted for DELETE). The response includes an `affected_rows` field in the JSON response body.

**Use case:** Affected-row counts let clients validate mutation success and detect silent failures.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`, `tests/integration/test_cypher_endpoint.py`, `provisa-ui/e2e/cypher-write-affected-rows.spec.ts`

### REQ-671 · Cypher Mutations {#REQ-671}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

DETACH DELETE (cascading delete across derived relationships) is not supported. Attempting DETACH DELETE is a hard error at parse or translation time. Clients must explicitly DELETE each table in the correct dependency order.

**Use case:** Explicit cascade prevention avoids accidental data deletion across unintended table relationships.

**Code:** `provisa/cypher/write_translator.py`, `provisa/cypher/parser.py`

**Tests:** `tests/e2e/test_mutations.py`, `tests/unit/test_cypher_translator.py`, `tests/unit/test_sql_to_cypher.py`

## 6. Execution, Routing, Caching & Performance

### REQ-027 · Execution & Routing {#REQ-027}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Single-source queries route to direct RDBMS connection; SQLGlot transpiles to target dialect. Target: sub-100ms to low hundreds of ms.

**Use case:** Single-source direct routing delivers sub-100ms query latency for the majority of requests.

**Code:** `provisa/executor/`, `provisa/transpiler/`

**Tests:** `tests/e2e/test_routing.py`, `tests/integration/test_direct_exec.py`, `tests/integration/test_execution_routing.py`, `tests/unit/test_routing.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-028 · Execution & Routing {#REQ-028}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cross-source queries route to Trino; SQLGlot transpiles to Trino SQL. Target: 300-500ms.

**Use case:** Cross-source Trino routing enables federated queries across databases within a 300–500ms target.

**Code:** `provisa/executor/`, `provisa/transpiler/`

**Tests:** `tests/unit/test_routing.py`, `tests/e2e/test_routing.py`, `tests/integration/test_execution_routing.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-029 · Execution & Routing {#REQ-029}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Large results above threshold redirect to blob storage with presigned URL and TTL.

**Use case:** Large-result redirect to blob storage prevents memory exhaustion on the Provisa server for big datasets.

**Code:** `provisa/executor/`, `provisa/transpiler/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/e2e/test_routing.py`, `tests/integration/test_direct_exec.py`, `tests/integration/test_execution_routing.py`, `tests/unit/test_redirect_s3.py`, `tests/unit/test_routing.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-030 · Execution & Routing {#REQ-030}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Steward routing override configurable per table/view for edge cases where default routing is inappropriate.

**Use case:** Per-table steward routing overrides handle edge cases where the default routing heuristic is suboptimal.

**Code:** `provisa/executor/`, `provisa/transpiler/`

**Tests:** `tests/unit/test_routing.py`, `tests/e2e/test_routing.py`, `tests/integration/test_execution_routing.py`

### REQ-031 · Execution & Routing {#REQ-031}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

DB mutations ALWAYS route to direct RDBMS connection — Trino never involved in mutation execution. Webhook mutations are NOT DB mutations: they reference an external HTTP endpoint that presumably changes underlying data. Provisa cannot verify whether anything changed, so this is the steward's responsibility to define (see [REQ-209](#REQ-209)).

**Use case:** DB mutations always bypass Trino, ensuring writes are atomic and target the correct source directly.

**Code:** `provisa/executor/`, `provisa/transpiler/`

**Tests:** `tests/e2e/test_routing.py`, `tests/integration/test_direct_exec.py`, `tests/integration/test_execution_routing.py`, `tests/unit/test_routing.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-052 · Data & Storage {#REQ-052}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Each registered RDBMS source maintains warm connection pool; min pool size configurable per source.

**Use case:** Warm connection pools eliminate per-request TCP handshake latency for frequently queried RDBMS sources.

**Code:** `provisa/executor/`, `provisa/core/`

**Tests:** `tests/unit/test_connection_pool.py`, `tests/integration/test_pool.py`

### REQ-053 · Data & Storage {#REQ-053}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

PgBouncer is the documented per-PostgreSQL-source opt-in via `use_pgbouncer` (default off); it is NOT forced on, because a default-on would require a running PgBouncer for every PG source. Default is direct asyncpg pooling ([REQ-052](#REQ-052)).

**Use case:** Optional PgBouncer per-source avoids mandatory sidecar infrastructure while offering connection pooling where needed.

**Code:** `provisa/executor/`, `provisa/core/`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/integration/test_pool.py`, `tests/unit/test_connection_pool.py`

### REQ-054 · Data & Storage {#REQ-054}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Trino read path maintains single persistent connection to coordinator.

**Use case:** Single persistent Trino coordinator connection avoids repeated authentication and session setup overhead.

**Code:** `provisa/executor/`, `provisa/core/`

**Tests:** `tests/unit/test_connection_pool.py`, `tests/integration/test_pool.py`

### REQ-230 · Hot Tables (Redis-Cached Lookups) {#REQ-230}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Hot tables stored as a single JSON blob in Redis and injected as a VALUES CTE; column governance (RLS/masking/visibility) is NOT applied at storage time but by Stage-2 apply_governance over the governed SQL that wraps the CTE (pipeline order: governance -> cache/CTE -> route). The per-row PK-hash/sorted-set storage in the spec was judged unnecessary because hot tables are tiny inline tables.

**Use case:** Centralised JSON blob storage with governance applied at query time simplifies cache maintenance and ensures all governance rules are enforced on hot data.

**Code:** `provisa/cache/`

**Tests:** `tests/integration/test_cache_store.py`, `tests/integration/test_hot_tables_real.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_hot_tables.py`, `tests/unit/test_mask_inject.py`

### REQ-231 · Hot Tables (Redis-Cached Lookups) {#REQ-231}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Hot table refresh follows MV pattern: TTL-based via `refresh_interval` (default: `materialized_views.default_ttl`), background refresh loop, stale fallback to live query. Mutations to the source table trigger immediate invalidation + async reload.

**Use case:** TTL-based hot table refresh keeps cached lookup data current without manual cache invalidation.

**Code:** `provisa/cache/`

**Tests:** `tests/integration/test_hot_tables_real.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_hot_tables.py`, `tests/unit/test_hot_tables_integration.py`

### REQ-232 · Hot Tables (Redis-Cached Lookups) {#REQ-232}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Hot table JOIN optimization -- when a query joins a hot table (e.g., `orders JOIN countries ON country_code`), the compiler injects the hot table data as constants into the SQL via a `VALUES`-based CTE. The DB engine sees literal rows, not a table reference. This eliminates the second table scan and works across sources (constants travel with the query to Trino/Snowflake/ClickHouse). SQLGlot transpiles the VALUES clause per dialect.

**Use case:** Hot table JOIN optimisation eliminates a second table scan by injecting lookup data as SQL constants.

**Code:** `provisa/cache/`

**Tests:** `tests/integration/test_hot_tables_real.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_hot_tables.py`, `tests/unit/test_hot_tables_integration.py`

### REQ-233 · Hot Tables (Redis-Cached Lookups) {#REQ-233}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Hot caching is bounded by both `max_rows` (own config default, falls back to `auto_threshold`) and `max_bytes` (default 10MB, measured after JSON serialization) — a wide table within `max_rows` but over the byte ceiling is not cached.

**Use case:** Dual byte and row limits prevent hot caching of unexpectedly wide tables that would exceed Redis memory budgets.

**Code:** `provisa/cache/`

**Tests:** `tests/integration/test_hot_tables_real.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_hot_tables.py`

### REQ-234 · Materialized View Lifecycle {#REQ-234}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Auto-materialized view storage reclamation -- when a view is removed from config, disabled, or its source table is unregistered, the backing MV table is dropped (Trino `DROP TABLE`). Background cleanup task runs on config reload and periodically (default: daily). Orphaned MV tables (present in target schema but not in MV registry) are flagged and optionally auto-dropped after a grace period.

**Use case:** MV storage reclamation prevents orphaned Iceberg tables accumulating when views are removed from config.

**Code:** `provisa/mv/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/unit/test_config_reload.py`, `tests/unit/test_mv_lifecycle.py`

### REQ-235 · Materialized View Lifecycle {#REQ-235}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Auto-materialized aggregate views must have a size guard -- `materialized_views.max_rows` (default: 1,000,000). Views whose source query would produce more rows than the limit skip materialization and fall back to live execution. Size estimated via `SELECT COUNT(*)` probe before CTAS. Configurable per view to override the global default.

**Use case:** MV size guard prevents materialising oversized views that would exhaust storage or exceed refresh budgets.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_mv_lifecycle.py`, `tests/integration/test_execution_routing.py`

### REQ-236 · Hot Table Auto-Detection {#REQ-236}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A table is auto-hot if it is the target of a many-to-one relationship OR its row count is <= `auto_threshold` (SELECT COUNT(*) at schema build via `detect_hot_tables_by_count`).

**Use case:** Auto-detection of small lookups and relationship targets caches them in Redis without explicit config.

**Code:** `provisa/cache/`

**Tests:** `tests/unit/test_hot_tables.py`, `tests/integration/test_hot_tables_real.py`

### REQ-237 · Hot Table Auto-Detection {#REQ-237}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Auto-hot tables can be opted out via `hot: false` on the table config. Explicit `hot: true` overrides the auto-detection criteria (forces caching regardless of row count or relationship status). Auto-detection runs on every schema rebuild; tables that grow beyond the threshold are automatically evicted from Redis on next rebuild.

**Use case:** Explicit hot:false opt-out lets stewards prevent specific tables from being auto-cached in Redis.

**Code:** `provisa/cache/`

**Tests:** `tests/unit/test_hot_tables.py`, `tests/integration/test_hot_tables_real.py`

### REQ-238 · Warm Tables (Replica Profile — Low-Latency Placement) {#REQ-238}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Warm tables -- frequently queried RDBMS tables materialized into a low-latency, engine-reachable location. In materialization_store terms ([REQ-844](#REQ-844)) this is the `warm` profile of the REPLICA role — an engine-relative low-latency PLACEMENT of a reachable source (distinct from `reactive`, which lands an unreachable source). The mechanism below is the current Trino realization of that placement rather than its definition — it materializes into the Iceberg results catalog so Trino's built-in file system cache (`fs.cache.enabled=true`) caches the Parquet files on local SSD; other engines realize the low-latency tier differently. Provides ~10-50ms reads vs 100ms+ network round-trip to remote source. Same TTL/refresh pattern as MVs.

**Use case:** Warm tables on local SSD reduce RDBMS round-trip latency from 100ms+ to ~10-50ms for frequent queries.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_catalog_cache.py`, `tests/unit/test_warm_tables.py`

### REQ-239 · Warm Tables (Replica Profile — Low-Latency Placement) {#REQ-239}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Warm table auto-promotion -- track query frequency per table (increment counter on each compiled query). Tables exceeding `warm_tables.query_threshold` (default: 100 queries per refresh interval) are auto-materialized into Iceberg. Tables falling below the threshold are demoted (backing table dropped) on next refresh cycle.

**Use case:** Auto-promotion of frequently queried tables to warm tier improves performance without manual configuration.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_warm_tables.py`, `tests/integration/test_execution_routing.py`

### REQ-240 · Warm Tables (Replica Profile — Low-Latency Placement) {#REQ-240}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Warm table config: `warm: true` to force, `warm: false` to opt out. Global `warm_tables.query_threshold`, `warm_tables.max_rows` (default: 10,000,000), `warm_tables.refresh_interval` (default: same as `materialized_views.default_ttl`). Trino file cache config (`fs.cache.enabled`, `fs.cache.directories`, `fs.cache.max-sizes`) managed in Trino catalog properties.

**Use case:** Warm table config guards (max_rows, threshold) prevent promotion of tables too large for local SSD cache.

**Code:** `provisa/mv/`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_catalog_cache.py`, `tests/unit/test_warm_tables.py`

### REQ-241 · Warm Tables (Replica Profile — Low-Latency Placement) {#REQ-241}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A table lives in at most one cache tier — hot wins over warm. `WarmTableManager.check_promotions` skips any table the hot tier manages (`HotTableManager.managed_tables()`).

**Use case:** Strict tier exclusivity prevents hot tables from being accidentally promoted to warm, ensuring optimal latency.

**Code:** `provisa/mv/`

**Tests:** `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_warm_tables.py`, `tests/integration/test_execution_routing.py`

### REQ-275 · Federation Performance {#REQ-275}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

On source registration, Provisa runs `ANALYZE` against the registered source's tables (where the connector supports it) to prime the federation engine's cost-based optimizer with baseline row counts and column statistics.

**Use case:** Running ANALYZE on registration primes the federation cost-based optimizer so join plans are efficient immediately.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-276 · Federation Performance {#REQ-276}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Admin API exposes a "Refresh Statistics" mutation per source that re-runs `ANALYZE` on demand. Useful for volatile sources where baseline stats have aged.

**Use case:** On-demand Refresh Statistics mutation lets stewards refresh cost stats for volatile sources without a restart.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-277 · Federation Performance {#REQ-277}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Per-query and per-table/view session property overrides: steward or developer can attach named session hints (`join_distribution_type`, `join_max_broadcast_table_size`, `join_reordering_strategy`). Provisa injects matching `SET SESSION` statements before execution.

**Use case:** Per-query session property hints let stewards tune federation join strategies for known expensive queries.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-278 · Federation Performance {#REQ-278}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Source-level default session properties: each registered source config accepts a `federation_hints:` block. Properties in that block are injected for any query that touches that source. Per-query hints override source-level defaults.

**Use case:** Source-level federation hints apply performance tuning to all queries touching a source without per-query config.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-279 · Federation Performance {#REQ-279}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Provisa-branded comment hint syntax `/*+ hint */` in query text. Supported hints: `BROADCAST(<table>)`, `NO_REORDER`, `BROADCAST_SIZE(<size>)`. Provisa parser strips the comment before forwarding SQL to the federation engine and translates to the equivalent session properties. The federation engine never sees the comment.

**Use case:** Provisa-branded hint syntax decouples query authors from Trino session property names and future engine changes.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-280 · Federation Performance {#REQ-280}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

ANALYZE runs on the API cache table after each materialization CTAS; ANALYZE failure is logged, not raised (connector tolerance, matching `analyze_source_tables` / [REQ-275](#REQ-275)).

**Use case:** ANALYZE after API cache CTAS keeps cost estimates fresh without failing the entire materialization on ANALYZE errors.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-281 · Federation Performance {#REQ-281}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Source-level `federation_hints` use the Provisa-branded @provisa vocabulary (`join=broadcast|partitioned`, `reorder=none|auto`, `broadcast_size=<size>`), translated to Trino session props by `provisa/compiler/directives.py:translate_federation_hints` at query time. Raw Trino session-prop keys still pass through (deprecated) for backward compatibility.

**Use case:** Source-level federation hints vocabulary is translated to Trino session props, keeping query authors decoupled from engine-specific property names.

**Code:** `provisa/executor/`, `provisa/compiler/`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/integration/test_execution_routing.py`, `tests/integration/test_federation_integration.py`, `tests/unit/test_compiler_requirements.py`, `tests/unit/test_federation.py`, `tests/unit/test_federation_hints.py`, `tests/unit/test_federation_v2_extended.py`

### REQ-397 · Execution & Routing {#REQ-397}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

When a PK is available, the exclusion WHERE clause must use `n.<pk_col> IN [<pk_value>]` instead of `id(n) IN [<nodeId>]`.

**Use case:** PK-based exclusion is semantic and stable across node re-materialization.

**Code:** `provisa-ui/src/components/graph/graph-model.ts`, `provisa/cypher/`

**Tests:** `tests/unit/test_graph_exclusion.py`, `tests/integration/test_execution_routing.py`

### REQ-536 · Execution & Routing {#REQ-536}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

All data responses include cache status headers: `X-Provisa-Cache: HIT|MISS` on every response, and `X-Provisa-Cache-Age: <seconds>` on cache HITs, indicating how many seconds old the cached result is.

**Use case:** Cache headers let clients and proxies distinguish fresh vs cached results and detect stale responses.

**Code:** `provisa/cache/middleware.py`

**Tests:** `tests/unit/test_cache_headers_requirements.py`, `tests/unit/test_misc_requirements.py`, `tests/unit/test_cache_headers_requirements.py`, `tests/integration/test_execution_routing.py`, `provisa-ui/e2e/security-rate-limiting.spec.ts`

### REQ-544 · Cache {#REQ-544}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cache TTL resolves in order: table-level TTL > source-level TTL > global default TTL. Setting `cache_enabled: false` on a source disables caching for all tables in that source regardless of table-level TTL. Cache keys always include `role_id` and RLS context values for security partitioning.

**Use case:** Hierarchical TTL lets operators tune cache freshness per source or per table while security-partitioned cache keys prevent cross-role result leakage.

**Code:** `provisa/cache/policy.py`, `provisa/cache/key.py`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_cache_policy.py`

### REQ-552 · Execution & Routing {#REQ-552}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cross-source JOINs routed through the federation engine apply automatic type coercion when joining columns across sources with differing native types.

**Use case:** Automatic type coercion during cross-source joins prevents type mismatch errors when federating tables from heterogeneous source systems.

**Code:** `provisa/transpiler/`, `provisa/executor/`

**Tests:** `tests/unit/test_routing.py`, `tests/e2e/test_routing.py`, `tests/integration/test_execution_routing.py`

### REQ-595 · Caching {#REQ-595}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

In multi-tenant mode, RedisCacheStore prefixes every cache key with the tenant_id (`provisa:cache:<tenant_id>:<sha256_key>`). Table invalidation keys follow `provisa:table:<tenant_id>:<table_id>`. The APQ cache uses `provisa:apq:<tenant_id>:<sha256_hex>`. Default APQ TTL is 86400 seconds, configurable via `PROVISA_APQ_TTL`. One tenant's cache flush never touches another tenant's cached results. check_cache and store_result in cache/middleware.py pass tenant_id (org_id) through to RedisCacheStore.get/set, so prefix isolation is active for query result caching.

**Use case:** Tenant-prefixed cache keys guarantee that cache invalidation, eviction, and key collisions cannot cross tenant boundaries on shared Redis infrastructure.

**Code:** `provisa/cache/store.py`, `provisa/cache/middleware.py`, `provisa/apq/cache.py`

**Tests:** `tests/e2e/test_caching.py`, `tests/integration/test_cache_store.py`, `tests/integration/test_execution_routing.py`, `tests/unit/test_cache_requirements.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_config_validation_edge_cases.py`, `tests/unit/test_mv_tenant_isolation.py`

## 7. Result Delivery

### REQ-047 · Output & Delivery {#REQ-047}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

JSON output preserves native GraphQL nested structure.

**Use case:** Native GraphQL JSON output preserves nested relationships, making response structure self-explanatory.

**Code:** `provisa/executor/`

**Tests:** `tests/unit/test_formats.py`, `tests/integration/test_output_formats.py`

### REQ-048 · Output & Delivery {#REQ-048}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

NDJSON streaming variant: one JSON object per line.

**Use case:** NDJSON streaming enables line-by-line processing of large results without loading the full payload into memory.

**Code:** `provisa/executor/`

**Tests:** `tests/unit/test_formats.py`, `tests/integration/test_output_formats.py`

### REQ-049 · Output & Delivery {#REQ-049}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Normalized tabular: flattened to relational tables with FK relationships preserved, Parquet or CSV.

**Use case:** Normalized tabular output with FK preservation lets BI tools load data into relational models without ETL.

**Code:** `provisa/executor/`

**Tests:** `tests/integration/test_output_formats.py`, `tests/unit/test_formats.py`, `tests/unit/test_normalize.py`, `tests/unit/test_normalized_endpoint.py`

### REQ-050 · Output & Delivery {#REQ-050}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Denormalized tabular: fully flattened single table, Parquet or CSV, single file or partitioned.

**Use case:** Denormalized flat-file output lets data science consumers load results directly into dataframes or ML pipelines.

**Code:** `provisa/executor/`

**Tests:** `tests/unit/test_formats.py`, `tests/integration/test_output_formats.py`

### REQ-051 · Output & Delivery {#REQ-051}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Arrow buffer via gRPC Arrow Flight endpoint; Trino produces Arrow natively.

**Use case:** Native Arrow Flight delivery eliminates serialization overhead for high-throughput analytics consumers.

**Code:** `provisa/executor/`

**Tests:** `tests/unit/test_formats.py`, `tests/integration/test_output_formats.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-137 · Large Result Redirect & CTAS {#REQ-137}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Client-controlled redirect via `X-Provisa-Redirect-Format` and `X-Provisa-Redirect-Threshold` headers. Format without threshold implies force redirect.

**Use case:** Client-controlled redirect format and threshold lets high-volume consumers choose their preferred output path.

**Code:** `provisa/executor/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_blob_upload.py`, `tests/unit/test_redirect.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-138 · Large Result Redirect & CTAS {#REQ-138}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Trino-native formats (Parquet, ORC) use CTAS — Trino writes directly to S3 via Iceberg, data never passes through Provisa.

**Use case:** Trino-native CTAS redirect lets Trino write Parquet and ORC directly to S3 without data passing through Provisa.

**Code:** `provisa/executor/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_blob_upload.py`, `tests/unit/test_redirect.py`

### REQ-139 · Large Result Redirect & CTAS {#REQ-139}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Non-native formats (JSON, NDJSON, CSV, Arrow IPC) serialized by Provisa and uploaded to S3 via boto3.

**Use case:** Provisa-side serialization handles non-native formats for consumers that need JSON, CSV, or Arrow IPC on S3.

**Code:** `provisa/executor/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_blob_upload.py`, `tests/unit/test_redirect.py`

### REQ-140 · Large Result Redirect & CTAS {#REQ-140}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Threshold-based redirect uses LIMIT threshold+1 probe — no COUNT(*), no double execution for inline results.

**Use case:** Threshold probe via LIMIT avoids double execution overhead when deciding whether to redirect large results.

**Code:** `provisa/executor/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_blob_upload.py`, `tests/integration/test_direct_exec.py`, `tests/unit/test_redirect.py`

### REQ-141 · Large Result Redirect & CTAS {#REQ-141}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

S3 data cleanup scheduled after presigned URL TTL expires.

**Use case:** Scheduled S3 data cleanup prevents indefinite storage cost accumulation from expired redirect results.

**Code:** `provisa/executor/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_blob_upload.py`, `tests/unit/test_redirect.py`

### REQ-142 · Large Result Redirect & CTAS {#REQ-142}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Default redirect format configurable via `PROVISA_REDIRECT_FORMAT` (default: parquet).

**Use case:** Configurable default redirect format lets operators choose the storage format best suited to their consumer mix.

**Code:** `provisa/executor/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_blob_upload.py`, `tests/unit/test_redirect.py`

### REQ-143 · Arrow Flight {#REQ-143}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Arrow Flight server (port 8815) streams record batches via gRPC. Full security pipeline applied.

**Use case:** Arrow Flight server streams record batches with full security applied, enabling zero-copy governed data delivery.

**Code:** `provisa/api/flight/`

**Tests:** `tests/integration/test_arrow_flight_integration.py`, `tests/unit/test_flight_server_helpers.py`, `tests/unit/test_zaychik_flight_unit.py`, `provisa-ui/e2e/execution-routing.spec.ts`

### REQ-144 · Arrow Flight {#REQ-144}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Zaychik Arrow Flight SQL proxy translates between Flight SQL clients and Trino JDBC.

**Use case:** Zaychik Flight SQL proxy lets Flight SQL clients query Trino without requiring a native Trino Flight endpoint.

**Code:** `provisa/api/flight/`

**Tests:** `tests/integration/test_arrow_flight_integration.py`, `tests/unit/test_flight_server_helpers.py`, `tests/unit/test_zaychik_flight_unit.py`

### REQ-145 · Arrow Flight {#REQ-145}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Flight server streams batch-by-batch via GeneratorStream — full result never materialized in Provisa memory. Unbounded result support.

**Use case:** GeneratorStream batch delivery means arbitrarily large result sets never fully materialize in Provisa memory.

**Code:** `provisa/api/flight/`

**Tests:** `tests/integration/test_arrow_flight_integration.py`, `tests/unit/test_flight_server_helpers.py`, `tests/unit/test_zaychik_flight_unit.py`

### REQ-146 · Arrow Flight {#REQ-146}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Falls back to materializing via Trino REST if Zaychik unavailable.

**Use case:** Fallback to Trino REST ensures Flight delivery works even when the Zaychik proxy is unavailable.

**Code:** `provisa/api/flight/`

**Tests:** `tests/integration/test_arrow_flight_integration.py`, `tests/unit/test_flight_server_helpers.py`, `tests/unit/test_zaychik_flight_unit.py`

## 8. Client Access & Protocols

### REQ-043 · API & Integration {#REQ-043}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

GraphQL endpoint is primary entry point for queries and mutations.

**Use case:** GraphQL as the primary entry point gives consumers a strongly typed, self-documenting API surface.

**Code:** `provisa/api/rest/`, `provisa/api/jsonapi/`, `provisa/api/flight/`

**Tests:** `tests/unit/test_rest_generator.py`, `tests/unit/test_jsonapi.py`, `tests/integration/test_client_access_integration.py`

### REQ-044 · API & Integration {#REQ-044}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Presigned URL redirect for large result consumers with TTL-bounded access.

**Use case:** Presigned URL redirect gives large-result consumers time-bounded, secure access without server-side buffering.

**Code:** `provisa/api/rest/`, `provisa/api/jsonapi/`, `provisa/api/flight/`

**Tests:** `tests/e2e/test_large_result.py`, `tests/integration/test_client_access_integration.py`, `tests/unit/test_jsonapi.py`, `tests/unit/test_redirect_s3.py`, `tests/unit/test_rest_generator.py`

### REQ-045 · API & Integration {#REQ-045}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

gRPC Arrow Flight endpoint for high-throughput consumers; Trino produces Arrow natively for zero-copy delivery.

**Use case:** Arrow Flight endpoint enables high-throughput consumers to stream data with zero-copy native Arrow delivery.

**Code:** `provisa/api/rest/`, `provisa/api/jsonapi/`, `provisa/api/flight/`

**Tests:** `tests/unit/test_rest_generator.py`, `tests/unit/test_jsonapi.py`, `tests/integration/test_arrow_flight_integration.py`

### REQ-126 · JDBC/ODBC Integration {#REQ-126}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

JDBC driver that exposes registered tables and views as virtual tables. Connection authenticates against Provisa, maps user to role.

**Use case:** JDBC driver exposes registered tables/views as virtual tables, enabling any JDBC-compatible tool to query governed data.

**Code:** `provisa-client/`

**Tests:** `tests/unit/test_drivers.py`, `tests/unit/test_flight_modes.py`, `tests/integration/test_arrow_flight_integration.py`, `tests/integration/test_client_access_integration.py`

### REQ-127 · JDBC/ODBC Integration {#REQ-127}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`getTables()` returns registered tables and views visible to the authenticated role, by their registered names.

**Use case:** getTables() returning registered tables/views lets BI tools discover governed datasets through standard JDBC metadata.

**Code:** `provisa-client/`

**Tests:** `tests/unit/test_drivers.py`

### REQ-128 · JDBC/ODBC Integration {#REQ-128}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`getColumns(tableName)` introspects the registered table/view output schema — column names and types from compiled metadata, filtered by role visibility.

**Use case:** getColumns() introspection lets JDBC tools render correct column types without manual schema configuration.

**Code:** `provisa-client/`

**Tests:** `tests/integration/test_pgwire_column_visibility_integration.py`, `tests/unit/test_drivers.py`, `tests/unit/test_view_column_types.py`

### REQ-129 · JDBC/ODBC Integration {#REQ-129}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`executeQuery(sql)` runs arbitrary SQL against registered tables/views, passes it through Stage 2 governance, executes via Provisa's HTTP API, and deserializes the result into a JDBC ResultSet. (Amended 2026-06-19: transport is Arrow IPC / JSON, not Parquet — Arrow IPC streams a ResultSet far better than a columnar file format, which would buffer the whole result.)

**Use case:** executeQuery lets JDBC tools run governed SQL against registered tables/views using standard SQL syntax.

**Code:** `provisa-client/`

**Tests:** `tests/unit/test_drivers.py`

### REQ-130 · JDBC/ODBC Integration {#REQ-130}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Full security pipeline (RLS, masking, sampling) applied at query time — not baked into views.

**Use case:** Full security pipeline at JDBC query time ensures BI tool users receive only data their role permits.

**Code:** `provisa-client/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/unit/test_drivers.py`

### REQ-131 · JDBC/ODBC Integration {#REQ-131}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Connection string format: `jdbc:provisa://host:port`. Authentication via standard JDBC username/password properties.

**Use case:** Standard JDBC connection string format means no custom driver configuration is needed in most BI tools.

**Code:** `provisa-client/`

**Tests:** `tests/unit/test_drivers.py`

### REQ-132 · JDBC/ODBC Integration {#REQ-132}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The driver ships as a single self-contained shaded JAR. (Amended 2026-06-19: the Flight transport ([REQ-293](#REQ-293)) requires gRPC/Netty, so "JDK + Arrow only" is not achievable; instead all bundled third-party packages — Gson, gRPC, Netty, protobuf, guava — are shade-relocated under `io.provisa.shaded.*` so they never collide with a host application's own copies. The JDBC SPI and Apache Arrow remain at their original packages.)

**Use case:** Single self-contained JAR eliminates dependency conflicts when deploying the driver in enterprise environments.

**Code:** `provisa-client/`

**Tests:** `tests/unit/test_drivers.py`

### REQ-161 · Query Development Tools {#REQ-161}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`POST /data/compile` returns compiled SQL with RLS/masking applied, route decision, and params without executing.

**Use case:** compile endpoint lets developers inspect governed SQL and routing before submitting a query for approval.

**Code:** `provisa/api/rest/`, `provisa/api/data/`

**Tests:** `tests/integration/test_compile_endpoint.py`, `tests/unit/test_client_access.py`, `tests/unit/test_compile_route.py`

### REQ-163 · Query Development Tools {#REQ-163}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

GraphiQL Provisa plugin with View SQL (compiled governed SQL preview).

**Use case:** GraphiQL plugin with View SQL integrates governed-SQL inspection into the IDE.

**Code:** `provisa/api/rest/`, `provisa/api/data/`

**Tests:** `tests/integration/test_compile_endpoint.py`, `tests/unit/test_client_access.py`

### REQ-256 · API & Integration {#REQ-256}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Auto-generated plain REST endpoints for every registered table via `GET /data/rest/{table}` with query string mapping to GraphQL args (`?limit=10&where.id.eq=1`). Compiles and executes via existing pipeline (RLS, masking, routing) with same security as GraphQL.

**Use case:** Auto-generated plain REST endpoints let REST-only tools query governed data without learning GraphQL.

**Code:** `provisa/api/rest/`, `provisa/api/jsonapi/`, `provisa/api/flight/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_client_access_integration.py`, `tests/unit/test_jsonapi.py`, `tests/unit/test_rest_generator.py`

### REQ-257 · API & Integration {#REQ-257}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Auto-generated JSON:API compliant endpoints for every registered table via `GET /data/jsonapi/{table}` following spec (jsonapi.org): resource objects with type/id/attributes/relationships, sparse fieldsets (`?fields[orders]=amount`), inclusion of related resources (`?include=customer`), filtering (`?filter[region]=US`), sorting (`?sort=-created_at`), pagination (`?page[number]=2&page[size]=25`), compound documents. Compiles and executes via existing pipeline.

**Use case:** JSON:API compliant endpoints let clients use standard JSON:API tooling (sparse fields, includes, pagination).

**Code:** `provisa/api/rest/`, `provisa/api/jsonapi/`, `provisa/api/flight/`

**Tests:** `tests/integration/test_jsonapi_integration.py`, `tests/unit/test_jsonapi.py`, `tests/unit/test_jsonapi_include.py`, `tests/unit/test_rest_generator.py`

### REQ-258 · API & Integration {#REQ-258}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SSE subscriptions via `GET /data/subscribe/{table}` with pluggable notification providers per source type. PostgreSQL uses LISTEN/NOTIFY via asyncpg, MongoDB uses Change Streams via motor collection.watch(), Kafka uses consumer groups. Each provider implements a common async watch() interface returning change events. RLS filtering and schema validation apply regardless of provider.

**Use case:** Pluggable SSE subscription providers let each source type (PG, Mongo, Kafka) use its native change mechanism.

**Code:** `provisa/api/rest/`, `provisa/api/jsonapi/`, `provisa/api/flight/`

**Tests:** `tests/integration/test_client_access_integration.py`, `tests/unit/test_jsonapi.py`, `tests/unit/test_rest_generator.py`, `tests/unit/test_subscription_providers.py`

### REQ-268 · SQL & Multi-Protocol Client Access {#REQ-268}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Python DB-API 2.0 (PEP 249) interface in `provisa-client`. `provisa_client.connect(url, user, password)` returns a PEP 249 connection. `cursor.execute(query)` accepts GraphQL (detected by leading `{` or `query`/`mutation` keyword) or SQL. Authenticates via username/password; server assigns role.

**Use case:** DB-API 2.0 interface lets Python code query Provisa with standard cursor.execute() without a custom client.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_client_role.py`, `tests/unit/test_proto_gen.py`, `tests/unit/test_protocol_clients.py`

### REQ-269 · SQL & Multi-Protocol Client Access {#REQ-269}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

DB-API 2.0 connection exposes all registered tables and views the user's rights permit for arbitrary SQL. There is no connection `mode` parameter — governance is uniform via rights + Stage 2.

**Use case:** Uniform rights-governed SQL access in DB-API removes mode selection; every query is governed identically.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`

### REQ-270 · SQL & Multi-Protocol Client Access {#REQ-270}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SQLAlchemy dialect for Provisa. `create_engine("provisa+http://user:password@host:8001")`. Dialect maps SQLAlchemy Core expressions to Provisa SQL. `engine.connect()` returns a DB-API 2.0 connection. `pandas.read_sql(query, engine)` works out of the box. `inspector.get_table_names()` returns registered table and view names the role may access.

**Use case:** SQLAlchemy dialect lets pandas and ORM users read governed data with standard SQL tooling and no custom code.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`

### REQ-271 · SQL & Multi-Protocol Client Access {#REQ-271}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

ADBC (Arrow Database Connectivity) interface in `provisa-client`. `provisa_client.adbc_connect(url, user, password)` returns an ADBC connection using Provisa's Arrow Flight endpoint as transport. Results stream as Arrow RecordBatches natively. Compatible with `adbc_driver_manager` and `pandas.read_sql` via ADBC. Authenticates via username/password.

**Use case:** ADBC interface streams Arrow RecordBatches natively so analytics tools get zero-copy columnar data.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`

### REQ-272 · SQL & Multi-Protocol Client Access {#REQ-272}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

JDBC driver runs arbitrary SQL against registered tables and views with full Stage 2 governance — RLS, masking, sampling, and ceilings applied on every query. There is no ungoverned access path.

**Use case:** Uniform Stage 2 governance in JDBC ensures arbitrary SQL from BI tools always has RLS and masking applied.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`

### REQ-273 · SQL & Multi-Protocol Client Access {#REQ-273}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

All clients (DB-API, SQLAlchemy, ADBC, JDBC) authenticate via username/password. Server assigns role from configured auth provider. No client-supplied role parameter accepted or required.

**Use case:** Server-assigned roles prevent clients from escalating privileges by supplying their own role parameter.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_client_role.py`, `tests/unit/test_proto_gen.py`, `tests/unit/test_protocol_clients.py`

### REQ-274 · SQL & Multi-Protocol Client Access {#REQ-274}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Query language selection is per-call for DB-API and GraphQL clients: pass a GraphQL string to execute via Stage 1+2; pass a SQL string to execute via Stage 2 only. ADBC and SQLAlchemy always use SQL (Stage 2 only). JDBC uses SQL always.

**Use case:** Per-call query language selection lets the same client send GraphQL or SQL based on what it has available.

**Code:** `provisa-client/`, `provisa/api/`

**Tests:** `tests/integration/test_adbc.py`, `tests/integration/test_sqlalchemy_dialect.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`

### REQ-288 · Automatic Persisted Queries (APQ) {#REQ-288}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa implements the Apollo APQ wire protocol (GraphQL over HTTP with `extensions.persistedQuery.sha256Hash`). Client sends hash only; if server has it, executes without the query text. If server does not have it, returns `PersistedQueryNotFound`; client resends with full query text and hash; server stores and executes. Standard Apollo client behaviour works without modification.

**Use case:** APQ wire protocol support lets Apollo clients reduce bandwidth by sending only a hash after the first request.

**Code:** `provisa/apq/`

**Tests:** `tests/integration/test_abac_hook_endpoint.py`, `tests/integration/test_apq_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_apq.py`, `tests/unit/test_registry_removed.py`

### REQ-289 · Automatic Persisted Queries (APQ) {#REQ-289}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

APQ cache stored in Redis (existing `cache.redis_url`). TTL configurable via `apq.ttl` (default: 24h). Cache miss on cold start is expected and handled by the standard APQ retry flow — no preloading required.

**Use case:** Redis APQ cache with configurable TTL reuses query text across process restarts without preloading.

**Code:** `provisa/apq/`

**Tests:** `tests/integration/test_apq_integration.py`, `tests/integration/test_cache_store.py`, `tests/unit/test_apq.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_config_reload.py`

### REQ-290 · Automatic Persisted Queries (APQ) {#REQ-290}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

APQ applies to any query the authenticated caller's rights permit. APQ is fully automatic — any successfully executed query is registered in the APQ cache and reusable by hash on subsequent calls. No steward involvement.

**Use case:** Automatic APQ registration lets clients benefit from hash-based execution with no config.

**Code:** `provisa/apq/`

**Tests:** `tests/unit/test_apq.py`, `tests/integration/test_apq_integration.py`

### REQ-291 · Automatic Persisted Queries (APQ) {#REQ-291}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

APQ registration is gated only by the standard rights + Stage 2 governance check. A query referencing tables or columns the caller's rights do not permit is rejected at the governance check — before it can be registered in the APQ cache. Only queries that pass governance can be cached.

**Use case:** Rights-gated APQ registration prevents hash-only execution of queries the caller is not permitted to run.

**Code:** `provisa/apq/`

**Tests:** `tests/unit/test_apq.py`, `tests/integration/test_apq_integration.py`

### REQ-293 · JDBC/ODBC Integration {#REQ-293}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

JDBC driver transport via Arrow Flight — connect to Provisa's existing Flight server (`grpc://host:8815`) for streaming query results. Arrow record batches stream from the first row with backpressure, zero serialization overhead, and no full-result buffering. Flight is used automatically when the server is reachable; falls back to HTTP silently if not. No user configuration required. The Flight ticket carries the SQL or GraphQL query + role + variables as JSON; Stage 2 governance applies uniformly.

**Use case:** JDBC transport via Arrow Flight streams results with backpressure so BI tools handle large result sets safely.

**Code:** `provisa-client/`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/unit/test_drivers.py`, `tests/integration/test_client_access_integration.py`

### REQ-398 · API & Integration {#REQ-398}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The `/data/graph-schema` REST endpoint must expose `pk_columns` (list of column names per node label) so the UI can determine exclusion eligibility.

**Use case:** Schema endpoint pk_columns expose lets UI conditionally enable/disable "Exclude from query" per node.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `tests/unit/test_graph_schema.py`, `tests/integration/test_client_access_integration.py`

### REQ-405 · API & Integration {#REQ-405}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

`graphql_api` and `graphql_remote` source sub-types are collapsed into a single `graphql` type in the SourcesPage UI; `grpc_api` and `grpc_remote` are collapsed into a single `grpc` type.

**Use case:** Collapsing API/remote variants eliminates redundant form paths and simplifies steward source registration.

**Code:** `provisa-ui/src/pages/SourcesPage.tsx`

**Tests:** `tests/integration/test_client_access_integration.py`, `tests/integration/test_graphql_remote_source.py`, `tests/unit/test_admin_requirements.py`

### REQ-406 · API & Integration {#REQ-406}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

OpenAPI source registration supports an inline spec editor (monospace textarea) as an alternative to providing a file path or URL; the mode is selected via a radio toggle "Spec path / URL" vs "Write spec inline". Inline content is sent as `spec_content`; file/URL path is sent as `spec_path`.

**Use case:** Inline spec editing lets stewards paste a spec directly into the UI without first writing it to a file.

**Code:** `provisa-ui/src/pages/SourcesPage.tsx`, `provisa/api/admin/openapi_router.py`

**Tests:** `tests/unit/test_admin_requirements.py`, `tests/integration/test_client_access_integration.py`

### REQ-407 · API & Integration {#REQ-407}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

OpenAPI source backend accepts optional `spec_content: str` on `OpenAPIRegisterRequest` and `OpenAPIPreviewRequest`; when provided it is parsed (YAML then JSON fallback) and used in place of loading from disk; `path` is stored as `":inline:"` sentinel.

**Use case:** Backend inline-spec support decouples OpenAPI registration from file-system access on the server.

**Code:** `provisa/api/admin/openapi_router.py`

**Tests:** `tests/unit/test_admin_requirements.py`, `tests/integration/test_client_access_integration.py`

### REQ-408 · API & Integration {#REQ-408}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

OpenAPI operations can carry `x-provisa-kind: query` or `x-provisa-kind: mutation` extension to override the GET-heuristic that classifies POST operations as mutations by default. Allows POST-as-read endpoints to be exposed as GraphQL queries.

**Use case:** `x-provisa-kind` override lets stewards correctly classify non-standard POST read operations without changing the upstream API.

**Code:** `provisa/openapi/mapper.py`

**Tests:** `tests/unit/test_admin_requirements.py`, `tests/integration/test_client_access_integration.py`

### REQ-527 · pgwire Server {#REQ-527}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

pgwire server is disabled by default and starts only when `PROVISA_PGWIRE_PORT` environment variable is set to a non-zero integer. Binds to `0.0.0.0` on the configured port.

**Use case:** Disabled-by-default pgwire prevents accidental exposure of the PostgreSQL wire protocol endpoint when it is not needed.

**Code:** `provisa/api/app.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-529 · pgwire Server {#REQ-529}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

pgwire authentication uses PG auth type 3 (cleartext password) for both trust and simple modes. Trust mode: username becomes role_id, password ignored. Simple mode: password verified via bcrypt through SimpleAuthProvider. Any `provider` value other than `none` or `simple` returns a FATAL login error.

**Use case:** Restricting pgwire auth to trust or simple providers prevents misconfiguration with OIDC or Firebase providers that require HTTP redirect flows incompatible with the wire protocol.

**Code:** `provisa/pgwire/server.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-530 · pgwire Server {#REQ-530}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

pgwire TLS is enabled by setting `PROVISA_PGWIRE_CERT` and `PROVISA_PGWIRE_KEY` to PEM certificate and key paths. When both are set the server wraps connections in TLS. When absent the server replies `N` to SSL negotiation requests.

**Use case:** Optional TLS for pgwire allows encrypted connections in production without requiring TLS in local development.

**Code:** `provisa/pgwire/server.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-532 · pgwire Server {#REQ-532}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Queries against `information_schema` and `pg_catalog` are intercepted and answered from an in-memory DuckDB database built per request from the role's compilation context — no Trino round-trip occurs. Intercepted tables include `pg_namespace`, `pg_class`, `pg_attribute`, `pg_type`, `pg_constraint`, `pg_roles`, `pg_settings`, `pg_stat_activity`, `schemata`, `tables`, `columns`, `views`, `table_constraints`, `key_column_usage`, `referential_constraints`, and others. Stubs return zero rows.

**Use case:** Local catalog intercept allows BI tools (DBeaver, Tableau, JDBC schema browsers) to populate their schema tree from `pg_catalog` without generating Trino queries.

**Code:** `provisa/pgwire/catalog.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_catalog.py`, `tests/unit/pgwire/test_phase3.py`, `tests/unit/pgwire/test_phase4.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-538 · JDBC/ODBC Integration {#REQ-538}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The auto-generated `.proto` file maps Provisa/Trino column types to protobuf scalar types: `integer → int32`, `bigint → int64`, `varchar → string`, `decimal → double`, `boolean → bool`, `timestamp → google.protobuf.Timestamp`. Each registered table produces one proto `message`. Relationships between tables produce nested message fields.

**Use case:** Documented proto type mapping lets gRPC clients generate correct typed stubs without manual type translation.

**Code:** `provisa/grpc/proto_gen.py`

**Tests:** `tests/unit/test_grpc_requirements.py`

### REQ-579 · pgwire Server {#REQ-579}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The pgwire server reports server version `14.0.provisa` to connecting clients. Tools that gate features on the PostgreSQL version number behave as though connected to PostgreSQL 14.

**Use case:** Reporting a stable PostgreSQL version ensures JDBC drivers, DBeaver, and other tools that gate features on version behave predictably.

**Code:** `provisa/pgwire/server.py`, `provisa/pgwire/catalog.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-580 · pgwire Server {#REQ-580}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

pgwire simple-query protocol supports multi-statement queries. Semicolon-separated statements are split and executed sequentially within a single simple-query message.

**Use case:** Multi-statement simple queries allow JDBC tools and psql scripts that send batched DDL or DML to work without modification.

**Code:** `provisa/pgwire/server.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-581 · pgwire Server {#REQ-581}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Parameterized queries using `$1`, `$2`, ... positional parameters are supported in both simple-query and extended-query (Bind/Execute) protocol modes. Parameters are substituted as SQL literals before execution — the upstream engine never sees a prepared statement.

**Use case:** Parameterized query support enables standard JDBC/psycopg2 prepared statement patterns. Literal substitution avoids prepared-statement protocol complexity with Trino.

**Code:** `provisa/pgwire/server.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_phase2.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-582 · pgwire Server {#REQ-582}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

pgwire DDL dispatches to one of two paths based on `ddl_catalog`. Trino path (non-registered catalog such as `iceberg`, `hive`, `otel`, `results`): supports only `CREATE TABLE` and `CREATE VIEW`; ALTER, DROP, and CREATE INDEX are rejected. Direct path (registered source ID): supports full DDL including CREATE, ALTER, DROP, indexes, and sequences.

**Use case:** Two DDL paths let stewards write tables to Iceberg for lake storage or directly to registered RDBMS sources for full DDL control.

**Code:** `provisa/pgwire/ddl_handler.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-583 · pgwire Server {#REQ-583}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

After DDL execution on either the Trino or direct path, the new table is registered into the role's compilation context so it is immediately queryable without a server restart.

**Use case:** Immediate post-DDL registration lets analytic workflows create tables and query them in the same session.

**Code:** `provisa/pgwire/ddl_handler.py`

**Tests:** `tests/integration/test_direct_exec.py`, `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-584 · pgwire Server {#REQ-584}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

DDL write target is resolved from the domain's `ddl_catalog` and `ddl_schema` config fields. If `ddl_catalog` is unset, the system defaults to the Iceberg catalog. If `ddl_schema` is unset, it defaults to the domain ID. The domain is resolved through the role's `domain_access` list.

**Use case:** Domain-driven DDL target resolution ensures DDL writes land in the correct catalog and schema without per-query configuration.

**Code:** `provisa/pgwire/ddl_handler.py`, `provisa/api/app.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-585 · pgwire Server {#REQ-585}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

pgwire supports `COPY ... TO STDOUT` (export) and `COPY ... FROM STDIN` (import). COPY TO STDOUT supports both table reference and arbitrary subquery forms. Supported formats for COPY output: `text` (tab-delimited, default) and `csv`. Binary COPY format is not supported.

**Use case:** COPY protocol support enables bulk data export and import via psql, pgdump-compatible tools, and JDBC copy managers without requiring the HTTP API.

**Code:** `provisa/pgwire/copy_handler.py`

**Tests:** `tests/e2e/test_output_formats.py`, `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-586 · pgwire Server {#REQ-586}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

`COPY ... FROM STDIN` is restricted to sources with types `postgresql`, `mysql`, `sqlite`, or `mariadb`. Attempting COPY FROM against a Trino-only source raises SQLSTATE 42501. If no column list is provided, columns are inferred from the registered schema.

**Use case:** Restricting COPY FROM to writable source types prevents accidental writes to read-only Iceberg or Trino-federated tables.

**Code:** `provisa/pgwire/copy_handler.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-587 · pgwire Server {#REQ-587}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Transaction control commands (SET, BEGIN, START TRANSACTION, COMMIT, ROLLBACK, SAVEPOINT, RELEASE, DISCARD, RESET, DEALLOCATE) are intercepted and return an empty success response. The server is stateless with respect to transactions — `in_transaction()` always returns False; there is no transaction isolation or rollback support.

**Use case:** Accepting but ignoring transaction commands allows JDBC drivers and ORMs that issue implicit BEGIN/COMMIT to function without errors, while making the stateless nature explicit.

**Code:** `provisa/pgwire/server.py`, `provisa/pgwire/catalog.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_phase2.py`, `tests/unit/pgwire/test_server.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-588 · pgwire Server {#REQ-588}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The catalog intercept layer handles scalar expression queries: `current_user` / `session_user` → authenticated `role_id`; `current_database()` → `"provisa"`; `current_schema()` → `"public"`; `version()` → `"PostgreSQL 14.0 on Provisa"`; `pg_backend_pid()` → `0`; `current_setting(...)` → value from a fixed settings table; `SHOW <setting>` → value from the same settings table.

**Use case:** Scalar expression intercepts prevent JDBC drivers and ORMs from failing on startup queries that probe server identity and settings.

**Code:** `provisa/pgwire/catalog.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_phase2.py`, `tests/unit/pgwire/test_phase4.py`, `tests/unit/pgwire/test_server.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-589 · pgwire Server {#REQ-589}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The extended-query protocol (Bind/Execute) supports binary-encoded parameters. Supported OIDs for binary decode: 16 (bool), 17 (bytea), 20 (int8), 21 (int2), 23 (int4), 25 (text), 700 (float4), 701 (float8), 1043 (varchar), 1082 (date), 1114 (timestamp), 1184 (timestamptz), 1700 (numeric), 2950 (uuid). Unsupported OIDs raise `"Unsupported binary parameter type: <oid>"`. Result columns are also sent in binary when the client requests it.

**Use case:** Binary parameter encoding support enables psycopg2 and asyncpg to send typed parameters without text serialization, preserving full type fidelity for numeric and temporal values.

**Code:** `vendor/buenavista/buenavista/postgres.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-590 · pgwire Server {#REQ-590}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

pgwire enforces hard-coded timeouts: DDL operations time out after 60 seconds; query operations time out after 120 seconds. These are enforced in the handler threads via `future.result(timeout=N)`.

**Use case:** Hard-coded timeouts prevent long-running DDL or query operations from blocking pgwire handler threads indefinitely.

**Code:** `provisa/pgwire/ddl_handler.py`, `provisa/pgwire/server.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-606 · SQL & Multi-Protocol Client Access {#REQ-606}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

ProvisaClient accepts a bearer token (`token` parameter) as an alternative to username/password for authentication. When a token is present, it is sent as `Authorization: Bearer <token>` on every request. When omitted, the client relies on password-based login or no auth (unsecured deployment).

**Use case:** Bearer token support lets ProvisaClient integrate with identity systems that issue tokens directly, without requiring a Provisa-local username/password pair.

**Code:** `provisa-client/provisa_client/client.py`

**Tests:** `provisa-client/tests/test_client.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`, `tests/integration/test_client_access_integration.py`

### REQ-607 · SQL & Multi-Protocol Client Access {#REQ-607}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

ProvisaClient error contract: `query()` raises `httpx.HTTPStatusError` on HTTP-level errors (4xx/5xx). `query_df()` raises `RuntimeError` when the GraphQL response body contains an `errors` field, regardless of HTTP status.

**Use case:** Distinct error types for transport errors vs. GraphQL errors let callers handle network failures and schema/data errors with separate recovery paths.

**Code:** `provisa-client/provisa_client/client.py`

**Tests:** `provisa-client/tests/test_client.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`, `tests/integration/test_client_access_integration.py`

### REQ-608 · SQL & Multi-Protocol Client Access {#REQ-608}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The ADBC interface (`adbc_connect`) connects to the Arrow Flight server on port 8815. The Flight port is hardcoded in `adbc_connect` and is not configurable via any parameter — callers must run the Flight server on port 8815 or reconfigure the server.

**Use case:** Hardcoded Flight port in adbc_connect simplifies the ADBC connection signature; the server config is the single place to change the port.

**Code:** `provisa-client/provisa_client/adbc.py`

**Tests:** `provisa-client/tests/test_adbc.py`, `tests/unit/test_client_access.py`, `tests/unit/test_proto_gen.py`, `tests/integration/test_client_access_integration.py`

### REQ-614 · pgwire {#REQ-614}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The pgwire listener accepts only SQL statements. GraphQL and Cypher query strings are not parsed or executed over the pgwire protocol.

**Use case:** SQL-only restriction on pgwire keeps the protocol surface minimal and avoids ambiguity between query languages on a single connection.

**Code:** `provisa/pgwire/_pipeline.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_cypher_graph_fns.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-615 · pgwire {#REQ-615}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The pgwire listener does not support DML mutations (INSERT, UPDATE, DELETE). SQL statements of these types are not routed to a write path.

**Use case:** Read-only restriction on pgwire keeps the connection appropriate for analytics tools and prevents accidental data modification via SQL clients.

**Code:** `provisa/pgwire/_pipeline.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-616 · pgwire {#REQ-616}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

COPY and DDL statements submitted over pgwire require the `ddl` role capability. Roles without `ddl` receive a permission error (PG error code 42501). Roles with `ddl` have COPY and DDL executed via the DDL handler and COPY handler respectively.

**Use case:** Capability-gated DDL and COPY lets power users perform schema operations via psql or DBeaver while protecting other roles from accidental destructive statements.

**Code:** `provisa/pgwire/server.py`, `provisa/pgwire/ddl_handler.py`, `provisa/pgwire/copy_handler.py`

**Tests:** `tests/integration/test_pgwire_integration.py`, `tests/unit/pgwire/test_wire_protocol.py`, `tests/unit/test_pgwire_requirements.py`

### REQ-617 · gRPC {#REQ-617}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Role selection on every Provisa gRPC RPC is via the `x-provisa-role` metadata key. Missing or unrecognised role metadata causes the call to be rejected with UNAUTHENTICATED. Streaming query RPCs emit one response message per result row; mutation RPCs are unary.

**Use case:** Metadata-based role selection lets a single gRPC channel serve multiple roles by setting the key per-call without reconnecting.

**Code:** `provisa/grpc/server.py`

**Tests:** `tests/e2e/test_grpc_query.py`, `tests/unit/test_grpc_requirements.py`, `tests/integration/test_client_access_integration.py`

## 9. Live Data & Events

### REQ-172 · Dataset Change Events {#REQ-172}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Mutations emit a dataset change event to a Kafka topic — no row-level detail, just `{table, source, timestamp}`.

**Use case:** Mutation change events let downstream consumers react to data changes without polling the source.

**Code:** `provisa/events/`

**Tests:** `tests/unit/test_kafka_change_events.py`, `tests/integration/test_live_data_integration.py`

### REQ-173 · Dataset Change Events {#REQ-173}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Change events fire on the same mutation hook that invalidates cache and marks MVs stale.

**Use case:** Change events and cache invalidation fire together, keeping all derived state consistent after mutations.

**Code:** `provisa/events/`

**Tests:** `tests/unit/test_kafka_change_events.py`, `tests/integration/test_live_data_integration.py`

### REQ-174 · Dataset Change Events {#REQ-174}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Producers running complex ETL outside Provisa can signal changes via a trivial mutation (touch operation).

**Use case:** Touch mutations let external ETL pipelines signal Provisa that source data changed without a direct DB write.

**Code:** `provisa/events/`

**Tests:** `tests/unit/test_kafka_change_events.py`, `tests/integration/test_live_data_integration.py`

### REQ-175 · Dataset Change Events {#REQ-175}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Change event topic configurable via `PROVISA_CHANGE_EVENT_TOPIC` (default: `provisa.change-events`).

**Use case:** Configurable change event topic lets operators isolate event streams by environment or team.

**Code:** `provisa/events/`

**Tests:** `tests/unit/test_kafka_change_events.py`, `tests/integration/test_live_data_integration.py`

### REQ-176 · Kafka Sinks (Table/View Publishing) {#REQ-176}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Registered tables and views can optionally have a Kafka sink — results published to a topic on trigger. Sinks may use change_event, schedule, manual, or poll depending on source capability.

**Use case:** Table/view Kafka sinks push governed results to topics so consumers never bypass security.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_sink.py`, `tests/integration/test_kafka_sink.py`

### REQ-177 · Kafka Sinks (Table/View Publishing) {#REQ-177}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Sink triggers: `change_event` (re-run when source table changes), `schedule` (cron/interval), `manual` (on-demand), `poll` (watermark-based continuous polling — required when CDC is unavailable).

**Use case:** Flexible sink triggers (CDC, schedule, manual, poll) match delivery to what each source physically supports.

**Code:** `provisa/kafka/`

**Tests:** `tests/integration/test_kafka_sink.py`, `tests/unit/test_event_triggers.py`, `tests/unit/test_kafka_sink.py`

### REQ-178 · Kafka Sinks (Table/View Publishing) {#REQ-178}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Sinks are opt-in per registered table or view, configured by the steward.

**Use case:** Opt-in sink configuration prevents accidental data publishing without steward intent.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_sink.py`, `tests/integration/test_kafka_sink.py`

### REQ-180 · Kafka Sinks (Table/View Publishing) {#REQ-180}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Sinks can be added to or removed from a table/view at any time, independently of other config.

**Use case:** Independent sink attachment lets operators add or remove delivery channels without touching other config.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_sink.py`, `tests/integration/test_kafka_sink.py`

### REQ-181 · Kafka Sinks (Table/View Publishing) {#REQ-181}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Sink output format is JSON (one message per row, keyed by optional column).

**Use case:** JSON-per-row Kafka messages let any downstream consumer deserialize results without custom decoders.

**Code:** `provisa/kafka/`

**Tests:** `tests/unit/test_kafka_sink.py`, `tests/integration/test_kafka_sink.py`

### REQ-260 · Subscriptions {#REQ-260}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Polling-based subscription provider for sources without native CDC. A `watermark_column` (monotonic timestamp or sequence, e.g. `updated_at`) must be declared on the table config. Without a `watermark_column`, poll subscriptions are unavailable for that source. Part of Phase AM (Live Query Engine). Note: Poll-based subscriptions cannot detect hard deletes — a deleted row leaves no updated watermark value and will not appear in any poll. Applications requiring delete visibility must use soft deletes (e.g. a `deleted_at` flag) that update the watermark column; the delete is then visible as an update event containing the soft-delete marker.

**Use case:** Poll-based subscriptions enable live delivery for sources that do not support CDC or LISTEN/NOTIFY.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/e2e/test_subscriptions.py`, `tests/integration/test_sse_subscriptions.py`, `tests/unit/test_live_engine_full.py`, `tests/unit/test_subscribe.py`

### REQ-261 · Subscriptions {#REQ-261}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Debezium CDC subscription provider for non-PG RDBMS sources (MySQL, MariaDB, SQL Server, Oracle). Debezium captures changes from the source's transaction log and publishes to Kafka topics. Provisa's Kafka notification provider consumes these CDC events and streams them as SSE subscriptions. Requires Kafka + Debezium connector infrastructure.

**Use case:** Debezium CDC subscriptions enable real-time events for MySQL, MariaDB, SQL Server, and Oracle sources.

**Code:** `provisa/subscriptions/`

**Tests:** `tests/e2e/test_subscriptions.py`, `tests/integration/test_debezium_cdc.py`, `tests/integration/test_sse_subscriptions.py`, `tests/unit/test_debezium_provider.py`, `tests/unit/test_subscribe.py`

### REQ-282 · Live Query Engine (Unified Subscription & Sink Delivery) {#REQ-282}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A single Live Query Engine powers all poll-based live delivery. It is the common implementation for: (a) SSE subscriptions on tables/views without CDC, (b) Kafka sinks on tables/views without CDC. Delivery mode (CDC vs poll) is orthogonal to the output mechanism (SSE vs Kafka sink).

**Use case:** Single Live Query Engine powers all poll-based delivery so CDC vs poll and SSE vs sink are independent choices.

**Code:** `provisa/live/`, `provisa/subscriptions/`

**Tests:** `tests/integration/test_live_sse_integration.py`, `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`

### REQ-283 · Live Query Engine (Unified Subscription & Sink Delivery) {#REQ-283}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

`watermark_column` is a required config field for any poll-based live delivery. The column must be monotonically increasing (e.g. `updated_at`, `created_at`) and is declared on the table/view config ([REQ-260](#REQ-260)). Without `watermark_column`, poll delivery is unavailable and config validation fails at startup.

**Use case:** Required watermark_column config makes poll delivery explicit and prevents unbounded or duplicate event delivery.

**Code:** `provisa/live/`, `provisa/subscriptions/`

**Tests:** `tests/integration/test_live_sse_integration.py`, `tests/unit/test_config_validation_edge_cases.py`, `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`

### REQ-285 · Live Query Engine (Unified Subscription & Sink Delivery) {#REQ-285}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Tables declare delivery mode in the subscription/sink config: `delivery: cdc` or `delivery: poll`. `cdc` is available for PostgreSQL (LISTEN/NOTIFY), Debezium-connected sources ([REQ-261](#REQ-261)), and MongoDB (Change Streams per [REQ-258](#REQ-258)). All other sources — Trino-federated, JDBC with restricted access, Kafka topics, API sources — must use `delivery: poll`. Config validation rejects `delivery: cdc` for sources that do not support it.

**Use case:** Delivery mode validation at config load time prevents misconfiguring CDC on sources that do not support it.

**Code:** `provisa/live/`, `provisa/subscriptions/`

**Tests:** `tests/integration/test_live_sse_integration.py`, `tests/unit/test_available_tables.py`, `tests/unit/test_config_validation_edge_cases.py`, `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`

### REQ-286 · Live Query Engine (Unified Subscription & Sink Delivery) {#REQ-286}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SSE subscription and Kafka sink are equivalent output mechanisms for a table or view. A single live definition may list both `sse_subscription` and `kafka_sink` in its `outputs`. The poll engine executes once per interval; each output tracks its own watermark independently in a `live_query_state` table (keyed by source name + output_type). A slow Kafka consumer does not block SSE delivery and vice versa.

**Use case:** Independent watermark tracking per output type ensures a slow Kafka consumer never delays SSE subscribers.

**Code:** `provisa/live/`, `provisa/subscriptions/`

**Tests:** `tests/integration/test_live_sse_integration.py`, `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`

### REQ-287 · Live Query Engine (Unified Subscription & Sink Delivery) {#REQ-287}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Live query state persisted in a `live_query_state` PG table: `(source, output_type, last_watermark, last_polled_at, status)`. On restart, polling resumes from the last committed watermark — no events are replayed or skipped due to process restart. `status` tracks `active`, `paused`, `error`.

**Use case:** Persisted live query state lets poll delivery resume from exactly the last committed watermark after a restart.

**Code:** `provisa/live/`, `provisa/subscriptions/`

**Tests:** `tests/integration/test_live_sse_integration.py`, `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`

### REQ-565 · Subscriptions {#REQ-565}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

At startup, Provisa idempotently installs `AFTER INSERT OR UPDATE OR DELETE` triggers on all registered PostgreSQL subscription tables. Each trigger calls `pg_notify('provisa_{table}', json_build_object('op', lower(TG_OP), 'row', ...))` so that raw DML from any writer is picked up by SSE subscriptions.

**Use case:** Auto-installed triggers ensure external writes to PostgreSQL tables are captured by SSE subscriptions without requiring the writer to call pg_notify manually.

**Code:** `provisa/subscriptions/pg_triggers.py`

**Tests:** `tests/e2e/test_subscriptions.py`, `tests/integration/test_sse_subscriptions.py`, `tests/unit/test_event_triggers.py`, `tests/unit/test_subscribe.py`

### REQ-566 · Subscriptions {#REQ-566}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When PostgreSQL trigger installation fails (e.g. insufficient privilege), Provisa logs a warning and falls back to watermark-based polling for that table, provided `watermark_column` is configured on the table. Tables where installation succeeds are tracked in-memory to select the LISTEN/NOTIFY provider path.

**Use case:** Graceful fallback to polling prevents a trigger permission failure from making a table entirely unsubscribable.

**Code:** `provisa/subscriptions/pg_triggers.py`

**Tests:** `tests/e2e/test_subscriptions.py`, `tests/integration/test_live_data_integration.py`, `tests/unit/test_subscribe.py`

### REQ-567 · Subscriptions {#REQ-567}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When a subscription field selects columns from joined tables via registered relationships, the subscription engine collects all physical table names referenced by the join walk and calls `watch_many(all_watch_tables)` on the PG notification provider. A change to any joined physical table re-fires the subscription query.

**Use case:** Watching all involved physical tables ensures that relationship-traversing subscriptions surface changes to nested data without requiring a separate subscription per table.

**Code:** `provisa/api/data/subscription_sse.py`

**Tests:** `tests/e2e/test_subscriptions.py`, `tests/integration/test_sse_subscriptions.py`, `tests/unit/test_audit_requirements.py`, `tests/unit/test_live_engine_full.py`

### REQ-568 · Subscriptions {#REQ-568}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Each SSE change event is emitted as a JSON object with the shape `{"event": "<op>", "table": "<table>", "row": {...}}` where `op` is one of `insert`, `update`, or `delete`.

**Use case:** Standardized event shape lets clients deserialize change events without custom parsing per table.

**Code:** `provisa/api/data/subscription_sse.py`, `provisa/subscriptions/pg_provider.py`

**Tests:** `tests/e2e/test_subscriptions.py`, `tests/integration/test_sse_subscriptions.py`, `tests/unit/test_subscribe.py`

## 10. UI & Admin Surfaces

### REQ-058 · UI & Frontend {#REQ-058}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Branded, custom React-based UI — rendered surface determined entirely by user's assembled role set.

**Use case:** Role-driven rendered UI ensures each user sees only the capabilities relevant to their assigned permissions.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_ui_role_requirements.py`

### REQ-059 · UI & Frontend {#REQ-059}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Role composition system: admin assembles capabilities from independently assignable building blocks.

**Use case:** Composable role building blocks let admins precisely grant capabilities without overprivileging any user.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_ui_role_requirements.py`

### REQ-060 · UI & Frontend {#REQ-060}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Capabilities: Source Registration, Table Registration, Relationship Registration, Security Configuration, Query Development, Admin. Holders of a create capability (e.g. Relationship/View Registration) also execute creation requests queued by users lacking that capability ([REQ-063](#REQ-063)).

**Use case:** Distinct assignable capabilities let organizations enforce separation of duties across data governance roles.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_ui_role_requirements.py`

### REQ-061 · UI & Frontend {#REQ-061}

**Status:** ✅ complete · **Priority:** MUST · **Type:** ui

Every destructive or consequential action requires explicit confirmation with consequence summary.

**Use case:** Explicit confirmation for destructive actions prevents accidental data loss or misconfiguration in the UI.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_ui_role_requirements.py`

### REQ-062 · UI & Frontend {#REQ-062}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Test endpoint execution shows RLS filters applied, columns excluded, schema scope enforced in result metadata.

**Use case:** Test endpoint metadata display lets developers see exactly which security rules applied to their query result.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_ui_role_requirements.py`

### REQ-063 · UI & Frontend {#REQ-063}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Creation-request queue: when a user lacks authority for a create operation (view, relationship), they submit a request that enters a queue visible to anyone holding the rights to execute it. Designed for steward efficiency; rejection reasons must be specific and actionable.

**Use case:** Actionable rejection reasons in the creation-request queue help requesters quickly fix and resubmit.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_ui_role_requirements.py`

### REQ-164 · Admin & Configuration {#REQ-164}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

`GET/PUT /admin/config` for config YAML download/upload with backup and reload.

**Use case:** Config YAML download/upload with backup lets admins version-control platform configuration outside the UI.

**Code:** `provisa/api/admin/`

**Tests:** `tests/e2e/test_admin_flow.py`, `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`, `tests/unit/test_config_reload.py`, `tests/unit/test_config_secrets.py`

### REQ-165 · Admin & Configuration {#REQ-165}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

`GET/PUT /admin/settings` for runtime platform settings (redirect, sampling, cache).

**Use case:** Runtime settings API lets operators adjust redirect, sampling, and cache behaviour without a restart.

**Code:** `provisa/api/admin/`

**Tests:** `tests/e2e/test_admin_flow.py`, `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`

### REQ-166 · Admin & Configuration {#REQ-166}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Editable relationships page with materialize toggle, delete, and add form.

**Use case:** Editable relationships page with materialize toggle lets stewards manage cross-source joins interactively.

**Code:** `provisa/api/admin/`

**Tests:** `tests/e2e/test_admin_flow.py`, `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`

### REQ-167 · Admin & Configuration {#REQ-167}

**Status:** ✅ complete · **Priority:** MAY · **Type:** ui

AI-suggested relationships via LLM discovery integration on relationships page.

**Use case:** AI-suggested relationships reduce steward effort when registering foreign-key links across complex schemas.

**Code:** `provisa/api/admin/`

**Tests:** `tests/e2e/test_admin_flow.py`, `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`

### REQ-242 · Commands UI {#REQ-242}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin UI "Commands" page listing all registered functions and webhooks. Grouped by type (DB Function / Webhook). Shows source, domain, exposed_as (mutation/query), governance level, return table, argument count.

**Use case:** Commands admin page gives stewards visibility into all registered functions and webhooks in one place.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_commands_requirements.py`

### REQ-243 · Commands UI {#REQ-243}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Add command form with type selector: DB Function (source, schema, function name, exposed_as, returns registered table, arguments, visible_to/writable_by) or Webhook (name, URL, method, timeout_ms, returns registered table or inline type, arguments, visible_to).

**Use case:** Add command form lets stewards register DB functions and webhooks without editing YAML files directly.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_commands_requirements.py`

### REQ-244 · Commands UI {#REQ-244}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Inline type builder for webhook return types — dynamic rows of field name + GraphQL type. Used when webhook returns a custom shape not backed by a registered table.

**Use case:** Inline type builder lets stewards define custom webhook return shapes visually without writing JSON Schema.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_commands_requirements.py`

### REQ-245 · Commands UI {#REQ-245}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Test command button — execute function/webhook with sample arguments, display result + governance pipeline applied (which columns masked, RLS filters, role). Same pattern as query test endpoint.

**Use case:** Test command button lets developers verify function execution and governance pipeline before approving.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/e2e/test_query_pipeline.py`, `tests/unit/test_commands_requirements.py`

### REQ-248 · UI & Design Patterns {#REQ-248}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

GraphQL Voyager integration uses iframe with React 18 CDN standalone bundle (not a native React component fork). The iframe approach avoids MUI v5/React 19 incompatibility and provides reliable isolation. No component fork is planned.

**Use case:** GraphQL Voyager iframe integration provides schema visualisation without React version compatibility issues.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_commands_requirements.py`

### REQ-249 · UI & Design Patterns {#REQ-249}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Column-level masking configuration stored as inline fields on ColumnConfig in models.py (mask_type, mask_pattern, etc.) — not as a separate MaskingRule Pydantic model or separate database table. Masking rules are loaded from table_columns DB rows at startup in app.py. This co-locates masking config with the column definition it applies to.

**Use case:** Inline masking config on ColumnConfig keeps column governance co-located, avoiding split config lookups.

**Code:** `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/no-domain-mode.spec.ts`, `provisa-ui/e2e/relationships-header.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_column_governance_requirements.py`

### REQ-395 · UI & Frontend {#REQ-395}

**Status:** ✅ complete · **Priority:** MUST · **Type:** ui

The PK designation must be configurable in the TablesPage UI via checkbox per column, visible in both add-form and edit-form views, and in the read-only view.

**Use case:** UI-based PK designation lets stewards configure keys without editing YAML files.

**Code:** `provisa-ui/src/pages/TablesPage`, `provisa-ui/src/components/ColumnForm`

**Tests:** `provisa-ui/e2e/tables-register.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_column_governance_requirements.py`

### REQ-396 · UI & Frontend {#REQ-396}

**Status:** ✅ complete · **Priority:** MUST · **Type:** ui

The graph node context menu "Exclude from query" option must be disabled (greyed out) when no primary key is configured for that node label.

**Use case:** Disabling exclusion without a PK prevents incomplete or ambiguous row-level filtering.

**Code:** `provisa-ui/src/components/GraphContextMenu`

**Tests:** `provisa-ui/e2e/graph-query-panel-height.spec.ts`, `provisa-ui/e2e/graph-show-children.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_column_governance_requirements.py`

### REQ-401 · UI & Frontend {#REQ-401}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Foreign key (FK) and alternate key (AK) badges surface as read-only indicators in the column editor UI, showing which columns are linked by relationships.

**Use case:** FK/AK badges provide visual feedback on relationship column usage without allowing direct editing.

**Code:** `provisa-ui/src/components/ColumnForm`, `provisa-ui/src/pages/TablesPage`

**Tests:** `provisa-ui/e2e/tables-register.spec.ts`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_column_governance_requirements.py`

### REQ-404 · UI & Frontend {#REQ-404}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Security page RLS form includes "Apply To" toggle: "Specific Table" or "Entire Domain". Selection determines whether `table_id` or `domain_id` is populated; the other is NULL.

**Use case:** UI toggle lets stewards choose rule scope without understanding the underlying schema design.

**Code:** `provisa-ui/src/pages/SecurityPage`

**Tests:** `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_column_governance_requirements.py`

### REQ-410 · UI & Frontend {#REQ-410}

**Status:** ✅ complete · **Priority:** MUST · **Type:** ui

GraphFrame Cypher WHERE clause generation must use single-quoted string literals (not double-quoted SQL identifiers) for non-numeric PK values to produce valid Cypher syntax.

**Use case:** Correct single-quote quoting prevents SQL identifier errors when filtering by string PK values in the graph view.

**Code:** `provisa-ui/src/pages/GraphFrame.tsx`

**Tests:** `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_column_governance_requirements.py`, `tests/unit/test_cypher_graph_fns.py`

### REQ-533 · Admin & Configuration {#REQ-533}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The admin GraphQL API is a Strawberry-based endpoint mounted at POST /admin/graphql, served by FastAPI via strawberry.fastapi.GraphQLRouter. It is wholly separate from the data GraphQL endpoint at /data/graphql with distinct schemas, independent routers, and independent access control.

**Use case:** Distinct admin API endpoint prevents governance operations from being confused with data query operations and allows independent auth enforcement.

**Code:** `provisa/api/app.py`, `provisa/api/admin/schema.py`

**Tests:** `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`

### REQ-620 · Admin & Configuration {#REQ-620}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The admin GraphQL API is mounted at `/admin/graphql` on the backend server (default port 8001). It is distinct from the data GraphQL API at `/data/graphql` and is used for platform administration operations.

**Use case:** Separate admin GraphQL endpoint allows tooling and scripts to target administrative mutations and queries without mixing them with data queries.

**Code:** `provisa/api/app.py line 3389`

**Tests:** `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`

### REQ-622 · Admin & Configuration {#REQ-622}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

The admin GraphQL API ships with Strawberry's built-in GraphiQL IDE accessible via GET /admin/graphql in a browser, allowing interactive exploration of the full admin schema.

**Use case:** Integrated GraphiQL removes the need for external GraphQL tooling when exploring or debugging admin operations.

**Code:** `provisa/api/app.py`, `provisa/api/admin/schema.py`

**Tests:** `tests/integration/test_admin_api.py`, `tests/unit/test_admin_mv.py`, `tests/unit/test_admin_requirements.py`

### REQ-644 · Graph Grouping — View Transform {#REQ-644}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Node grouping in the graph UI is a view transform: the underlying node and edge data is never modified; the rendering layer derives a `groupKey` from any node attribute and applies visual encodings (color, hull ellipse) without altering model state. GroupingState is held in GraphFrame and passed to GraphCanvas and HullOverlay.

**Use case:** View-transform grouping keeps the data model clean and allows switching between grouping attributes without re-fetching data.

**Code:** `provisa-ui/src/components/graph/GraphFrame.tsx`, `provisa-ui/src/components/graph/graph-clusters.ts`

**Tests:** `provisa-ui/src/__tests__/GraphFrame.test.tsx`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_graph_grouping_requirements.py`

### REQ-645 · Graph Grouping — Attribute Discovery {#REQ-645}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

After any query result, the UI scans all node properties to build a list of groupable attributes: attributes with more than one distinct value across nodes are included. `domain` (derived from the node label prefix) is always the first entry. Schema cluster virtuals (`schema_L1`, `schema_L2`, `schema_L3`) are included when their underlying `scl1`/`scl2`/`scl3` properties have multiple distinct values. The grouping dropdown is populated dynamically from this list — no attribute names are hardcoded.

**Use case:** Dynamic attribute discovery lets users group by any categorical property in the current result set without prior configuration.

**Code:** `provisa-ui/src/components/graph/GraphFrame.tsx`

**Tests:** `provisa-ui/src/__tests__/GraphFrame.test.tsx`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_graph_grouping_requirements.py`, `tests/unit/test_llm_discovery.py`

### REQ-646 · Graph Grouping — Color Encoding {#REQ-646}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

When a grouping attribute is active, each node's `background-color` in the Cytoscape style function is overridden to the color assigned to that node's group value. Color assignment uses `clusterColor(groupValue)` which derives a stable color from the group key string.

**Use case:** Color-by-group gives immediate visual differentiation of node clusters without requiring a separate legend interaction.

**Code:** `provisa-ui/src/components/graph/GraphCanvas.tsx`, `provisa-ui/src/components/graph/graph-clusters.ts`

**Tests:** `provisa-ui/src/__tests__/GraphFrame.test.tsx`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_graph_grouping_requirements.py`

### REQ-647 · Graph Grouping — Hull SVG Overlay {#REQ-647}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

When a grouping attribute is active, an SVG ellipse is drawn over the Cytoscape canvas around each group's node positions. Ellipses are computed by fitting bounding-box centroids and radii from node positions in graph coordinates, then converted to screen coordinates using Cytoscape zoom and pan. The SVG layer re-renders on every Cytoscape `viewport` event so hulls track pan and zoom.

**Use case:** SVG hull ellipses provide spatial grouping cues that remain accurate during pan and zoom without requiring re-layout.

**Code:** `provisa-ui/src/components/graph/GraphCanvas.tsx`

**Tests:** `provisa-ui/src/__tests__/GraphFrame.test.tsx`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_graph_grouping_requirements.py`

### REQ-648 · Graph Grouping — Collapse to Supernode {#REQ-648}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Each group cluster supports a collapse toggle. When collapsed: member nodes are removed from the Cytoscape graph; a synthetic supernode with id `__collapsed_<level>_<cidToId(cid)>` is added showing the group label and member count; edges crossing the group boundary are rewritten to connect to/from the supernode. Double-clicking the supernode expands the group and restores original nodes and edges. Collapsed state is held in a `Set<string>` per frame and reset when the grouping attribute changes.

**Use case:** Collapse-to-supernode reduces visual complexity for large clusters while preserving cross-cluster edge visibility.

**Code:** `provisa-ui/src/components/graph/graph-clusters.ts`, `provisa-ui/src/components/graph/GraphCanvas.tsx`

**Tests:** `provisa-ui/src/__tests__/GraphFrame.test.tsx`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_graph_grouping_requirements.py`

### REQ-649 · Graph Node Size Encoding {#REQ-649}

**Status:** ✅ complete · **Priority:** MAY · **Type:** ui

Node size can be driven by any numeric node property via `sizeByProperty` (a per-label map of property name). The property value is linearly scaled between the observed min and max across all nodes of that label to produce a node diameter between `base` and `base * multiplier`. The `sizeByProperty` setting is persisted in localStorage under key `provisa.graph.sizeByProperty`.

**Use case:** Size-by-property encoding lets analysts visually compare centrality scores or any other numeric metric across nodes without reading raw values.

**Code:** `provisa-ui/src/components/graph/GraphCanvas.tsx`, `provisa-ui/src/pages/GraphPage.tsx`

**Tests:** `provisa-ui/src/__tests__/GraphFrame.test.tsx`, `provisa-ui/src/__tests__/ui_requirements.test.ts`, `tests/unit/test_graph_grouping_requirements.py`

## 11. Platform, Infrastructure & Delivery

### REQ-055 · Infrastructure {#REQ-055}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Docker Compose for development/small-team: single command, Provisa + Trino coordinator + configurable workers, all connectors pre-loaded.

**Use case:** Single-command Docker Compose setup lets developers run the full Provisa stack without manual configuration.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-056 · Infrastructure {#REQ-056}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Helm chart for production Kubernetes: horizontal Trino worker scaling, resource groups, HPA autoscaling.

**Use case:** Helm chart with HPA autoscaling lets production deployments scale Trino workers automatically under load.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/e2e/test_helm_minikube.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-057 · Infrastructure {#REQ-057}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Provisa container is stateless; deployment topology behind Trino endpoint is configuration concern.

**Use case:** Stateless Provisa container enables horizontal scaling and rolling deployments without session affinity concerns.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-064 · Error Handling & Reliability {#REQ-064}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Never add fallback values or silent error handling — all errors must be explicit and fail-fast.

**Use case:** Fail-fast error handling ensures production failures surface immediately instead of silently returning bad data.

**Code:** —

**Tests:** `tests/unit/test_error_handling.py`, `tests/unit/test_query_timeout.py`

### REQ-065 · Error Handling & Reliability {#REQ-065}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

No migrations in version 1 development.

**Use case:** No-migration policy eliminates schema migration risk and keeps V1 development iteration fast.

**Code:** —

**Tests:** `tests/unit/test_error_handling.py`

### REQ-069 · Architecture & Design Patterns {#REQ-069}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Architecture docs in `docs/arch/` ARE the planning documents — update when requirements change, don't implement without planning.

**Use case:** Architecture docs as planning documents ensure implementation always reflects a reviewed design.

**Code:** —

**Tests:** `tests/unit/test_process_requirements.py`

### REQ-070 · Architecture & Design Patterns {#REQ-070}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Maximum brevity in communications — code and facts only, no pleasantries or explanations unless asked.

**Use case:** Maximum brevity discipline keeps communications focused on actionable information rather than ceremony.

**Code:** —

**Tests:** `tests/unit/test_process_requirements.py`

### REQ-071 · Architecture & Design Patterns {#REQ-071}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

New requirements tracked via requirements-tracker agent appending to `docs/arch/requirements.md`.

**Use case:** Automated requirements tracking ensures every design decision is recorded without interrupting development flow.

**Code:** —

**Tests:** `tests/unit/test_infra_requirements.py`

### REQ-072 · Commercial Positioning {#REQ-072}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Core product is open source: Docker Compose, Helm chart, UI, compiler, SQLGlot layer, Trino backend.

**Use case:** Open-source core makes Provisa accessible to teams that cannot budget for commercial data access platforms.

**Code:** —

**Tests:** `tests/unit/test_infra_requirements.py`

### REQ-073 · Commercial Positioning {#REQ-073}

**Status:** ✅ complete · **Priority:** MAY · **Type:** infrastructure

SaaS tier: hosted control plane with customer-hosted data plane option.

**Use case:** SaaS tier with customer-hosted data plane lets organizations adopt managed control without moving their data.

**Code:** —

**Tests:** `tests/unit/test_control_plane.py`, `tests/unit/test_process_requirements.py`

### REQ-074 · Commercial Positioning {#REQ-074}

**Status:** ✅ complete · **Priority:** MAY · **Type:** infrastructure

Enterprise tier: SLA guarantees, dedicated support, advanced audit logging, compliance reporting.

**Use case:** Enterprise tier SLA and audit logging meets regulated-industry compliance and procurement requirements.

**Code:** —

**Tests:** `tests/unit/test_process_requirements.py`, `tests/unit/test_sla_monitor.py`

### REQ-169 · Infrastructure {#REQ-169}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Trino 480 with Iceberg results catalog (JDBC on PG, native S3 filesystem).

**Use case:** Trino 480 with Iceberg results catalog provides modern open-table-format storage for all redirected results.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-170 · Infrastructure {#REQ-170}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

`start-ui.sh --reset-volumes` for Docker crash recovery.

**Use case:** reset-volumes flag gives developers a one-command recovery path after Docker volume corruption.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-171 · Infrastructure {#REQ-171}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

MinIO results bucket auto-created at startup.

**Use case:** MinIO bucket auto-creation at startup prevents first-run failures due to missing redirect storage.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-223 · Installer & Packaging {#REQ-223}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Single-executable installer that bundles the Provisa platform into one download. Underlying technology (Python server, PostgreSQL admin DB, Trino query engine, React UI) is hidden from the user. Source datasets are NOT bundled -- they connect over the wire.

**Use case:** Single-executable installer hides platform complexity so analysts can run Provisa without DevOps knowledge.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`, `tests/unit/test_live_engine_full.py`

### REQ-224 · Installer & Packaging {#REQ-224}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Installer expands into a hidden directory (`~/.provisa/`) containing all services. User interacts via `provisa start`, `provisa stop`, `provisa status`, `provisa open` (opens browser).

**Use case:** CLI commands (start/stop/status/open) give non-technical users a simple interface to manage the platform.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-225 · Installer & Packaging {#REQ-225}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Default deployment uses embedded PostgreSQL (postgres:16) for admin DB and bundled Trino for query federation. Vertical scaling by default -- single machine, increase resources as needed.

**Use case:** Embedded PostgreSQL and Trino let a single machine run the full platform out of the box.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-226 · Installer & Packaging {#REQ-226}

**Status:** ✅ complete · **Priority:** MAY · **Type:** infrastructure

Users can later connect their own Trino cluster, Spark, external auth provider, or external PostgreSQL for the admin DB via config. Pointing to an external Trino instance is the primary scale-out mechanism.

**Use case:** External Trino and auth provider support lets organisations scale Provisa onto existing enterprise infrastructure.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-227 · Installer & Packaging {#REQ-227}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

AF2 is delivered in OS phases. AF2a (immediate): macOS arm64/x86 signed + notarized `.dmg`. AF2b (follow-on): Linux `.AppImage`. AF2c (follow-on): Windows signed `.exe`. Each phase bundles the container runtime for its platform — no Docker, OrbStack, or Colima prerequisite visible to the user.

**Use case:** OS-native signed bundles (DMG/AppImage/EXE) let users install Provisa like any desktop app without Docker.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-228 · Installer & Packaging {#REQ-228}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Phase 1 (AF1, complete): Shell script installer with Docker Compose, `provisa` CLI wrapper, state in `~/.provisa/`. Requires Docker/OrbStack/Colima. Phase 2 (AF2): Airgapped native app bundle — Lima + containerd embedded, all service images bundled as `.tar` archives, zero outbound network at install or runtime. AF3 (native OS packages via Homebrew/.deb/.rpm) is superseded by AF2.

**Use case:** AF2 airgapped bundle removes the Docker prerequisite so enterprise users with no internet access can install.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-254 · Testing & Quality {#REQ-254}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Integration tests must use Docker — spin up required containers (PG, Trino, Redis, etc.), install dependencies, run tests, and tear down containers. Tests must not assume a pre-existing Docker Compose stack.

**Use case:** Integration tests that spin up Docker containers ensure tests validate against real service behaviour.

**Code:** —

**Tests:** `tests/unit/test_infra_requirements.py`

### REQ-255 · Testing & Quality {#REQ-255}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Unit tests must mock all external components (databases, Trino, Redis, HTTP endpoints, file system where applicable). No unit test should require a running external service.

**Use case:** Unit tests that mock all external components run fast and independently of any running infrastructure.

**Code:** —

**Tests:** `tests/unit/test_infra_requirements.py`

### REQ-294 · Installer & Packaging {#REQ-294}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The distribution must be fully airgap-capable. No outbound network calls at install time or first launch. All container images are bundled inside the app and loaded via `ctr images import`. Image references in docker-compose use digests, not tags. Suitable for enterprise environments with no internet access.

**Use case:** Fully airgapped bundle lets enterprises in no-internet environments install and run Provisa from local media.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-302 · OpenTelemetry Instrumentation {#REQ-302}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

OpenTelemetry (OTel) instrumentation across all Provisa components — traces, metrics, and logs exported via OTLP. Covers: FastAPI app, GraphQL compiler, SQL executor, Trino driver, admin GraphQL API, Arrow Flight server, cache layer, MV subsystem, discovery pipeline, and all source adapters. Structured log emission with trace_id/span_id correlation.

**Use case:** OTel instrumentation across all components gives operators unified traces, metrics, and logs in one backend.

**Code:** `provisa/otel_compat.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_executor_trino.py`, `tests/unit/test_flight_modes.py`, `tests/unit/test_otel_requirements.py`, `tests/unit/test_otel_trace_pipeline.py`

### REQ-303 · OpenTelemetry Instrumentation {#REQ-303}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

OTel integration via opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-httpx, and OTLP exporter. Auto-instrumentation applies to FastAPI and httpx. Manual span/metric instrumentation at: compile_query, execute_trino, execute_direct, handle_api_query, cache check/store, MV rewrite, RLS inject, masking inject. Export endpoint configured via environment variable (default: `http://localhost:4317`).

**Use case:** Auto-instrumentation of FastAPI and httpx combined with manual spans gives full request visibility end-to-end.

**Code:** `provisa/otel_compat.py`, `provisa/api/app.py`

**Tests:** `tests/integration/test_cache_store.py`, `tests/integration/test_direct_exec.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_mask_inject.py`, `tests/unit/test_otel_requirements.py`

### REQ-330 · Infrastructure & Observability {#REQ-330}

**Status:** ✅ complete · **Priority:** MAY · **Type:** infrastructure

Development observability stack available in docker-compose under `observability` profile (opt-in). OTel Collector receives OTLP on 4317/4318 and exports metrics to Prometheus and traces to Tempo. Grafana on port 3100 with Prometheus and Tempo datasources pre-provisioned and a Provisa dashboard included. Provisa app configured via `OTEL_EXPORTER_OTLP_ENDPOINT` env var (default: `http://localhost:4317`).

**Use case:** Opt-in observability stack with OTel, Prometheus, Tempo, and Grafana lets developers debug performance locally.

**Code:** `helm/`, `docker-compose.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-528 · Infrastructure {#REQ-528}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The path to the Provisa config file is controlled by the `PROVISA_CONFIG` environment variable (default: `config/provisa.yaml`).

**Use case:** Configurable config path lets operators run multiple Provisa instances with different configs on the same host.

**Code:** `provisa/api/app.py`, `provisa/api/admin/_config_io.py`

**Tests:** `tests/integration/test_infra.py`, `tests/unit/test_admin_requirements.py`

### REQ-539 · Infrastructure {#REQ-539}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The `GET /health` (or `HEAD /health`) and `GET /setup/status` endpoints are always unauthenticated — they bypass the `Authorization: Bearer` requirement even when an auth provider is configured. All other endpoints require authentication when `auth.provider` is set.

**Use case:** Unauthenticated health and setup-status endpoints let load balancers, orchestrators, and first-run UIs check server availability without credentials.

**Code:** `provisa/api/app.py`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-545 · OpenTelemetry Instrumentation {#REQ-545}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Provisa runs two independent OTLP export paths: the operator's internal collector and an optional Provisa support endpoint. Each path has its own telemetry filter configuration and applies independently.

**Use case:** Independent export paths let operators share diagnostic telemetry with Provisa support while maintaining separate redaction policies for their internal collector.

**Code:** `provisa/api/otel_setup.py`

**Tests:** `tests/unit/test_otel_requirements.py`

### REQ-546 · OpenTelemetry Instrumentation {#REQ-546}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Telemetry filters run inside a wrapping `_FilteringExporter` before spans leave the process. Original span objects are never mutated — filtering operates on copies.

**Use case:** Non-mutating span filtering ensures telemetry redaction cannot corrupt spans in the internal pipeline.

**Code:** `provisa/api/otel_setup.py`

**Tests:** `tests/unit/test_otel_requirements.py`

### REQ-547 · OpenTelemetry Instrumentation {#REQ-547}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

SQL literal redaction defaults to `true` on the support telemetry path (`support_telemetry_filter.redact_sql_literals`), ensuring query data is not shared with Provisa support by default.

**Use case:** Default-on SQL redaction on the support path protects sensitive query data without requiring operator configuration.

**Code:** `provisa/api/otel_setup.py`

**Tests:** `tests/unit/test_otel_requirements.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-548 · OpenTelemetry Instrumentation {#REQ-548}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The Provisa support OTLP endpoint (`support_endpoint` config or `PROVISA_SUPPORT_OTLP_ENDPOINT` env var) is disabled by default. When unset, no telemetry data leaves the operator's infrastructure via this path.

**Use case:** Opt-in support endpoint ensures no data reaches Provisa infrastructure without explicit operator configuration.

**Code:** `provisa/api/otel_setup.py`

**Tests:** `tests/unit/test_otel_requirements.py`

### REQ-549 · OpenTelemetry Instrumentation {#REQ-549}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa auto-detects OTLP transport from the endpoint URL scheme: `http://` or `https://` schemes use OTLP/HTTP with `/v1/traces`, `/v1/metrics`, and `/v1/logs` path suffixes appended automatically; any other scheme uses OTLP/gRPC with insecure mode.

**Use case:** URL-scheme-based transport detection lets operators configure the OTLP endpoint without a separate transport config field.

**Code:** `provisa/api/otel_setup.py`

**Tests:** `tests/unit/test_otel_requirements.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-558 · Infrastructure {#REQ-558}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

The development backend API (`uvicorn main:app`) listens on port 8001 when launched via `start-ui.sh` or manually. The AppImage and installer paths use port 8000 as the default.

**Use case:** Consistent port convention between dev and production paths prevents connection errors when switching environments.

**Code:** `start-ui.sh`, `provisa/api/app.py`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-559 · Infrastructure {#REQ-559}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

The `start-ui.sh` script starts the Vite UI dev server on port 3000. The production installer serves the UI via `provisa/ui_server.py` on port 3000.

**Use case:** Fixed port 3000 for the UI dev server ensures developers can rely on a consistent URL during local development.

**Code:** `start-ui.sh`, `provisa/ui_server.py`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-560 · Installer & Packaging {#REQ-560}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The Provisa API default port is 8000 in AppImage/installer/Kubernetes deployments (configurable via `--api-port` wizard prompt or `api_port` in config). The Arrow Flight server listens on port 8815 by default.

**Use case:** Documented default ports let operators configure firewalls and load balancers without reading source code.

**Code:** `provisa/core/models.py`, `provisa/api/app.py`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/unit/test_core_registration.py`

### REQ-561 · Installer & Packaging {#REQ-561}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

In a multi-node AppImage deployment, secondary nodes run only the Provisa API and a federation engine worker. Singleton stateful services (PostgreSQL, PgBouncer, MinIO, Redis) run exclusively on the primary node.

**Use case:** Clear primary/secondary service split prevents operators from accidentally running duplicate stateful services that would cause data inconsistency.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`, `tests/unit/test_live_engine_full.py`

### REQ-562 · Installer & Packaging {#REQ-562}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

In a multi-node deployment, secondary Provisa API instances are stateless and read all configuration (sources, tables, relationships, roles, RLS rules) from the primary node's PostgreSQL database at startup. No manual config sync between nodes is required.

**Use case:** Stateless secondary nodes reading from a single PostgreSQL source ensures all API nodes serve identical, authoritative configuration without synchronization overhead.

**Code:** `provisa/api/app.py`, `provisa/core/catalog.py`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_core_registration.py`

### REQ-563 · Installer & Packaging {#REQ-563}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The AppImage first-launch in `--non-interactive` mode installs a systemd unit (`/etc/systemd/system/provisa.service`) so Provisa starts automatically on boot. The AppImage generates credentials during first-launch and writes them to `~/.provisa/config.yaml`.

**Use case:** Systemd integration and auto-generated credentials enable unattended cloud-init and Terraform provisioning without manual post-install steps.

**Code:** `packaging/`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-564 · Infrastructure {#REQ-564}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

The Terraform AWS deployment path provisions a full multi-node Provisa cluster including VPC with two public subnets across two AZs, EC2 instances, an ALB on port 8000 for the HTTP API, and an NLB on port 8815 for Arrow Flight/gRPC. All nodes are attached to both load balancers.

**Use case:** Declarative Terraform provisioning lets operators stand up a production-grade multi-node cluster reproducibly without manual AWS console steps.

**Code:** `terraform/aws/`

**Tests:** `tests/e2e/test_installer.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-592 · Multi-Tenancy {#REQ-592}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Each tenant maps to an org. The `orgs` table stores org namespaces. The `root` org is seeded automatically for single-tenant deployments. In multi-tenant mode, one org is created per customer via the admin API. `user_org_memberships` tracks which users belong to which org. The current interim model adds org_id FK columns to domains and roles for row-level scoping; these columns are superseded by [REQ-695](#REQ-695) schema-per-org, after which org membership is implicit in schema placement and the org_id columns on domains/roles can be dropped.

**Use case:** Org registry and user membership model — the provisioning anchor consumed by [REQ-695](#REQ-695) (schema creation) and [REQ-699](#REQ-699) (PG role creation). The orgs table and user_org_memberships remain valid under the schema-per-org model; the org_id FK columns on domains/roles do not.

**Code:** `provisa/core/schema.sql`, `provisa/api/admin/orgs_router.py`

**Tests:** `tests/unit/test_core_registration.py`

### REQ-593 · Multi-Tenancy {#REQ-593}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The KMS client reads `AWS_KMS_REGION` from the environment and defaults to `us-east-1`. Standard AWS credential chain (environment variables, instance profile, ECS task role) applies.

**Use case:** Region configuration via environment variable lets operators deploy the KMS client to any AWS region without code changes.

**Code:** `provisa/api/billing/kms.py`

**Tests:** `tests/unit/test_tenancy_requirements.py`

### REQ-618 · Infrastructure {#REQ-618}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Backend API server runs on port 8001 with uvicorn hot-reload enabled for changes to `provisa/` and `config/` directories. Log output is written to `.logs/server.log`. The server starts as part of `start-ui.sh`.

**Use case:** Hot-reload on source and config changes lets developers iterate without restarting the server manually.

**Code:** `start-ui.sh line 243`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_config_reload.py`, `tests/unit/test_infra_requirements.py`

### REQ-619 · Infrastructure {#REQ-619}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`start-ui.sh` manages the full dev lifecycle: Ctrl+C stops the backend, UI dev server, and all Docker Compose services and reverts any Trino jvm.config patches. `--keep-docker` flag leaves Docker services running on Ctrl+C. Ctrl+R (SIGUSR1) restarts only the backend without affecting Docker services or the UI.

**Use case:** Lifecycle controls (stop-all, keep-docker, backend-only restart) give developers precise control over which components restart during development.

**Code:** `start-ui.sh lines 248–258`, `start-ui.sh lines 301–306`, `start-ui.sh lines 340–347`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/integration/test_infra.py`, `tests/unit/test_infra_requirements.py`

### REQ-630 · Installer & Packaging {#REQ-630}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Provisa ships as three distinct artifact packages per platform (Core, Obs, Demo) to stay within the GitHub Actions 2 GB artifact limit. CI builds all three in parallel, each uploaded as a separate artifact. The package split maps directly to the three docker-compose layers.

**Use case:** Three-package split keeps each CI artifact under the 2 GB GitHub Actions limit while preserving logical separation between core, observability, and demo service sets.

**Code:** `packaging/macos/build-dmg.sh`, `packaging/macos/build-dmg-obs.sh`, `packaging/macos/build-dmg-demo.sh`, `packaging/windows/build-sfx.ps1`, `packaging/windows/build-installer-obs.ps1`, `packaging/windows/build-installer-demo.ps1`, `packaging/linux/build-appimage.sh`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-631 · Installer & Packaging {#REQ-631}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Package dependency chain: Core must be installed before Obs; Obs must be installed before Demo. The Demo installer checks for the presence of `~/.provisa/extensions/observability/` before proceeding. Dependency is enforced at install time by the installer scripts, not enforced by CI job ordering.

**Use case:** Explicit install-time dependency checks prevent Demo from being installed without Obs, which would leave demo services unable to emit observability data.

**Code:** `packaging/macos/build-dmg-demo.sh`, `packaging/windows/build-installer-demo.ps1`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-632 · Installer & Packaging {#REQ-632}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Linux AppImage bundles both core and observability (obs) images directly — there is no separate Obs download for Linux. The Demo package is not included in Linux distribution. The Linux `first-launch.sh` always starts core + obs together; no extension mechanism is used.

**Use case:** Bundling obs directly on Linux serves the server/technical user base where observability is production-useful, and eliminates the extension download step on a platform where a single monolithic AppImage is simpler to distribute.

**Code:** `packaging/linux/build-appimage.sh`, `packaging/linux/first-launch.sh`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-633 · Installer & Packaging {#REQ-633}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

macOS and Windows use an extension model: the container/compute tier is the only tier that creates the container runtime (Lima + containerd on macOS; WSL2 + containerd on Windows — never VirtualBox). Obs and Demo are extension packages that load images into the existing runtime and write a compose file into `~/.provisa/extensions/<name>/`. The launcher enumerates `extensions/*/docker-compose.*.yml` at startup and appends each found file to the compose file list, composing the service set dynamically. `PROVISA_REDIRECT_ENABLED`, MinIO, and OTel env vars are set only when the obs extension is present. The native base tier ([REQ-979](#REQ-979)) has no runtime VM at all.

**Use case:** Extension model lets users install observability and demo services incrementally without reinstalling Core, and lets the launcher automatically discover installed extensions without hard-coded feature flags.

**Code:** `packaging/macos/build-dmg-obs.sh`, `packaging/macos/build-dmg-demo.sh`, `packaging/macos/ProvisaLauncher/Sources/ProvisaLauncher/Services/ScriptRunner.swift`, `packaging/windows/build-container.ps1`, `packaging/windows/install-container.ps1`, `packaging/windows/provisa-container.ps1`, `packaging/windows/wsl/provision-containerd.sh`, `packaging/windows/wsl/start-containerd.sh`, `packaging/windows/build-installer-obs.ps1`

**Tests:** `tests/e2e/test_installer.py`, `tests/unit/test_infra_requirements.py`

### REQ-634 · Installer & Packaging {#REQ-634}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The dev environment runs the Python backend (uvicorn, port 8000) and UI (vite dev server, port 3000) on the host, never in containers. `docker-compose.app.yml` and `docker-compose.airgap.yml` are packaged-product only and must never be included in dev compose stacks. `start-ui-install.sh --demo` is the only dev mode flag; `--demo` always implies obs. Ports 8000 and 3000 must never appear in any dev compose file.

**Use case:** Host-based uvicorn + vite in dev avoids port conflicts with containerised services and enables hot-reload without rebuilding images.

**Code:** `start-ui-install.sh`, `docker-compose.dev-install.yml`, `docker-compose.observability.yml`, `docker-compose.demo.yml`

**Tests:** `tests/e2e/test_health_checks.py`, `tests/unit/test_infra_requirements.py`

## 12. Migration & Compatibility (Hasura)

### REQ-182 · Hasura Migration Converters {#REQ-182}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Hasura v2 metadata converter -- CLI tool that reads a Hasura v2 metadata export directory and emits valid Provisa YAML config. Converts tracked tables, relationships, permissions, roles, and auth.

**Use case:** Hasura v2 parity — automates migration from Hasura v2 so teams avoid manual YAML rewriting.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-183 · Hasura Migration Converters {#REQ-183}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Hasura DDN (v3) HML converter -- CLI tool that reads a DDN supergraph project and emits valid Provisa YAML config. Converts ObjectTypes, Models, Relationships, TypePermissions, ModelPermissions, and DataConnectorLinks.

**Use case:** Hasura v2 parity — automates migration from Hasura DDN so teams avoid rewriting supergraph configs.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-184 · Hasura Migration Converters {#REQ-184}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Shared boolean expression-to-SQL converter for Hasura filter expressions. Supports `_eq`, `_neq`, `_gt`, `_gte`, `_lt`, `_lte`, `_in`, `_nin`, `_like`, `_ilike`, `_regex`, `_is_null`, `_and`, `_or`, `_not`. Session variable mapping: `X-Hasura-<Name>` -> `current_setting('provisa.<name>')`.

**Use case:** Hasura v2 parity — shared filter converter ensures RLS rules from Hasura boolean expressions work correctly.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-185 · Hasura Migration Converters {#REQ-185}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter maps `select_permissions[].columns` per role -> Provisa column `visible_to`. `columns: "*"` means all columns visible to that role.

**Use case:** Hasura v2 parity — column visibility from v2 select permissions is preserved automatically on import.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-186 · Hasura Migration Converters {#REQ-186}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter maps `insert/update_permissions[].columns` per role -> Provisa column `writable_by`.

**Use case:** Hasura v2 parity — column write permissions from v2 are preserved automatically on import.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-187 · Hasura Migration Converters {#REQ-187}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter maps `select_permissions[].filter` -> Provisa `rls_rules[]` via boolean expression-to-SQL conversion. `filter: {}` (empty) means no RLS filter.

**Use case:** Hasura v2 parity — RLS filters from v2 select permissions are preserved automatically on import.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-188 · Hasura Migration Converters {#REQ-188}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter maps `object_relationships` -> cardinality=many-to-one and `array_relationships` -> cardinality=one-to-many. Physical column names used directly (no GraphQL resolution needed).

**Use case:** Hasura v2 parity — relationship cardinality from v2 metadata is preserved automatically on import.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-189 · Hasura Migration Converters {#REQ-189}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

DDN converter resolves GraphQL field names to physical column names through `ObjectType.dataConnectorTypeMapping[].fieldMapping` for all field references in relationships, permissions, and column definitions.

**Use case:** Hasura v2 parity — DDN field mappings are resolved to physical columns so no manual remapping is needed.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-190 · Hasura Migration Converters {#REQ-190}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 auth conversion via optional `--auth-env-file` flag. JWT with `jwk_url` -> Provisa `provider: oauth`. JWT `claims_map` -> Provisa `role_mapping[]`. Admin secret -> Provisa `superuser`. Webhook auth emits warning (no Provisa equivalent).

**Use case:** Hasura v2 parity — auth config is migrated automatically so JWT and role mappings carry over.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-191 · Hasura Migration Converters {#REQ-191}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

DDN AggregateExpression metadata preserved in sidecar `provisa-aggregates.yaml` and converted to Provisa aggregate config.

**Use case:** Hasura v2 parity — aggregate expressions from DDN are preserved so analytics queries keep working.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-192 · Hasura Migration Converters {#REQ-192}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Converters emit warnings for unmappable features (event_triggers, remote_schemas, cron_triggers, BooleanExpressionType) without failing conversion. v2 Actions and DDN Commands convert to Provisa `functions` config where backed by stored procedures; webhook-backed actions emit warning with handler URL. Part of Phases AG/AH (Hasura Converters).

**Use case:** Hasura v2 parity — unmappable features emit warnings instead of failing so partial migrations complete.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_event_triggers.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-193 · Hasura Migration Converters {#REQ-193}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Both converters produce output that passes `ProvisaConfig.model_validate()` -- Pydantic-valid config or nothing. Part of Phases AG/AH (Hasura Converters).

**Use case:** Hasura v2 parity — converter output is always Pydantic-valid so imported configs never cause silent runtime errors.

**Code:** `provisa/hasura_v2/`, `provisa/ddn/`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`

### REQ-212 · Hasura v2 Parity: Low-Complexity Features {#REQ-212}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Upsert mutations -- `INSERT ... ON CONFLICT ... DO UPDATE`. New `upsert_<table>` mutation field. Conflict columns inferred from primary key metadata. SQLGlot transpiles to dialect-specific syntax (MySQL `ON DUPLICATE KEY UPDATE`, etc.).

**Use case:** Hasura v2 parity — upsert mutations remove the need for client-side read-then-write logic.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/unit/test_upsert_distinct_on.py`, `tests/unit/test_column_presets.py`, `tests/integration/test_mutation_parity_integration.py`

### REQ-213 · Hasura v2 Parity: Low-Complexity Features {#REQ-213}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

`DISTINCT ON` query argument -- deduplicate results by specified columns. Added as `distinct_on` arg on root query fields. SQLGlot handles dialect differences (window function fallback for non-PG).

**Use case:** Hasura v2 parity — DISTINCT ON deduplication lets clients request unique result sets in a single query.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/unit/test_upsert_distinct_on.py`, `tests/unit/test_column_presets.py`, `tests/integration/test_mutation_parity_integration.py`

### REQ-214 · Hasura v2 Parity: Low-Complexity Features {#REQ-214}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Column presets -- auto-set column values on insert/update from session variables or built-in functions. Config per table: `column_presets: [{column: created_by, source: header, name: x_user_id}, {column: updated_at, source: now}]`. Preset columns removed from user input, injected before SQL generation.

**Use case:** Hasura v2 parity — column presets auto-stamp audit fields like created_by without trusting client input.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/unit/test_upsert_distinct_on.py`, `tests/unit/test_column_presets.py`, `tests/integration/test_mutation_parity_integration.py`

### REQ-215 · Hasura v2 Parity: Low-Complexity Features {#REQ-215}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Inherited roles -- role hierarchy where child role inherits capabilities and domain_access from parent. Config: `parent_role_id` on role definition. Flattened at startup (merge capabilities/domain_access up the chain). Lookups remain O(1).

**Use case:** Hasura v2 parity — inherited roles eliminate redundant permission duplication across similar role definitions.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/integration/test_mutation_parity_integration.py`, `tests/unit/test_column_presets.py`, `tests/unit/test_inherited_roles_extended.py`, `tests/unit/test_upsert_distinct_on.py`

### REQ-216 · Hasura v2 Parity: Low-Complexity Features {#REQ-216}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Scheduled triggers -- time-based execution of registered webhooks or internal functions. APScheduler in-process, cron expression syntax. Config per trigger in `provisa.yaml`. Reuses existing async background task pattern.

**Use case:** Hasura v2 parity — scheduled triggers run webhooks or functions on a cron schedule without external cron infra.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/integration/test_mutation_parity_integration.py`, `tests/unit/test_column_presets.py`, `tests/unit/test_scheduled_triggers.py`, `tests/unit/test_scheduler.py`, `tests/unit/test_upsert_distinct_on.py`

### REQ-217 · Hasura v2 Parity: Low-Complexity Features {#REQ-217}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Batch mutations already supported by GraphQL spec -- multiple mutations in one request execute sequentially. Document existing behavior.

**Use case:** Hasura v2 parity — documenting existing batch mutation support clarifies the capability for developers.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/integration/test_mutation_parity_integration.py`, `tests/unit/test_batch_mutations.py`, `tests/unit/test_column_presets.py`, `tests/unit/test_upsert_distinct_on.py`

### REQ-218 · Hasura v2 Parity: Medium-Complexity Features {#REQ-218}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cursor-based pagination -- `first`, `after`, `last`, `before` args on root query fields. Returns `edges[{cursor, node}]` + `pageInfo{hasNextPage, hasPreviousPage, startCursor, endCursor}`. Cursor encoded as base64 of sort key values. Coexists with existing offset/limit.

**Use case:** Hasura v2 parity — cursor-based pagination lets clients page large result sets stably without offset drift.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/integration/test_mutation_parity_integration.py`, `tests/unit/test_column_presets.py`, `tests/unit/test_cursor_pagination.py`, `tests/unit/test_upsert_distinct_on.py`

### REQ-219 · Hasura v2 Parity: Medium-Complexity Features {#REQ-219}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Subscriptions via Server-Sent Events (SSE) -- `GET /data/subscribe/<table>` endpoint using FastAPI StreamingResponse. PostgreSQL LISTEN/NOTIFY via asyncpg `.add_listener()` for change detection. No WebSocket complexity. Streams INSERT/UPDATE/DELETE events.

**Use case:** Hasura v2 parity — SSE subscriptions push live DB changes to clients without WebSocket complexity.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/unit/test_upsert_distinct_on.py`, `tests/unit/test_column_presets.py`, `tests/integration/test_mutation_parity_integration.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-220 · Hasura v2 Parity: Medium-Complexity Features {#REQ-220}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Database event triggers -- table changes (insert/update/delete) fire webhooks. PostgreSQL trigger + `pg_notify()` -> asyncpg listener -> HTTP POST to configured URL. Config per table with operation filter and retry policy.

**Use case:** Hasura v2 parity — DB event triggers automate webhook calls on row changes without application-layer polling.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/integration/test_event_triggers_integration.py`, `tests/integration/test_mutation_parity_integration.py`, `tests/unit/test_column_presets.py`, `tests/unit/test_event_triggers.py`, `tests/unit/test_upsert_distinct_on.py`

### REQ-221 · Hasura v2 Parity: Medium-Complexity Features {#REQ-221}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Enum table auto-detection -- introspect `pg_enum` at schema build time, generate GraphQL enum types for columns using PostgreSQL user-defined enums. Map enum columns to GraphQL enum type instead of String.

**Use case:** Hasura v2 parity — enum type auto-detection exposes PostgreSQL enums as GraphQL enums with no manual config.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/unit/test_column_presets.py`, `tests/unit/test_enum_detection.py`, `tests/unit/test_upsert_distinct_on.py`, `tests/integration/test_registration_governance_integration.py`

### REQ-222 · Hasura v2 Parity: Medium-Complexity Features {#REQ-222}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

REST endpoint auto-generation -- for each root query field, generate `GET /data/rest/<table>` FastAPI endpoint. Map query args to URL query params (`?limit=10&where.id.eq=1`). Reuses GraphQL compilation pipeline internally.

**Use case:** Hasura v2 parity — auto-generated REST endpoints let REST-only clients consume governed data without GraphQL.

**Code:** `provisa/compiler/`, `provisa/scheduler/`

**Tests:** `tests/e2e/test_query_pipeline.py`, `tests/integration/test_rest_endpoints.py`, `tests/unit/test_column_presets.py`, `tests/unit/test_upsert_distinct_on.py`, `provisa-ui/e2e/infrastructure.spec.ts`

### REQ-621 · Hasura Migration Converters {#REQ-621}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Both Hasura v2 and DDN converters emit placeholder connection credentials in the output config (host: localhost, password: ${env:DB_PASSWORD}). Operators must update connection details before starting Provisa.

**Use case:** Placeholder credentials ensure the generated config is syntactically valid while signaling that real credentials must be supplied before production use.

**Code:** `provisa/hasura_v2/mapper.py`, `provisa/ddn/mapper.py`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-623 · Hasura Migration Converters {#REQ-623}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter maps Hasura source `kind` to Provisa `SourceType`: `pg`/`postgres` -> `postgresql`, `mssql` -> `sqlserver`, `bigquery` -> `bigquery`, `citus` -> `postgresql`, `mysql` -> `mysql`. Connection URL (`database_url`) is parsed into host/port/database/username/ password components. Pool settings (`pool_settings.min_connections`, `pool_settings.max_connections`) are preserved as `pool_min`/`pool_max`.

**Use case:** Hasura v2 parity — source kind and connection URL normalization ensures source configs carry over without manual rewriting.

**Code:** `provisa/hasura_v2/mapper.py`

**Tests:** `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-624 · Hasura Migration Converters {#REQ-624}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter upgrades a role to `write` capability when that role has any `delete_permissions` entry on any table. No per-table delete mapping is produced — the capability upgrade is the only artefact.

**Use case:** Hasura v2 parity — delete permission grants are reflected as write capability in Provisa role definitions.

**Code:** `provisa/hasura_v2/mapper.py`

**Tests:** `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-625 · Hasura Migration Converters {#REQ-625}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When a Hasura v2 source `database_url` is an environment variable reference (`{"from_env": "VAR"}`) or cannot be parsed, the converter substitutes placeholder values: `host: localhost`, `port: 5432`, `database: default`, `username: postgres`, `password: ${env:DB_PASSWORD}`. Operators must fix these via `--source-overrides` before the config is used.

**Use case:** Explicit placeholder values on parse failure make unparseable connection URLs visible rather than producing silent misconfiguration.

**Code:** `provisa/hasura_v2/mapper.py`

**Tests:** `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-626 · Hasura Migration Converters {#REQ-626}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Roles are collected exclusively from permission entries (select, insert, update, delete permissions on tables, action permissions, and inherited role definitions). A Hasura role that exists without any permission entry on any table or action does not appear in the converter output.

**Use case:** Permission-driven role collection ensures only roles that carry actual access rules are emitted, keeping the output minimal.

**Code:** `provisa/hasura_v2/mapper.py`

**Tests:** `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/unit/test_inherited_roles_extended.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-627 · Hasura Migration Converters {#REQ-627}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

v2 converter derives the Provisa table `alias` from `custom_root_fields.select` (first priority), then `custom_root_fields.select_by_pk` (second priority), then `custom_name` (third priority). All other custom root fields (`select_aggregate`, `insert`, `update`, `delete`) are not mapped to any Provisa equivalent.

**Use case:** Hasura v2 parity — table alias is derived from the select root field, the most common customization point, without requiring operators to manually specify aliases.

**Code:** `provisa/hasura_v2/mapper.py`

**Tests:** `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/unit/test_hasura_v2.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-628 · Hasura Migration Converters {#REQ-628}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When converting DDN HML, if a Model references an ObjectType name that is not found in any scanned .hml file, that table is skipped and a warning is emitted to the WarningCollector; the conversion continues rather than aborting.

**Use case:** Partial DDN projects (where some HML files are missing) produce a best-effort output with actionable warnings rather than a hard failure.

**Code:** `provisa/ddn/mapper.py`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`, `tests/steps/steps_hasura_migration_converters.py`

### REQ-629 · Hasura Migration Converters {#REQ-629}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Subgraph assignment for DDN HML documents is derived from the directory structure, not from a `subgraph` field in the document. The first directory component under the project root is taken as the subgraph name. Files under a `globals/` directory are assigned the `globals` subgraph and excluded from domain discovery.

**Use case:** Correct subgraph-to-domain mapping without requiring stewards to add metadata to individual HML files.

**Code:** `provisa/ddn/parser.py`

**Tests:** `tests/unit/test_ddn.py`, `tests/unit/test_ddn_converter.py`, `tests/unit/test_hasura_converter.py`

## 15. Proposed Features

### REQ-684 · Encryption {#REQ-684}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

An EncryptionService abstraction with pluggable implementations — LocalKeychain (AES-256-GCM, key from OS keychain), AwsKms, AzureKeyVault, and NullEncryption (dev/test passthrough). All encrypt/decrypt operations route through this interface. Configured via provisa.yaml encryption.provider and encryption.key_id.

**Use case:** Single interface decouples encryption policy from storage and transport layers, enabling the same application code to run encrypted on desktop, SMB SaaS, and enterprise BYOK deployments.

**Code:** `provisa/encryption/service.py`, `provisa/encryption/providers.py`, `provisa/encryption/factory.py`

**Tests:** `tests/unit/test_encryption.py`

### REQ-685 · Encryption {#REQ-685}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Envelope encryption for all encrypted payloads. A unique Data Encryption Key (DEK) is generated per payload, used to encrypt data locally with AES-256-GCM in streaming chunks. The DEK is then encrypted by the org master key via KMS. Storage format is encrypted_DEK | IV | ciphertext. DEKs are cached in-process with a short TTL to minimize KMS round-trips. Chunk size is configurable (default 64KB) to bound memory usage on large payloads.

**Use case:** Prevents large payload memory issues and excessive KMS costs. One KMS call per payload, not per row. Streaming chunks mean memory overhead is constant regardless of payload size.

**Code:** `provisa/encryption/envelope.py`

**Tests:** `tests/unit/test_encryption.py`

### REQ-686 · Encryption {#REQ-686}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Application-layer column encryption for sensitive PG data. API keys and RLS rules are encrypted via EncryptionService before write and decrypted after read; encrypted columns stored as BYTEA. Structural metadata (table names, column names) remains plaintext to support query planning. DONE (2026-07) — the two sensitive DB-resident columns are both encrypted at rest. (1) api_sources.auth (keys/tokens) is BYTEA, encrypted on write (openapi register) via the process-configured EncryptionService accessor and decrypted before use (api_source loader). (2) rls_rules.filter_expr — the RLS predicate is injected as SQL by inject_rls at every governance read, so it is sensitive metadata. Encryption is confined to the repository boundary (provisa/core/repositories/rls.py) where upsert encrypts and get_for_table_role/list_for_role/ list_all decrypt. The two raw reads that bypassed the repo (the app startup RLS loader and the admin GraphQL rls_rules field) were funnelled through list_all so decryption is applied uniformly; upsert is the single write choke point (config_loader + admin both use it). The meta-RLS projection exposes the row over the introspection surfaces as ciphertext, consistent with the threat model. NullEncryption passthrough stores plaintext-bytes in the same BYTEA column, so the encrypted path is exercised identically whether or not a provider is configured. Both proven end-to-end against the live Postgres store (ciphertext at rest, decrypted round-trip through the repo, wrong master key rejected). COMPLETE — these are the only sensitive DB-resident columns: the sources and kafka_sources tables persist no credentials at all (the schema stores host/username/auth_type only; passwords/keys are never written, resolved at runtime via the secrets provider), and there is no tenant_config table. Nothing else in the config DB requires column encryption.

**Use case:** Compromised PG instance exposes only opaque blobs for sensitive fields. Defense in depth beyond schema-per-org isolation.

**Code:** `provisa/encryption/runtime.py`, `provisa/api_source/loader.py`, `provisa/openapi/register.py`, `provisa/core/repositories/rls.py`, `provisa/core/schema_org.py`, `provisa/core/schema.sql`, `provisa/api/app.py`, `provisa/api/admin/schema.py`

**Tests:** `tests/integration/test_api_auth_encryption.py`, `tests/integration/test_rls_filter_encryption.py`, `tests/unit/test_encryption_runtime.py`, `tests/unit/test_rls.py`

### REQ-687 · Encryption {#REQ-687}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

DECIDED (bulk-only, client-held key): encrypt ONLY the S3/Iceberg BULK-RESULT payloads emitted by the large-result redirect path (provisa/executor/redirect.py), NOT the Trino non-SQL query cache. Mechanism = client-side envelope encryption: Provisa generates a per-result DEK, encrypts the serialized body (AES-256-GCM) via EncryptionService BEFORE upload, and stores CIPHERTEXT in S3. The presigned GET URL therefore delivers ciphertext; the client must possess the DEK to decrypt. The DEK travels to the client over the authenticated redirect response channel (not via S3), so a leaked presigned URL or a directly-accessed bucket object is useless without it. This is stronger than object-store SSE (which trusts the store to hold plaintext + the key). Presigned-URL compatibility is preserved precisely because Provisa hands back ciphertext — no SSE-C headers on the GET are required. WHY NOT the Trino non-SQL cache: that cache exists so Trino can QUERY cached rows as SQL (Phase-2 pushdown applies compiled WHERE/ORDER BY/LIMIT against the cache table; provisa/api_source/ trino_cache.py). Encrypting those payload columns makes them opaque to Trino — it can no longer filter/sort/join them, defeating the cache. The query cache is instead protected by short TTL + Trino/PG access control, and is explicitly OUT of scope for payload encryption. KEY-DELIVERY (IMPLEMENTED — role-bound grant + authenticated unwrap): the redirect response carries a GRANT = the raw per-result DEK sealed together with the creating role under the master key (a second envelope blob). To decrypt, the client POSTs the grant to the authenticated POST /data/redirect/unwrap; the server opens it under the MasterKeyProvider, verifies the caller is the creating role (or holds ADMIN/SUPERADMIN), and returns the DEK. The client then AES-256-GCM decrypts the iv/ciphertext parsed from the self-describing S3 blob (provisa.encryption. split_envelope). Possession of the presigned URL + grant alone is useless — decrypt-time authz is enforced AND the grant is integrity-protected so it cannot be re-scoped to another role. Rejected alternatives: inline raw DEK (no decrypt-time authz); SSE-C pre-shared key (breaks plain-HTTP clients). SECOND-CHOICE fallback, not selected: sensitive-data column-level encryption only. IMPLEMENTATION: opt-in via PROVISA_REDIRECT_ENCRYPT (default off, fail-closed — requires a real envelope provider); provisa/executor/redirect.py::_encrypt_payload seals the grant; endpoint at provisa/api/data/endpoint.py::redirect_unwrap. Proven against live MinIO end-to-end (tests/integration/test_redirect_encryption_minio.py) + unit round-trip (tests/unit/test_redirect_encryption.py).

**Use case:** Bulk result objects in S3/Iceberg are unreadable from the object store or a leaked presigned URL without the per-result DEK, which is released only over the authenticated redirect channel.

**Code:** —

**Tests:** `tests/unit/test_redirect_encryption.py`, `tests/integration/test_redirect_encryption_minio.py`

### REQ-688 · Encryption {#REQ-688}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Redis hot table payloads (CTE fragment row data) encrypted via EncryptionService at _store_rows() before write and decrypted at get_rows() after read. Redis ACL prefix isolation and payload encryption operate as two independent controls. Encrypted payload stored as bytes blob under the existing key scheme.

**Use case:** Redis ACL isolation prevents accidental cross-tenant key access. Payload encryption ensures that even a Redis ACL misconfiguration does not expose plaintext row data.

**Code:** `provisa/cache/hot_tables.py`

**Tests:** `tests/unit/test_hot_tables_encryption.py`

### REQ-689 · Encryption {#REQ-689}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Audit log query text column encrypted via EncryptionService before write and decrypted on authorised admin reads. Query text may reveal schema shape and data intent. Stored as BYTEA.

**Use case:** Audit log compliance records must not expose query semantics to a party who has PG access but not the org encryption key.

**Code:** `provisa/audit/query_log.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_audit_query_encryption.py`, `tests/integration/test_audit_encryption.py`

### REQ-690 · Encryption {#REQ-690}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Client-side decryption in the JDBC driver. Provisa backend passes encrypted blobs through without decrypting. Driver decrypts result columns flagged as encrypted via column metadata using EncryptionService (Java) configured via connection URL params (kms_provider, kms_key_arn). DEK cached in driver process with TTL. Backend never sees plaintext query results.

**Use case:** Strongest threat posture — a fully compromised Provisa backend cannot read query result data. Modelled on MongoDB CSFLE pattern.

**Code:** `jdbc-driver/src/main/java/io/provisa/jdbc/EnvelopeDecryptor.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/KmsProvider.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/LocalKmsProvider.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/AwsKmsProvider.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/AzureKmsProvider.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/GcpKmsProvider.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/KmsProviders.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/ArrowStreamResultSet.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/ProvisaConnection.java`, `jdbc-driver/src/main/java/io/provisa/jdbc/ProvisaDriver.java`, `jdbc-driver/pom.xml`

**Tests:** `jdbc-driver/src/test/java/io/provisa/jdbc/AwsKmsProviderTest.java`, `jdbc-driver/src/test/java/io/provisa/jdbc/AzureKmsProviderTest.java`, `jdbc-driver/src/test/java/io/provisa/jdbc/GcpKmsProviderTest.java`, `jdbc-driver/src/test/java/io/provisa/jdbc/KmsProvidersTest.java`, `jdbc-driver/src/test/java/io/provisa/jdbc/EnvelopeDecryptorTest.java`

### REQ-691 · Encryption {#REQ-691}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Client-side decryption in the Python client (DB-API, SQLAlchemy dialect, ADBC). EncryptionService Python implementation shared across dbapi.py, sqlalchemy_dialect.py, and adbc.py. KMS provider and key configured via connection params. DEK cached in process with TTL.

**Use case:** Python users (pandas, notebooks, data pipelines) get same client-side decrypt guarantees as JDBC users without a separate decrypt step.

**Code:** `provisa-client/provisa_client/encryption.py`, `provisa-client/provisa_client/dbapi.py`, `provisa-client/provisa_client/adbc.py`, `provisa-client/provisa_client/sqlalchemy_dialect.py`

**Tests:** `tests/unit/test_client_encryption.py`

### REQ-692 · Encryption {#REQ-692}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Client-side decryption for GraphQL clients via a thin wrapper. Provisa GraphQL schema marks encrypted fields with an @encrypted directive. The Provisa GQL client wrapper intercepts responses and decrypts flagged fields using EncryptionService before returning to caller. Scoped to non-browser clients only — browser-side KMS access weakens the threat model.

**Use case:** GraphQL API consumers (non-browser) participate in client-side decrypt without backend plaintext exposure.

**Code:** `provisa-client/provisa_client/graphql_decrypt.py`, `provisa/api/data/encrypted_directive.py`, `provisa/api/data/sdl.py`

**Tests:** `tests/unit/test_client_encryption.py`, `tests/unit/test_high_security_mode.py`

### REQ-693 · Encryption {#REQ-693}

**Status:** ✅ complete · **Priority:** MAY · **Type:** constraint

High-security mode (security.mode=high in provisa.yaml). When enabled — pgwire server is not started; REST and GraphQL data API endpoints return 403; JDBC and Python client connections without kms_key_arn are rejected at auth. Only JDBC and Python clients with client-side decrypt configured may connect. Provisa backend handles only encrypted blobs and query planning metadata.

**Use case:** Regulated or airgapped environments where the Provisa backend must not handle plaintext data under any circumstances.

**Code:** `provisa/security/high_security.py`, `provisa/api/app.py`, `provisa/api/app_startup.py`, `provisa/api/app_loaders.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_high_security_mode.py`

### REQ-694 · Encryption {#REQ-694}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Client-owned KMS key model. Org master key is a CMK in the client's own cloud account (AWS KMS, Azure Key Vault, or GCP KMS). Client grants Provisa a scoped role permission (kms:Decrypt and kms:GenerateDataKey) via cross-account IAM or equivalent. Provisa never stores raw key material. Client revokes grant to immediately render all their data unreadable. Supported for both SMB managed tier and enterprise BYOB tier.

**Use case:** Client holds the kill switch — revoking KMS access immediately cuts Provisa's ability to read their data without requiring data deletion or migration.

**Code:** `provisa-client/provisa_client/encryption.py`

**Tests:** `tests/unit/test_client_encryption.py`

## 11. Platform, Infrastructure & Delivery

### REQ-695 · Multi-Tenancy {#REQ-695}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

All internal PostgreSQL tables (semantic metadata, non-SQL cache, audit log, materialized view definitions) are scoped to a per-org schema named org_<org_id>. The asyncpg connection pool sets search_path=org_<org_id> on every connection via the connection init hook. A default org_id of "default" is used when ORG_ID env var is not set, producing schema org_default. Existing single-org deployments are unaffected — they transparently use org_default.

**Use case:** Schema-per-org provides structural isolation between tenants on a shared PostgreSQL instance. Accidental cross-tenant data access requires bypassing both the search_path and the role-level grants.

**Code:** `provisa/core/db.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_org_isolation.py`, `tests/unit/test_table_search.py`

### REQ-696 · Multi-Tenancy {#REQ-696}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Platform tables (tenants, tenant_config) reside in a dedicated platform schema, never touched by org search_path. The platform schema has a separate asyncpg pool or uses fully-qualified queries (platform.tenants). Org schemas cannot reference or join to the platform schema.

**Use case:** Tenant registry and billing data must not co-mingle with per-org application data. A compromised org connection cannot enumerate other tenants.

**Code:** `provisa/core/db.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_org_isolation.py`, `tests/unit/test_table_search.py`

### REQ-697 · Multi-Tenancy {#REQ-697}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

init_schema() in provisa/core/db.py accepts an org_id parameter. It creates the org_<org_id> schema if not exists, then runs core schema SQL, audit schema SQL, and MV schema SQL within that schema. Called at startup from _init_pg_pool_and_schema() in provisa/api/app.py using ORG_ID env var (default "default").

**Use case:** Schema creation is idempotent and org-scoped from first boot. No manual schema setup required for new tenant provisioning.

**Code:** `provisa/core/db.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_org_isolation.py`

### REQ-698 · Multi-Tenancy {#REQ-698}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The Trino non-SQL cache schema is org-scoped. cache_location() in provisa/api_source/trino_cache.py accepts org_id and sets cache_schema=org_<org_id>_api_cache. All callers (endpoint.py, router_integration.py, cypher_router.py) pass org_id from request context. The provisa_admin Trino catalog requires no change as Trino uses fully-qualified catalog.schema.table references.

**Use case:** Non-SQL cached rows for different orgs land in separate PG schemas within the provisa_admin catalog. A Trino query for org A cannot reference org B cache tables without an explicit schema qualifier.

**Code:** `provisa/api_source/trino_cache.py`, `provisa/api/data/endpoint.py`

**Tests:** `tests/unit/test_catalog_cache.py`, `tests/unit/test_org_isolation.py`

### REQ-699 · Multi-Tenancy {#REQ-699}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Each org has a dedicated PostgreSQL role named role_<org_id> with USAGE and CREATE on org_<org_id> schema only. The asyncpg pool connects as this role. The role has no access to platform schema or any other org schema.

**Use case:** Database-level role scoping provides a second isolation layer below search_path. A malformed query using an unqualified table name cannot accidentally read another org's data even if search_path is misconfigured.

**Code:** `provisa/core/db.py`

**Tests:** `tests/unit/test_org_isolation.py`

### REQ-700 · Multi-Tenancy {#REQ-700}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Extends [REQ-595](#REQ-595) for on-prem multi-org deployments. Redis cache keys, hot table keys, APQ keys, and query result keys are all prefixed with org_id (org:<org_id>:...) rather than tenant_id. Redis ACL rules scope each org's credentials to their key prefix. One org's cache flush or eviction never touches another org's keys. Requires wiring org_id through check_cache/store_result in cache/middleware.py (the same gap blocking [REQ-595](#REQ-595) completion).

**Use case:** Redis key prefix isolation complements PG schema isolation ([REQ-695](#REQ-695)). Both layers must be independently correct — ACL misconfiguration in one should not cascade to the other.

**Code:** `provisa/cache/middleware.py`, `provisa/cache/store.py`, `provisa/core/org_provisioning.py`

**Tests:** `tests/integration/test_cache_store.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_org_isolation.py`

### REQ-701 · Multi-Tenancy {#REQ-701}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Org provisioning creates all required infrastructure atomically — org PG schema, PG role, Redis ACL user, and Trino cache schema — in a single provisioning transaction or compensating workflow. Partial provisioning leaves no orphaned resources. Deprovisioning drops all org resources in reverse order.

**Use case:** Atomic provisioning prevents half-initialised orgs from entering a queryable but misconfigured state. Clean deprovisioning enables tenant offboarding without manual cleanup.

**Code:** `provisa/core/org_provisioning.py`

**Tests:** `tests/unit/test_org_isolation.py`

### REQ-702 · Multi-Tenancy {#REQ-702}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Demo init code and static demo assets are org-scoped. _seed_built_in_sources() passes state.org_id to _seed_meta_domain(); _seed_ops_pg() inherits the correct schema via the asyncpg connection pool search_path set by [REQ-695](#REQ-695). Static demo file assets (Parquet, SQLite, CSV) contain no schema references. No migration tool is required — org scoping is handled entirely through the connection pool and seed function wiring.

**Use case:** Demo environments must participate in org isolation ([REQ-695](#REQ-695) through [REQ-701](#REQ-701)) without special-casing. Seeded data lands in org_<org_id> automatically via the pool init hook, ensuring demo deployments and production multi-tenant deployments use the same code path.

**Code:** `provisa/api/app.py`

**Tests:** `tests/unit/test_org_isolation.py`

## 10. Execution & Query Planning

### REQ-703 · High Availability {#REQ-703}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Provisa implements a two-tier recovery model for all deployment modes. Tier 1 retries read operations for up to 30 seconds on transient errors using exponential backoff with full jitter (configurable via PROVISA_RETRY_BUDGET_SECS). Tier 2 uses an internal watcher to detect and restart failed software components within 2–3 minutes. Write operations are never retried internally. Machine-level and cluster-level failures are the customer's responsibility.

**Use case:** Read-heavy query workloads may encounter transient coordinator unavailability, brief network partitions, or temporary query queue saturation. Automatic retry provides resilience without requiring client-side retry logic for all read operations. Software component failures are self-healing within the deployment boundary; cluster-wide HA is configured by the customer based on their failure domain (single-node, VM/metal, K8s).

**Code:** `provisa/executor/trino.py`, `provisa/executor/direct.py`, `provisa/scheduler/jobs.py`, `terraform/aws/main.tf`, `terraform/gcp/main.tf`, `terraform/azure/main.tf`

**Tests:** `tests/unit/test_high_availability.py`

## 8. Client Access & Protocols

### REQ-711 · ADBC Client {#REQ-711}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

adbc_connect() must accept an optional port parameter (default 8815) so callers can connect to Arrow Flight servers running on non-default ports. The hardcoded port 8815 must be replaced with this configurable value throughout provisa_client/adbc.py.

**Use case:** Operators who cannot run the Flight server on port 8815 (due to port conflicts or network policy) can connect via ADBC without falling back to DB-API or SQLAlchemy.

**Code:** `provisa_client/adbc.py`

**Tests:** `provisa-client/tests/test_adbc.py`

## 6. Execution, Routing, Caching & Performance

### REQ-712 · Cache {#REQ-712}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

check_cache and store_result in provisa/cache/middleware.py pass tenant_id (org_id) through to RedisCacheStore.get/set, so the per-tenant cache key prefix implemented in the store is applied at the call site and multi-tenant cache key isolation is active for query result caching.

**Use case:** Tenant-prefixed cache keys at the middleware layer ensure one tenant's cached query results can never be returned to another tenant.

**Code:** `provisa/cache/middleware.py`

**Tests:** `tests/integration/test_cache_store.py`, `tests/unit/test_apq_cache.py`, `tests/unit/test_cache_key.py`, `tests/unit/test_cache_store.py`, `tests/unit/test_mv_tenant_isolation.py`

## 10. Graph Export

### REQ-713 · Neo4j Export {#REQ-713}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Neo4j export endpoint accepts graph nodes and edges via POST /data/neo4j-export, including target URL, credentials, database, and node/edge lists.

**Use case:** Provisa users can export subgraphs queried via Cypher to external Neo4j servers for analysis, backup, or integration with other tools.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-714 · Neo4j Export {#REQ-714}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Nodes are MERGE'd into Neo4j using _provisa_id as the deduplication key, with labels derived from tableLabel or compound "Domain:Table" label fields.

**Use case:** Preserves node identity across export runs and handles domain-scoped node types.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-715 · Neo4j Export {#REQ-715}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Node properties are written to Neo4j using SET n += {...} with Cypher literal encoding to safely handle nested objects, arrays, nulls, and special types.

**Use case:** Ensures property values (booleans, numbers, strings, nested structures) are correctly transmitted to Neo4j without injection or serialization errors.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-716 · Neo4j Export {#REQ-716}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Edges are MERGE'd by matching source and target nodes on _provisa_id, with relationship type preserved from the source graph.

**Use case:** Creates or updates edges in Neo4j with correct source and target identity, supporting graph structure reconstruction.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-717 · Neo4j Export {#REQ-717}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Neo4j export endpoint uses HTTP Basic authentication via Authorization header (base64-encoded username:password) to authenticate with the target server.

**Use case:** Securely authenticates to external Neo4j instances without embedding credentials in the URL.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-718 · Neo4j Export {#REQ-718}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Connection errors to Neo4j return HTTP 502; timeouts return 504; authentication failures return 401.

**Use case:** Clients can distinguish transient (502, 504) from permanent (401) failures and retry or fix credentials appropriately.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-719 · Neo4j Export {#REQ-719}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Neo4j export endpoint returns {imported: N, errors: [...]} structure, allowing partial success where some statements fail while others succeed.

**Use case:** Clients can detect which nodes/edges failed to import and retry or log them without treating the entire export as a failure.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

### REQ-720 · Neo4j Export {#REQ-720}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

E2E export flow batches nodes and edges separately to avoid transaction timeouts, with default batch size of 200 nodes per request.

**Use case:** Allows exporting large graphs by chunking them into manageable transaction sizes, avoiding Neo4j timeout errors on large statements.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_export.py`

## 4. Source Connectors

### REQ-721 · Splunk Connector {#REQ-721}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Splunk is registered as a source type and exposed via Apache Calcite connector (kenstott/calcite), enabling Splunk search results to be queried as tables through Trino.

**Use case:** Allows users to register Splunk instances as data sources and query their search results via the Provisa GraphQL API.

**Code:** `provisa/core/models.py`, `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/splunk-connector.spec.ts`, `tests/unit/test_splunk_connector.py`

### REQ-722 · Splunk Connector {#REQ-722}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Splunk connector accepts a URL (from base_url or constructed via https://host:port, default port 8089) to connect to the Splunk management API.

**Use case:** Provides a flexible connection method: users can specify a complete base_url or rely on host:port defaults to simplify configuration.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/splunk-connector.spec.ts`, `tests/unit/test_splunk_connector.py`

### REQ-723 · Splunk Connector {#REQ-723}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Splunk authentication: when use_token=true (default), the password field is passed as token; when use_token=false, username and password fields are passed as separate credentials.

**Use case:** Splunk supports both token-based (preferred for API automation) and username/password authentication, allowing users to choose their auth method.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/splunk-connector.spec.ts`, `tests/unit/test_splunk_connector.py`

### REQ-724 · Splunk Connector {#REQ-724}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Splunk connector supports optional app and datamodel-filter properties from source.database and source.mapping, plus disable-ssl-validation for self-signed certificates.

**Use case:** Allows filtering queries to a specific Splunk app and disabling SSL validation for development/test environments with self-signed certs.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/splunk-connector.spec.ts`, `tests/unit/test_splunk_connector.py`

### REQ-725 · Splunk Connector {#REQ-725}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

Splunk connector always sets case-insensitive-name-matching=true, allowing case-insensitive table and column lookups.

**Use case:** Ensures consistent behavior when users reference Splunk table names regardless of case, matching Splunk's own case-insensitive semantics.

**Code:** `provisa/core/catalog.py`

**Tests:** `tests/unit/test_splunk_connector.py`

### REQ-726 · SharePoint Connector {#REQ-726}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SharePoint is registered as a source type in the Provisa data connector registry and can be configured as a queryable data source via the Apache Calcite sharepoint connector.

**Use case:** Users can register SharePoint lists as tables in their Provisa domain, enabling SQL queries and GraphQL access to SharePoint data.

**Code:** `provisa/core/models.py`

**Tests:** `provisa-ui/e2e/sharepoint-connector.spec.ts`, `tests/unit/test_sharepoint_connector.py`

### REQ-727 · SharePoint Connector {#REQ-727}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SharePoint connector supports two authentication methods: CLIENT_CREDENTIALS (default) and certificate-based authentication via PFX certificate.

**Use case:** Different organizational security policies require different auth methods; CLIENT_CREDENTIALS suits service accounts while certificates support PKI-based auth.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/sharepoint-connector.spec.ts`, `tests/unit/test_sharepoint_connector.py`

### REQ-728 · SharePoint Connector {#REQ-728}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SharePoint connector builds connection properties from source base_url/host (site-url), username (client-id), password (client-secret), database field (tenant-id), and optional mapping fields (certificate_path, certificate_password, auth_type).

**Use case:** Consolidates SharePoint auth and connection configuration into the standard Provisa Source model, allowing centralized secret management.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/sharepoint-connector.spec.ts`, `tests/unit/test_sharepoint_connector.py`

### REQ-729 · SharePoint Connector {#REQ-729}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Secret values in SharePoint connector mapping (auth_type, certificate_path, certificate_password) are resolved via the secrets engine before being passed to the Calcite connector.

**Use case:** Prevents plaintext credentials from being hardcoded in configuration, enabling vault/environment variable secret injection.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/sharepoint-connector.spec.ts`, `tests/unit/test_sharepoint_connector.py`

### REQ-730 · SharePoint Connector {#REQ-730}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

SharePoint connector always enables case-insensitive-name-matching when building connection properties to the Calcite connector.

**Use case:** SharePoint list names are case-insensitive; enabling this flag ensures consistent behavior across queries regardless of case variation in table/column references.

**Code:** `provisa/core/catalog.py`

**Tests:** `tests/unit/test_sharepoint_connector.py`

### REQ-731 · SharePoint Connector {#REQ-731}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SharePoint lists are enumerated as schemas and exposed as queryable tables via Trino, allowing users to discover available lists and register them as Provisa tables.

**Use case:** Users can browse SharePoint lists through the Provisa UI without manual configuration, improving UX for data source discovery.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/sharepoint-connector.spec.ts`, `tests/unit/test_sharepoint_connector.py`

### REQ-732 · SharePoint Connector {#REQ-732}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

SharePoint tables can be registered in Provisa with known column definitions when the Calcite connector does not expose information_schema.columns.

**Use case:** Allows users to manually define columns obtained from the Microsoft Graph API, bypassing the connector limitation and enabling table registration and queries.

**Code:** `provisa/core/catalog.py`

**Tests:** `provisa-ui/e2e/sharepoint-connector.spec.ts`, `tests/unit/test_sharepoint_connector.py`

## 3. Source Registration & Data Modeling

### REQ-733 · Aggregate Configuration {#REQ-733}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Table edit form exposes `enable_aggregates` and `enable_group_by` as checkboxes in the admin UI. Stewards toggle these per-table settings to enable or disable aggregate and group-by root fields for the schema.

**Use case:** Admin UI checkboxes give stewards an intuitive way to manage aggregate/group-by settings without manually editing YAML configuration files.

**Code:** `provisa/api/admin/types.py`, `provisa/api/admin/db_queries.py`, `provisa-ui/src`

**Tests:** `provisa-ui/e2e/tables-enable-aggregates.spec.ts`

### REQ-734 · Aggregate Configuration {#REQ-734}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Table configuration persists `enable_aggregates` and `enable_group_by` boolean columns in the `registered_tables` database table. Both columns default to `false`. Values are readable via the table edit API endpoint and reflect the current configuration state.

**Use case:** Persistent storage of aggregate/group-by settings ensures steward configuration choices survive across deployments and are accessible via admin APIs.

**Code:** `provisa/api/admin/db_queries.py`, `provisa/core/models.py`

**Tests:** `provisa-ui/e2e/tables-enable-aggregates.spec.ts`, `tests/unit/test_aggregate_config.py`

## 1. Access Governance & Security

### REQ-740 · Security {#REQ-740}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Masking SELECT expressions only; WHERE, JOIN ON, and other predicates use physical unmasked columns unchanged. Masking injection operates on the SELECT projection list exclusively, preserving query filtering and join semantics.

**Use case:** Separating masked SELECT projections from unmasked predicates ensures query filters remain executable while users see masked values.

**Code:** `provisa/compiler/mask_inject.py`, `provisa/compiler/stage2.py`

**Tests:** `tests/unit/test_masking_edge_cases.py`, `tests/e2e/test_masking.py`, `tests/integration/test_security_integration_extra.py`

### REQ-741 · Security {#REQ-741}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Column masking output uses ANSI SQL dialects (REGEXP_REPLACE, DATE_TRUNC, SQL literals) independent of source type. Dialect-specific transpilation via SQLGlot occurs downstream, after masking injection.

**Use case:** Dialect-agnostic masking generation ensures consistent governance behavior across heterogeneous federated sources (PostgreSQL, MySQL, Snowflake, BigQuery).

**Code:** `provisa/security/masking.py`, `provisa/compiler/mask_inject.py`

**Tests:** `tests/unit/test_masking_edge_cases.py`, `tests/e2e/test_masking.py`

### REQ-742 · Security {#REQ-742}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Masking respects column type constraints — NULL masks allowed only on nullable columns; regex and truncate masks validated for applicable types (regex on text, truncate on temporal). Validation enforced at config load time.

**Use case:** Type-aware masking validation at config time prevents runtime errors and ensures masks produce valid SQL.

**Code:** `provisa/security/masking.py`

**Tests:** `tests/unit/test_masking_edge_cases.py`, `tests/e2e/test_masking.py`

### REQ-743 · Security {#REQ-743}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Masking constant expressions must emit syntactically valid SQL for their type — numeric literals without quotes, string literals with single quotes (escaped apostrophes doubled), NULL keyword (not quoted), boolean TRUE/FALSE keywords (not Python string literals). Special constants MAX/MIN resolve to integer subtype bounds (tinyint=127, smallint=32767, integer=2147483647, bigint=9223372036854775807).

**Use case:** Correct literal syntax ensures masked values are valid SQL across all dialects without requiring downstream escaping.

**Code:** `provisa/security/masking.py`

**Tests:** `tests/unit/test_masking_edge_cases.py`, `tests/e2e/test_masking.py`

### REQ-744 · Security {#REQ-744}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Masking preserves query structure — ORDER BY, LIMIT, GROUP BY, and other clauses remain unchanged; only SELECT projection is rewritten. Masking returns a new CompiledQuery object (immutable transformation), never mutating the input.

**Use case:** Immutable masking injection and preservation of query structure ensure correct sorting, pagination, and aggregation behavior.

**Code:** `provisa/compiler/mask_inject.py`

**Tests:** `tests/unit/test_masking_edge_cases.py`, `tests/e2e/test_masking.py`, `tests/integration/test_security_integration_extra.py`

### REQ-745 · Security {#REQ-745}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Role-based masking — different roles can have different masks for the same column; admin roles with no masking rules see raw values; other roles see role-specific masks (regex, constant, or truncate). Selection determined by (table_id, role) key in masking rules dictionary.

**Use case:** Role-specific masking lets different users see different representations of the same data based on their role permissions.

**Code:** `provisa/compiler/mask_inject.py`, `provisa/compiler/stage2.py`

**Tests:** `tests/unit/test_masking_edge_cases.py`, `tests/e2e/test_masking.py`, `tests/integration/test_security_integration_extra.py`

### REQ-746 · Security {#REQ-746}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Capability enforcement via check_capability and has_capability functions — capabilities are independently assigned per role (query_development, source_registration, table_registration, relationship_definition, security_configuration, query_authorization, ignore_relationships). Admin capability grants all permissions. InsufficientRightsError raised on missing capability.

**Use case:** Fine-grained, independently verifiable capabilities prevent privilege escalation.

**Code:** `provisa/security/rights.py`

**Tests:** `tests/unit/test_rights.py`

### REQ-747 · Security {#REQ-747}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

SQL validator bypass for remote same-source relationship pairs — when bypass_uncovered_relationships=True and both tables share the same remote source (graphql_remote or grpc_remote from same source_id), uncovered joins pass V002 validation. Cross-source joins (local↔remote or different remote sources) always require a covered_pair registration. Registered pairs always require correct column matches regardless of bypass flag.

**Use case:** Remote relationship bypass allows federated GraphQL/gRPC sources to define their own relationship model without Provisa relationship registration.

**Code:** `provisa/compiler/sql_validator.py`

**Tests:** `tests/unit/test_sql_validator_bypass.py`

### REQ-748 · Security {#REQ-748}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Inverse relationship collision fix — joins producing inverse relationships (e.g., pets←assignments, assignments→pets) must emit distinct rel_types based on target label, not just column pairs. Keyed by (source_label, target_label, join_columns) to ensure reverse joins resolve to correct relationship types.

**Use case:** Correct inverse relationship resolution prevents relationship type confusion in bidirectional federation scenarios.

**Code:** `provisa/cypher/sql_to_cypher.py`

**Tests:** `tests/unit/test_sql_validator_bypass.py`, `tests/integration/test_security_integration_extra.py`

### REQ-749 · Security {#REQ-749}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Domain policy tri-state mode (legacy/single-domain/namespaced) — legacy mode (use_domains absent) stores declared domain_id verbatim; single-domain mode (use_domains=False) coerces all tables to default_domain; namespaced mode (use_domains=True) stores domain_id as declared. Reload validation sweep rejects pre-existing foreign domains when switching to single-domain mode.

**Use case:** Domain policy modes enable multi-tenancy architectures without breaking legacy single-domain deployments.

**Code:** `provisa/core/domain_policy.py`, `provisa/core/config_loader.py`

**Tests:** `tests/integration/test_domain_policy_integration.py`, `tests/unit/test_domain_policy.py`

## 5. Query Languages, Compilation & Operations

### REQ-750 · Cypher Query Frontend (Phase AU) {#REQ-750}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher graph variables (nodes, edges, paths) returned in the RETURN clause are serialized as JSON objects with canonical shape. Nodes include `id`, `label`, `tableLabel`, and `properties`. Edges include `identity`, `start`, `end`, `type`, `properties`, `startNode`, and `endNode`. Paths include `nodes`, `edges`, and `length`/`hops`.

**Use case:** Consistent JSON serialization of graph objects enables client libraries to deserialize typed node/edge/path representations.

**Code:** `provisa/cypher/assembler.py`, `provisa/cypher/graph_rewriter.py`

**Tests:** `tests/unit/test_cypher_assembler.py`, `tests/unit/test_cypher_graph_rewriter.py`, `tests/integration/test_cypher_integration_extra.py`, `provisa-ui/e2e/cypher-api.spec.ts`

### REQ-751 · Cypher Query Frontend (Phase AU) {#REQ-751}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Variable-length relationship patterns `[*1..n]` are compiled to recursive CTEs in Trino SQL. The CTE enforces max-hop bounds and returns edge sequences as JSON arrays. Multiple edge hops are preserved in the array for flat-join paths; recursive paths project NULL (intermediate edges not tracked in CTE).

**Use case:** Variable-length traversals enable clients to explore multi-hop relationships without writing nested joins.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `provisa-ui/e2e/cypher-variable-length-path.spec.ts`, `tests/integration/test_cypher_integration_extra.py`

### REQ-752 · Cypher Query Frontend (Phase AU) {#REQ-752}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Intermediate node property access in multi-hop MATCH patterns (e.g., `(p:Person)-[:WORKS_AT]->(c:Company)-[:HAS_DEPT]->(d:Department)`) resolves all three table aliases and preserves property references in WHERE and RETURN clauses. No aliasing conflicts occur.

**Use case:** Complex traversals require filtering and returning properties from intermediate nodes without manual projection.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-753 · Cypher Query Frontend (Phase AU) {#REQ-753}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Path object RETURN (e.g., `RETURN p`) emits a JSON_OBJECT with `nodes` (array of node objects), `edges` (array of edge objects), `start`/`end` (node identities), and `length` (hop count). For recursive CTE paths, `hops` field counts intermediate hops.

**Use case:** Returning entire paths as objects enables clients to navigate graph traversals without unpacking individual nodes/edges.

**Code:** `provisa/cypher/translator.py`, `provisa/cypher/graph_rewriter.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `provisa-ui/e2e/cypher-variable-length-path.spec.ts`, `tests/integration/test_cypher_integration_extra.py`

### REQ-754 · Cypher Query Frontend (Phase AU) {#REQ-754}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Correlated CALL subqueries (e.g., `CALL { WITH x MATCH (x)-[:REL]->(y) RETURN y }`) translate to CROSS JOIN LATERAL subqueries in Trino, preserving the outer variable binding.

**Use case:** Correlated subqueries enable filtering inner results based on outer row context without materializing cartesian joins.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-755 · Cypher Query Frontend (Phase AU) {#REQ-755}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Node label alternation (e.g., `(n:Label1|Label2)`) translates to UNION ALL branches in SQL, one per label candidate. Each branch includes only the relevant node type and relationships that originate from it.

**Use case:** Label alternation enables flexible node matching without explicit OR conditions.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-756 · Cypher Query Frontend (Phase AU) {#REQ-756}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Subquery predicates — EXISTS { ... }, COUNT { ... }, COLLECT { ... } — translate to correlated SQL subqueries. EXISTS becomes EXISTS (SELECT ...), COUNT becomes (SELECT count(...)), COLLECT becomes ARRAY(...).

**Use case:** Subquery predicates enable complex filtering based on related-node aggregations without intermediate MATCH clauses.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-757 · Cypher Query Frontend (Phase AU) {#REQ-757}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Map projections — `n { .prop1, .prop2 }`, `n { .* }`, `n { key: expr }` — translate to Trino MAP(...) function calls with ARRAY keys and ARRAY values. Star projections expand schema properties; custom keys are added as named entries.

**Use case:** Map projections enable client-friendly reshaping of returned objects without manual column aliasing.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-758 · Cypher Query Frontend (Phase AU) {#REQ-758}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Bidirectional edge traversal (e.g., `(a)-[]-(b)` without direction marker) expands to UNION ALL when multiple relationship directions exist in the schema. Each branch covers one direction; single-direction schemas skip the UNION.

**Use case:** Bidirectional traversal enables undirected graph exploration without explicit forward/backward branches.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-759 · Cypher Query Frontend (Phase AU) {#REQ-759}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Backward relationship traversal (e.g., `(c:Company)<-[:WORKS_AT]-(p:Person)`) inverts the join condition from the forward relationship definition. Join columns are swapped; ON condition evaluates to tgt.id = src.join_column.

**Use case:** Backward traversal enables reverse-direction path queries without schema bidirectionality.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-760 · Cypher Query Frontend (Phase AU) {#REQ-760}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

String, list, and numeric functions (LEFT, RIGHT, TRIM, SIZE, COUNT DISTINCT, REDUCE) translate directly to Trino equivalents or adapters. REDUCE syntax is normalized to Trino lambda form. Legacy `{param}` syntax is normalized to `$param` before parsing.

**Use case:** Function support enables rich query expressions without learning Trino-specific syntax.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-761 · Cypher Query Frontend (Phase AU) {#REQ-761}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Implicit GROUP BY is inferred in the RETURN clause when non-aggregated columns are mixed with aggregate functions. All non-aggregated columns in RETURN are included in GROUP BY without explicit declaration.

**Use case:** Implicit GROUP BY matches Neo4j Cypher semantics and reduces query verbosity.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_exprs.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-762 · Cypher Query Frontend (Phase AU) {#REQ-762}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

CASE expressions (both searched `CASE WHEN ... THEN ... ELSE ... END` and simple `CASE x WHEN ... THEN ... ELSE ... END`) translate directly to Trino CASE syntax. Multiple WHEN branches and optional ELSE are preserved.

**Use case:** CASE expressions enable conditional logic in projections and filters without external control flow.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-763 · Cypher Query Frontend (Phase AU) {#REQ-763}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

UNWIND clauses (e.g., `UNWIND [1,2,3] AS x`) translate to Trino UNNEST with CROSS JOIN. Parameters in UNWIND (e.g., `$list`) are passed through and referenced as positional parameters in the SQL output.

**Use case:** UNWIND enables iteration over lists without explicit array expansion in the application.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-764 · Cypher Query Frontend (Phase AU) {#REQ-764}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

IN list predicates (e.g., `n.age IN [25, 30, 35]`) translate directly to SQL IN clauses with literal or parameter values.

**Use case:** IN predicates enable efficient filtering on multi-value sets without repeated OR conditions.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-765 · Cypher Query Frontend (Phase AU) {#REQ-765}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Pattern comprehensions (e.g., `[(p)-[:WORKS_AT]->(c:Company) | c.name]`) translate to ARRAY(...SELECT...) subqueries with the pattern matched inside and the pipe expression projected.

**Use case:** Pattern comprehensions enable inline aggregation of related rows without explicit GROUP BY.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-766 · Cypher Query Frontend (Phase AU) {#REQ-766}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

length(p) function on a path variable returns the hop count (number of edges). For recursive CTE paths, it returns the `hops` field value; for flat-join paths, it returns 1 or the explicit edge count.

**Use case:** length(p) enables path-based filtering and ordering without explicit edge counting.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-767 · Cypher Query Frontend (Phase AU) {#REQ-767}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Recursive CTE shortestPath and allShortestPaths functions enforce max-hop bounds at compile time. shortestPath injects LIMIT 1 and ORDER BY hops; allShortestPaths orders by hops without LIMIT.

**Use case:** Shortest-path functions enable efficient shortest-path queries without client-side traversal algorithms.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-768 · Cypher Query Frontend (Phase AU) {#REQ-768}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Node graph variable deserialization from SQL result rows. JSON columns marked as NODE in graph_vars are parsed into Node objects with id, label, tableLabel, and properties extracted.

**Use case:** Node deserialization enables type-safe access to node objects in the assembler stage before HTTP response serialization.

**Code:** `provisa/cypher/assembler.py`

**Tests:** `tests/unit/test_cypher_assembler.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-769 · Cypher Query Frontend (Phase AU) {#REQ-769}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Edge graph variable deserialization from SQL result rows. JSON columns marked as EDGE are parsed into Edge objects with identity, type, start/end node references, and properties. Legacy (id/startNode/endNode) and Neo4j (identity/start/end) formats are both supported.

**Use case:** Edge deserialization enables type-safe access to relationship objects with start/end node tracking.

**Code:** `provisa/cypher/assembler.py`

**Tests:** `tests/unit/test_cypher_assembler.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-770 · Cypher Query Frontend (Phase AU) {#REQ-770}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Path graph variable deserialization from SQL result rows. Rows with _path_id and _depth columns are collapsed into single Path objects with nodes and edges arrays. Multiple paths with different _path_ids produce separate Path objects.

**Use case:** Path deserialization enables client-friendly navigation of multi-hop graph results without manual array unpacking.

**Code:** `provisa/cypher/assembler.py`

**Tests:** `tests/unit/test_cypher_assembler.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-771 · Cypher Query Frontend (Phase AU) {#REQ-771}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Variable-length relationship edge columns (e.g., `[c*..5]` variable) are deserialized as lists of Edge objects from JSON_ARRAY columns. NULL values from CTE null-projection are preserved as None. Empty lists result from zero-edge paths.

**Use case:** Variable-length edge deserialization enables structured access to multi-hop relationship sequences.

**Code:** `provisa/cypher/assembler.py`

**Tests:** `tests/unit/test_cypher_assembler.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-772 · Cypher Query Frontend (Phase AU) {#REQ-772}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Graph rewriter applies JSON object wrapping to all graph variables in the SELECT clause. Scalar columns remain unwrapped. The wrapping ensures nodes/edges/paths are serializable to JSON.

**Use case:** Graph rewriter preserves data integrity by ensuring all graph variables are JSON-compatible before HTTP serialization.

**Code:** `provisa/cypher/graph_rewriter.py`

**Tests:** `tests/unit/test_cypher_graph_rewriter.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-773 · Cypher Query Frontend (Phase AU) {#REQ-773}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Domain-scoped node projections (e.g., `MATCH (n:PetStore) RETURN n`) include all properties from nodes in the domain in the JSON_OBJECT output. Multiple node types within a domain produce UNION ALL branches, each populating the full property set for that type.

**Use case:** Domain-scoped queries enable exploration of all entities within a business domain without explicit type specification.

**Code:** `provisa/cypher/graph_rewriter.py`, `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_graph_rewriter.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-774 · Cypher Query Frontend (Phase AU) {#REQ-774}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Duplicate JSON key prevention. When a table's id_column differs from 'id' but the table has a column named 'id', the JSON_OBJECT projection emits only one 'id' key (the primary identity key, not the column).

**Use case:** Duplicate key prevention ensures well-formed JSON output from federated tables with unconventional id column naming.

**Code:** `provisa/cypher/graph_rewriter.py`

**Tests:** `tests/unit/test_cypher_graph_rewriter.py`

### REQ-775 · Cypher Query Frontend (Phase AU) {#REQ-775}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Anonymous all-rels path pattern `MATCH p=()-->()` (no node/edge labels or variables) generates a valid SQL subquery referencing all relationships without crashing. No empty table-prefix column references are emitted.

**Use case:** Anonymous paths enable exploratory graph queries without schema knowledge.

**Code:** `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_cypher_translator_paths.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-776 · Cypher Query Frontend (Phase AU) {#REQ-776}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

OPTIONAL MATCH chains (e.g., `OPTIONAL MATCH (a)-[r1]->(b) OPTIONAL MATCH (b)-[r2]->(c)`) translate to left-joined multi-hop paths. Each OPTIONAL MATCH becomes a separate LEFT JOIN in sequence, preserving null propagation for unmatched branches.

**Use case:** OPTIONAL MATCH chains enable exploratory traversals where intermediate results may be absent without failing the entire query.

**Code:** `provisa/cypher/translator.py`

**Tests:** `provisa-ui/e2e/cypher-breed-filter.spec.ts`, `tests/unit/test_cypher_frontend.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-777 · Cypher Query Frontend (Phase AU) {#REQ-777}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

UNION ALL queries (e.g., `MATCH (n) RETURN ... UNION ALL MATCH (m) RETURN ...`) preserve distinct column aliases in both branches. Result column order matches the first UNION branch. Queries with property filters on both nodes and edges execute without FederationError.

**Use case:** UNION ALL queries enable multi-entity result merging and cross-entity property filtering in a single query.

**Code:** `provisa/cypher/translator.py`

**Tests:** `provisa-ui/e2e/cypher-union-all-email.spec.ts`, `tests/unit/test_cypher_frontend.py`, `tests/integration/test_cypher_integration_extra.py`

### REQ-778 · Cypher Query Frontend (Phase AU) {#REQ-778}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher /query/cypher endpoint returns a typed response with `columns` (array of column names), `rows` (array of result objects), `error` (null or error detail), and `type` (always "cypher"). No row truncation or implicit result limiting occurs beyond query-specified LIMIT.

**Use case:** Consistent response shape enables client libraries to deserialize typed Cypher query results.

**Code:** `provisa/api/rest/cypher_router.py`, `provisa/api/data/endpoint.py`

**Tests:** `provisa-ui/e2e/cypher-breed-filter.spec.ts`, `provisa-ui/e2e/cypher-variable-length-path.spec.ts`, `provisa-ui/e2e/cypher-union-all-email.spec.ts`, `tests/unit/test_cypher_frontend.py`, `tests/integration/test_cypher_integration_extra.py`

## 10. UI & Admin Surfaces

### REQ-779 · Graph Explorer {#REQ-779}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Graph Explorer Favorites panel persists and displays user-saved Cypher queries with custom labels, stored in browser localStorage.

**Use case:** Users need quick access to frequently-used Cypher queries without re-typing them.

**Code:** `provisa-ui/src/components/graph-explorer`

**Tests:** `provisa-ui/e2e/graph-favorites.spec.ts`

### REQ-780 · Graph Explorer {#REQ-780}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Graph Explorer Favorites: hovering over a favorite item reveals action buttons (run, rename, delete) that were previously hidden.

**Use case:** Hover-reveal of action buttons keeps the favorites panel uncluttered while making operations discoverable.

**Code:** `provisa-ui/src/components/graph-explorer`

**Tests:** `provisa-ui/e2e/graph-favorites.spec.ts`

### REQ-781 · Graph Explorer {#REQ-781}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Graph Explorer Favorites: clicking a favorite's label loads the Cypher query into the editor for execution or modification.

**Use case:** Users can quickly load and run a saved query or modify it before execution.

**Code:** `provisa-ui/src/components/graph-explorer`

**Tests:** `provisa-ui/e2e/graph-favorites.spec.ts`, `tests/unit/test_graph_explorer_favorites.py`

### REQ-782 · Graph Explorer {#REQ-782}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Graph Explorer Favorites: inline rename input allows editing a favorite's label with Enter to commit and Escape to cancel; label persists in localStorage.

**Use case:** Users can update favorite labels to keep them organized and meaningful as their work evolves.

**Code:** `provisa-ui/src/components/graph-explorer`

**Tests:** `provisa-ui/e2e/graph-favorites.spec.ts`, `tests/unit/test_graph_explorer_favorites.py`

### REQ-783 · Graph Explorer {#REQ-783}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Graph Explorer Favorites: delete button removes a favorite from the panel and localStorage immediately.

**Use case:** Users can clean up outdated or unwanted saved queries.

**Code:** `provisa-ui/src/components/graph-explorer`

**Tests:** `provisa-ui/e2e/graph-favorites.spec.ts`, `tests/unit/test_graph_explorer_favorites.py`

## 5. Query Languages, Compilation & Operations

### REQ-784 · Cypher Graph Analytics {#REQ-784}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Auto-impute endpoint generates relationship edges for visible graph nodes based on their labels and the schema relationship map, executing one Cypher query per relationship pair.

**Use case:** Graph Explorer users need to automatically discover edges between visible nodes without manually querying each relationship type.

**Code:** `provisa/api/rest/cypher_router.py`, `provisa/cypher/assembler.py`

**Tests:** `provisa-ui/e2e/cypher-impute-edges.spec.ts`, `tests/unit/test_cypher_analytics.py`

### REQ-785 · Cypher Graph Analytics {#REQ-785}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Imputed edge startNode/endNode ids must be stable integers (registered via node_ids table), not raw database primary keys, to enable canvas lookup and correct relationship rendering.

**Use case:** Graph Explorer relies on stable node ids (as registered in node_ids.id) to correlate imputed edges with visible nodes; raw PKs cause canvas lookup failures.

**Code:** `provisa/api/rest/cypher_router.py`, `provisa/cypher/assembler.py`

**Tests:** `provisa-ui/e2e/cypher-impute-edges.spec.ts`, `tests/unit/test_cypher_analytics.py`

### REQ-786 · Cypher Graph Analytics {#REQ-786}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Impute-relationships endpoint accepts visible node set with stable integer ids and resolves them to database primary keys via node_ids table lookup before executing relationship queries.

**Use case:** Graph Explorer passes only the stable integer ids (already loaded); the endpoint must map them to raw PK values for correct relationship filtering.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/cypher-impute-edges.spec.ts`, `tests/unit/test_cypher_analytics.py`

## 4. Source Connectors

### REQ-788 · File & Lake Sources {#REQ-788}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

File connector sources accept a directory glob pattern to enumerate CSV files. Discovered files are introspected to extract schema (column names and types). Multiple CSV files matching the glob are consolidated into a single logical table when registered.

**Use case:** File glob patterns enable users to query across multiple CSV files without manual discovery or registration of individual files.

**Code:** `provisa/source_adapters/file_connector.py`

**Tests:** `provisa-ui/e2e/file-connector.spec.ts`, `tests/unit/test_file_lake_sources.py`

### REQ-789 · File & Lake Sources {#REQ-789}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

CSV column headers are automatically mapped to GraphQL field names using LINQ4J convention: camelCase headers are converted to snake_case column names. This mapping is transparent at the schema definition level.

**Use case:** Automatic header normalization allows CSV files with mixed naming conventions to produce consistent, queryable schemas without manual column mapping.

**Code:** `provisa/source_adapters/file_connector.py`

**Tests:** `provisa-ui/e2e/file-connector.spec.ts`, `tests/unit/test_file_lake_sources.py`

### REQ-790 · File & Lake Sources {#REQ-790}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

File connector table enumeration (via directory glob discovery) is accessible through the Provisa UI table registration form. When a file source is selected in the form, a schema dropdown is populated with discovered schemas, and a table dropdown lists all tables available in the selected schema.

**Use case:** UI-driven table enumeration allows users to discover and register file-based tables without knowledge of directory structure or manual schema definitions.

**Code:** `provisa-ui/src/pages/tables`, `provisa/api/rest/tables_router.py`

**Tests:** `provisa-ui/e2e/file-connector.spec.ts`, `tests/unit/test_file_lake_sources.py`

### REQ-791 · File & Lake Sources {#REQ-791}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Registered file-based tables are queryable via the data GraphQL endpoint. Tables appear in the GraphQL schema with domain-prefixed field names. Queries execute read-only against the underlying CSV files with all columns visible (subject to column masking and RLS rules if defined).

**Use case:** File-based data becomes immediately consumable via standard GraphQL queries, enabling analytics and reporting without additional transformation.

**Code:** `provisa/api/data/endpoint.py`, `provisa/compiler/sql_gen.py`

**Tests:** `provisa-ui/e2e/file-connector.spec.ts`, `tests/unit/test_file_lake_sources.py`

## 10. Graph Export

### REQ-792 · Neo4j Export {#REQ-792}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

GET /data/graph-schema endpoint returns the current graph schema including all node labels (as array of {label: "Domain:Table"} objects) and relationship types (as array of {type, source, target} objects indicating source and target node labels).

**Use case:** E2E export clients can discover which node labels and relationship types exist in the current graph before querying and exporting them.

**Code:** `provisa/api/data/endpoint.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_cypher_endpoint.py`

### REQ-793 · Neo4j Export {#REQ-793}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

POST /data/cypher endpoint accepts parameterless Cypher queries and returns row data. Queries requiring parameters are rejected. Endpoint returns {rows: [...], error?: string} structure.

**Use case:** E2E export clients can execute discovery and validation queries against the graph without parameter substitution complexity.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_cypher_endpoint.py`

### REQ-794 · Neo4j Export {#REQ-794}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Query result values are introspected for node and edge structures. Nodes are identified by presence of {id, tableLabel, properties} fields. Edges are identified by {identity, startNode, endNode, type} fields. Both are recursively extracted from arbitrary result objects.

**Use case:** E2E export clients can query arbitrary graph patterns and automatically extract nodes and edges regardless of result structure nesting.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_cypher_endpoint.py`

### REQ-795 · Neo4j Export {#REQ-795}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

POST /data/neo4j-export endpoint accepts edge-only requests (empty nodes array) and exports relationships without requiring node re-export. Edges reference nodes via start/end node IDs (matching deduplication key).

**Use case:** E2E export workflows can batch export large graphs by sending nodes first, then edges in a separate request without resending node data.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_cypher_endpoint.py`

### REQ-796 · Neo4j Export {#REQ-796}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

X-Role header (e.g., "DEV") grants access to /data/cypher and /data/graph-schema endpoints. Requests without valid role are rejected.

**Use case:** Data export workflow endpoints require explicit role-based access control distinct from standard query execution permissions.

**Code:** `provisa/api/rest/cypher_router.py`, `provisa/api/data/endpoint.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_cypher_endpoint.py`

### REQ-797 · Neo4j Export {#REQ-797}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

E2E neo4j export validates exported graph integrity: node and relationship counts in target Neo4j instance match expected counts. MERGE deduplication on _provisa_id ensures node count ≤ exported count; relationship count must equal exported edge count.

**Use case:** E2E export clients can verify that export was complete and accurate by querying node and relationship counts in the target Neo4j instance.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `provisa-ui/e2e/neo4j-docker-export.spec.ts`, `tests/unit/test_neo4j_cypher_endpoint.py`

## 5. Query Languages, Compilation & Operations

### REQ-798 · Cypher Mutations {#REQ-798}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher mutations (CREATE/DELETE/UPDATE) must be transpiled through the full semantic SQL write pipeline, applying RLS injection, dialect transpilation, and all post-mutation hooks (response cache invalidation, MV stale marking, Kafka change events, Kafka sink triggers, hot-table reload).

**Use case:** Cypher mutations must enforce the same security policies, maintain consistency across multiple backends, and trigger the same change-propagation workflows as SQL and GraphQL mutations.

**Code:** `provisa/api/rest/cypher_router.py`

**Tests:** `tests/unit/test_cypher_mutations.py`, `tests/integration/test_mutations.py`, `tests/steps/steps_cypher_mutations.py`

## 6. Query Interfaces & Explorers

### REQ-799 · UI Explorer Pages {#REQ-799}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Browser-based gRPC Explorer UI page at /grpc allows interactive query execution against gRPC service methods. Proto schema displayed inline, request bodies edited as JSON, responses rendered as JSON. Method selection filters by operation type (query vs mutation). Auto-selects first query method on navigation.

**Use case:** Developers can explore and test gRPC queries without native gRPC client tools, using the same governance pipeline as the gRPC servicer.

**Code:** `provisa/api/data/endpoint_grpc_proxy.py`, `provisa-ui/src/pages/GrpcPage.tsx`

**Tests:** `provisa-ui/e2e/grpc-explorer.spec.ts`

### REQ-800 · UI Explorer Pages {#REQ-800}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Browser-based JSON:API Explorer UI page at /jsonapi allows interactive query composition with sparse fieldsets, filtering (eq, neq, gt, gte, lt, lte, like), sorting, pagination, and relationship inclusion. Table selection grouped by domain, field/include selectors with checkbox dropdowns, result summary view with resource cards and pagination controls.

**Use case:** Developers can explore and test JSON:API endpoints without manual URL construction, inspecting governance-filtered data across all domains and relationships.

**Code:** `provisa/api/jsonapi/spec.py`, `provisa-ui/src/pages/JsonApiPage.tsx`

**Tests:** `provisa-ui/e2e/jsonapi-explorer.spec.ts`

### REQ-801 · UI Explorer Pages {#REQ-801}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Browser-based OpenAPI Explorer UI page at /openapi embeds Swagger UI iframe serving dynamically generated OpenAPI 3.1 spec from /data/rest/docs. Supports auto-run navigation to pre-selected REST endpoints with method/path detection and try-it-out activation via Swagger UI DOM manipulation.

**Use case:** Developers can browse REST API endpoints with interactive try-it-out experience powered by Swagger UI, with support for navigation from other explorers (e.g., NL query results linking to generated REST endpoints).

**Code:** `provisa/api/rest/openapi_spec.py`, `provisa-ui/src/pages/OpenApiPage.tsx`

**Tests:** `provisa-ui/e2e/openapi-explorer.spec.ts`

## 4. Query Languages, Compilation & Operations

### REQ-802 · Protocol Support {#REQ-802}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Bolt protocol (Neo4j binary protocol) TCP server at port 5251 accepts Cypher queries and mutations, transpiles through Provisa query pipeline (compile, govern, route, execute). PackStream codec encodes/decodes scalars, lists, dicts, nodes, relationships, and messages. Framing layer chunks messages for TCP transport.

**Use case:** Neo4j-compatible clients (DBeaver, Cypher shells, drivers) can execute Cypher queries against Provisa schema with full governance applied, expanding query language surface beyond GraphQL/SQL/gRPC/REST.

**Code:** `provisa/bolt/server.py`, `provisa/bolt/packstream.py`, `provisa/bolt/session.py`, `provisa/bolt/messages.py`, `provisa/bolt/framing.py`, `provisa/bolt/websocket.py`

**Tests:** `tests/unit/test_bolt_packstream.py`, `tests/integration/test_bolt_server.py`, `tests/e2e/test_bolt_cypher.py`

### REQ-803 · Protocol Support {#REQ-803}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

HTTP→gRPC proxy endpoint POST /data/grpc/{TypeName} translates JSON request body into GraphQL query using the gRPC schema, compiles and executes through the full query pipeline, returns JSON rows. Supports read_mask with dot-notation for field filtering and native filter args (path_param, query_param) from request body["filter"].

**Use case:** Browser-based gRPC Explorer and programmatic gRPC clients can query via HTTP+JSON without requiring native gRPC/Protobuf tooling, while applying the same governance and backend routing as native gRPC.

**Code:** `provisa/api/data/endpoint_grpc_proxy.py`

**Tests:** `tests/unit/test_grpc_proxy_translation.py`, `tests/integration/test_grpc_proxy.py`

### REQ-804 · Protocol Support {#REQ-804}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Dynamic OpenAPI 3.1 spec generator for /data/rest endpoints. Generates per-table paths, typed query parameters (limit, offset, fields, filter, orderBy), response schemas with attributes and relationships. Supports domain filtering, describes limit/offset pagination and sparse fieldsets.

**Use case:** Swagger UI and REST API consumers can discover and understand auto-generated REST endpoints with parameter types, filter operators, pagination, and output schemas derived from the GraphQL schema.

**Code:** `provisa/api/rest/openapi_spec.py`

**Tests:** `tests/unit/test_openapi_spec.py`, `tests/integration/test_openapi_spec.py`

## 6. Query Interfaces & Explorers

### REQ-805 · UI Explorer Pages {#REQ-805}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Dynamic JSON:API OpenAPI 3.1 spec generator accessible at /data/jsonapi/openapi.json. Generates per-table paths with sparse fieldsets (fields[table]), filtering (filter[col], filter[col][op] for all filter operators), sorting (sort ± prefix), pagination (page[size], page[number]), and include (relationship names). Returns application/vnd.api+json in spec.

**Use case:** Tools and developers can discover JSON:API endpoints and their constraints (filter operators, field lists, relationship names) without manual inspection.

**Code:** `provisa/api/jsonapi/spec.py`

**Tests:** `tests/unit/test_jsonapi_spec.py`, `tests/integration/test_jsonapi_spec.py`

## 3. Data Storage & Indexing

### REQ-806 · Bolt Protocol {#REQ-806}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Bolt relationships receive durable integer IDs via a `rel_ids` Postgres table (BIGSERIAL id, composite_id "Type:startPk-endPk" UNIQUE, rel_type, properties JSONB), mirroring the `node_ids` design. The `register_rel_ids` function upserts edges and replaces composite identities with integer IDs.

**Use case:** Bolt clients require stable relationship identity across browser sessions and multiple query paths (Bolt native, REST cypher, REST impute); numeric IDs enable efficient indexing and relationship traversal.

**Code:** `provisa/core/schema.sql`, `provisa/bolt/session.py`, `provisa/bolt/packstream.py`

**Tests:** `tests/unit/test_bolt_rel_ids.py`, `tests/integration/test_bolt_rel_ids.py`

## 1. Access Governance & Security

### REQ-807 · Protocol Support {#REQ-807}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Bolt auth model establishes principal = user, database = role. A user holds a set of roles; SHOW DATABASES lists one database per (view × role) pair. Database names follow the pattern `provisa_<role>` (business domains only) and `provisa_ops_<role>` (includes system/meta/ops domains). The Bolt Browser's database dropdown / `:use` command selects the active role and view. Home database defaults to the user's first role.

**Use case:** Bolt Browser users can switch contexts by role and view (ops vs business) without reauthenticating. The database abstraction maps cleanly to role-based domain visibility.

**Code:** `provisa/bolt/session.py`, `provisa/bolt/packstream.py`

**Tests:** `tests/unit/test_bolt_show_databases.py`

### REQ-808 · Query Governance {#REQ-808}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The Bolt endpoint must never subvert the connecting user's domain rights. The active role's `domain_access` is the hard ceiling for every query, catalog call (db.labels), relationship counts, and relationship imputation. The ops/business view only narrows within that ceiling. Unknown or unauthorized `db` values return DatabaseNotFound error (no silent fallback). System, meta, and ops domains are identified by domain_id in {"", "meta", "ops"}.

**Use case:** Bolt's flexible query language (cypher) could bypass view/role filtering if not enforced at the session/catalog level. This requirement ensures domain visibility is the immovable constraint, not an advisory filter.

**Code:** `provisa/bolt/session.py`, `provisa/cypher/assembler.py`, `provisa/api/rest/cypher_router.py`

**Tests:** `tests/unit/test_bolt_domain_ceiling.py`, `tests/integration/test_bolt_domain_ceiling.py`

## 5. Query Languages, Compilation & Operations

### REQ-809 · SQLGlot Transpilation {#REQ-809}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Correlated scalar subqueries in PG-style canonical SQL are lifted into CTEs before the query is forwarded to Trino. rewrite_correlated_subqueries_for_trino in provisa/transpiler/transpile.py rewrites any correlated scalar subquery into a set-based CTE join so the federation engine receives a form it can execute across sources.

**Use case:** Correlated-subquery-to-CTE rewriting keeps cross-source SQL correct on Trino, which does not execute arbitrary correlated scalar subqueries the way a single RDBMS would.

**Code:** `provisa/transpiler/transpile.py`

**Tests:** `tests/unit/test_transpile.py`

## 6. Execution, Routing, Caching & Performance

### REQ-810 · Materialized Views {#REQ-810}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Materialized-view transparent rewriting supports partial join-pattern matches. rewrite_if_mv_match in provisa/mv/rewriter.py matches a query's joins against an MV per-join; when the MV covers only a subset of the query's joins, _partial_rewrite_to_mv removes the covered joins and preserves the remaining joins so the rewritten query reads the MV for the matched portion and joins the rest live.

**Use case:** Partial MV matching lets a materialized view accelerate queries that join additional tables beyond the MV's definition, instead of only queries whose join pattern matches the MV exactly.

**Code:** `provisa/mv/rewriter.py`

**Tests:** `tests/integration/test_execution_routing.py`, `tests/e2e/test_mv_optimization.py`, `tests/unit/test_mv_partial_rewrite.py`

### REQ-811 · Federation Performance {#REQ-811}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The `# @provisa key=value` GraphQL comment hint vocabulary includes a `route=federated|direct` directive parsed by extract_graphql_hints in provisa/compiler/hints.py. `route=federated` forces the query through the federation engine; `route=direct` forces single-source direct execution. This is distinct from the `/*+ BROADCAST(...) */` SQL block-hint family ([REQ-279](#REQ-279)) and source-level federation_hints ([REQ-281](#REQ-281)).

**Use case:** A per-query route hint lets query authors override automatic routing for a specific query without changing source configuration.

**Code:** `provisa/compiler/hints.py`

**Tests:** `tests/unit/test_routing.py`

## 8. Client Access & Protocols

### REQ-812 · API & Integration {#REQ-812}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The `X-Provisa-Sink` request header on a subscription request redirects the subscription's change-event output to a Kafka sink instead of the SSE response stream. subscription_sse.py parses the header and launches a Kafka sink consumer for the subscription, returning 202 Accepted rather than an open SSE stream.

**Use case:** An X-Provisa-Sink header lets a client wire a live subscription directly into a Kafka topic for downstream stream processing without holding an SSE connection.

**Code:** `provisa/api/data/subscription_sse.py`

**Tests:** `tests/unit/test_subscription_sink_header.py`, `tests/integration/test_kafka_sink.py`

## 9. Live Data & Events

### REQ-813 · Subscriptions {#REQ-813}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The `live` block (LiveDeliveryConfig on Table) is a unified change-feed / CDC abstraction. The binary `delivery: poll|cdc` field is replaced by a multi-strategy model: `strategy: poll|native|debezium|kafka`. Per-table params: poll → watermark_column/poll_interval; native → none; kafka → LiveKafkaParams (topic/format/key_column/field_mapping). Reconciled with [REQ-824](#REQ-824): Debezium/Kafka delta-transport (bootstrap_servers/topic_prefix/schema_registry_url/consumer_group_id) is inherited from Source.cdc, entered once per source — NOT repeated in per-table debezium params. query_id and outputs are optional so `live` covers both raw table change-feeds and live persisted-query output fan-out. Debezium topic names are derived from the source transport: {topic_prefix}.{schema}.{table}.

**Use case:** A single `live` config abstraction unifies all capture mechanisms (polling, PostgreSQL LISTEN/NOTIFY, MongoDB change streams, Debezium, arbitrary Kafka deltas), eliminating the misleading binary of "poll vs CDC" where poll IS a change-feed mechanism. Strategy-specific params prevent schema pollution. Optional query_id and outputs enable both low-level table subscriptions and high-level live persisted-query delivery.

**Code:** `provisa/core/models.py`

**Tests:** `tests/unit/test_live_delivery_config.py`

### REQ-814 · Subscriptions {#REQ-814}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Provider selection (_resolve_provider_type in provisa/api/data/subscribe.py + get_provider in provisa/subscriptions/registry.py) dispatches on `live.strategy`, NOT on source_type. Validation capability-gates strategy by source_type: warehouses → poll only; MongoDB → native/poll; PostgreSQL → native/debezium/poll/kafka; non-PG RDBMS (MySQL, MariaDB, SQL Server, Oracle) → debezium/poll/kafka (debezium/kafka require source-level cdc transport per [REQ-824](#REQ-824)); Kafka topics → kafka only. No source is registered with a synthetic "debezium" type; strategy=debezium routes to the Debezium provider.

**Use case:** Dispatching on strategy instead of source_type correctly models that strategy is a choice of delta-capture mechanism (independent of where the data is queried from), not a data source type. Capability-gating per source_type prevents misconfiguration (e.g., native change streams on Snowflake).

**Code:** `provisa/api/data/subscribe.py`, `provisa/subscriptions/registry.py`, `provisa/core/config_loader.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_source_cdc_config.py`, `tests/unit/test_live_delivery_config.py`, `tests/steps/steps_subscriptions.py`

## 8. Deployment & Infrastructure

### REQ-815 · Docker Compose Organization {#REQ-815}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Telemetry sink services (otel-collector, prometheus, tempo, grafana, otlp2parquet) are organized into an independently start/stoppable compose project (provisa-observability) separate from the core runtime stack. The app degrades gracefully (fire-and-forget) when no OTLP collector is listening. MinIO base definitions move to docker-compose.core.yml; Trino and trino-worker remain core with observability-specific env/exporter overlays applied only when telemetry is enabled.

**Use case:** Telemetry infrastructure is optional overhead for test and CI runs. Isolating telemetry sinks reduces startup time and resource consumption while ensuring app correctness does not depend on collector availability.

**Code:** `docker-compose.core.yml`, `docker-compose.observability.yml`, `provisa/telemetry/exporter.py`

**Tests:** `tests/unit/test_infra_requirements.py`

### REQ-816 · Docker Compose Organization {#REQ-816}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Demo/sample upstream sources (graphql-demo, petstore-mock) are organized into an independently start/stoppable compose project separate from the core runtime, categorized alongside other test-only services (neo4j, elasticsearch, kafka, schema-registry, fuseki, mongodb). Integration tests requiring these fixtures bring them up via harness markers rather than assuming they are already running in the dev stack.

**Use case:** Demo sources serve as test fixtures for federation and remote-source integration tests. Organizing them with other test services ensures test harness controls their lifecycle and reduces assumptions about the dev environment state.

**Code:** `docker-compose.demo.yml`, `docker-compose.test.yml`, `tests/conftest.py`

**Tests:** `tests/integration/test_openapi_pet_by_status.py`, `tests/integration/test_graphql_remote_integration.py`

### REQ-817 · Trino Memory Management {#REQ-817}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Trino memory overflow handling uses Fault-Tolerant Execution (FTE) with task.retry-policy=TASK and task.low-memory-killer.policy=total-reservation-on-blocked-nodes. Docker deployments use a shared named-volume filesystem exchange manager (trino/etc/exchange-manager.properties, /data/trino/exchange). Helm deployments require an S3/MinIO-backed exchange manager with a MinIO bucket-init Job or external S3 endpoint+credentials, enforced by a fail-loud guard.

**Use case:** FTE replaces legacy spill-to-disk (deprecated in Trino 481) and mitigates noisy-neighbor OOM in multi-tenant deployments by isolating task memory failures and enabling graceful task retry.

**Code:** `docker-compose.core.yml`, `trino/etc/config.properties`, `trino/etc/worker/config.properties`, `trino/etc/exchange-manager.properties`, `helm/provisa/templates/trino-exchange-secret.yaml`, `helm/provisa/templates/trino-exchange-bucket-job.yaml`

**Tests:** `tests/integration/test_trino_fte_exchange.py`

## 5. Query Languages, Compilation & Operations

### REQ-818 · Cypher Mutations {#REQ-818}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Cypher supports WRITES via the /data/cypher endpoint. CREATE, DELETE, and SET statements execute as direct table writes through the same write pipeline as GraphQL and SQL mutations, with RLS injection, dialect transpilation, and post-mutation hooks. Non-direct write patterns (MERGE, DETACH, REMOVE) are unsupported and rejected. All writes are governed by table/view write rights like any other write. This supersedes [REQ-346](#REQ-346) (read-only constraint), which is now inaccurate.

**Use case:** Write support enables graph users to mutate data via Cypher while maintaining governance consistency with SQL and GraphQL mutations.

**Code:** `provisa/api/rest/cypher_router.py`, `provisa/cypher/write_translator.py`, `provisa/cypher/parser.py`

**Tests:** `tests/integration/test_cypher_endpoint.py`, `tests/unit/test_cypher_parser.py`, `tests/unit/test_cypher_translator.py`

## 6. Real-Time & Live Data

### REQ-819 · Live Delivery Configuration {#REQ-819}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Per-table live delivery configuration is now configurable via the admin GraphQL API and admin UI (TablesPage). Configuration is persisted on registered_tables.live (JSONB column). Configuration includes: query_id (identifies the live query), watermark_column (tracks delivery progress), poll_interval (frequency in seconds), delivery mode ("poll" or "cdc"), and outputs array (SSE or Kafka sinks). Previously configuration was YAML-only.

**Use case:** UI-based configuration makes live delivery settings accessible to non-YAML-fluent operators and enables runtime changes without server restart.

**Code:** `provisa/api/admin/types.py`, `provisa/api/admin/schema.py`, `provisa/core/repositories/table.py`, `provisa/core/schema.sql`, `provisa-ui/src/pages/TablesPage.tsx`

**Tests:** `tests/unit/test_live_delivery_config.py`

### REQ-820 · Live Query Routing {#REQ-820}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Live poll execution routes through Trino (federated query), not the PostgreSQL connection pool. Any federated SQL source with a watermark column is pollable. The engine's pg_pool is used only for watermark bookkeeping (live_query_state table). This enables polling over remote sources (not just PostgreSQL) via Trino's federation layer.

**Use case:** Routing through Trino enables live polling of any federated SQL source (e.g., BigQuery, Snowflake, S3 via Iceberg) without embedding source-specific clients into the live engine.

**Code:** `provisa/live/engine.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_live_engine.py`

### REQ-821 · Live Delivery Mechanisms {#REQ-821}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

PostgreSQL data is never polled for live delivery. PostgreSQL live delivery uses delivery=cdc (LISTEN/NOTIFY with database triggers). Poll delivery (delivery=poll) is reserved for non-push federated sources reached through Trino. This prevents duplicate load on PostgreSQL replication and uses push mechanisms where available.

**Use case:** PostgreSQL's native LISTEN/NOTIFY is more efficient than polling. Restricting poll to federated sources avoids duplicate query load on the primary database.

**Code:** `provisa/live/engine.py`, `provisa/core/config_loader.py`

**Tests:** `tests/unit/test_live_delivery_config.py`

### REQ-822 · Live Delivery Mechanisms {#REQ-822}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

delivery=cdc is restricted to source types with a real push provider in the subscription registry: postgresql (LISTEN/NOTIFY triggers), debezium and kafka (Kafka consumers), mongodb (change streams). All other sources default to delivery=poll. CDC connection config is inherited from the source definition, never per-table. This ensures CDC is only used when a push mechanism is available.

**Use case:** Restricting CDC to sources with native push mechanisms prevents attempts to use CDC on sources that don't support it. Connection config inheritance avoids credential duplication per table.

**Code:** `provisa/core/config_loader.py`, `provisa/subscriptions/registry.py`

**Tests:** `tests/unit/test_live_delivery_config.py`

### REQ-823 · Live Delivery Configuration {#REQ-823}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The LiveEngine reconciles poll jobs from the database at startup and after every admin mutation via _rebuild_schemas(). This ensures that admin-set live config takes effect without server restart. Configuration changes trigger immediate engine reconciliation.

**Use case:** Startup reconciliation catches live jobs even if the engine crashes mid-execution. Admin mutations trigger immediate reconciliation so operators see changes reflected in real-time without restart overhead.

**Code:** `provisa/live/engine.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_live_engine.py`, `tests/unit/test_live_delivery_config.py`

### REQ-824 · Live Delivery Mechanisms {#REQ-824}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Source-level CDC transport configuration. Debezium/Kafka delta-transport (bootstrap_servers, topic_prefix, schema_registry_url, consumer_group_id) is entered ONCE per source via a `cdc` block on the Source model (persisted to sources.cdc JSONB), never repeated per-table. Per-table live config only selects delivery=cdc; the runtime inherits transport from the source. A source-level cdc block is only valid on CDC-capable RDBMS sources (postgresql uses native LISTEN/NOTIFY and needs no transport; mysql/mariadb/sqlserver/oracle require it to enable delivery=cdc via Debezium). Provider dispatch routes non-PG RDBMS sources with a cdc block to the Debezium provider. Polling remains per-table (watermark_column, poll_interval).

**Use case:** Eliminates repeated data entry of Debezium/Kafka connection details across every table of a source; transport is a per-source connector property, not a per-table one.

**Code:** `provisa/core/models.py`, `provisa/core/config_loader.py`, `provisa/core/repositories/source.py`, `provisa/core/schema.sql`, `provisa/api/app.py`, `provisa/api/data/subscribe.py`, `provisa/api/admin/types.py`, `provisa/api/admin/schema.py`, `provisa-ui/src/liveCapability.ts`, `provisa-ui/src/pages/SourcesPage.tsx`, `provisa-ui/src/types/admin.ts`, `provisa-ui/src/hooks/admin.graphql`

**Tests:** `tests/unit/test_source_cdc_config.py`, `provisa-ui/src/__tests__/liveCapability.test.ts`

## 6. Execution, Routing, Caching & Performance

### REQ-825 · Execution & Routing {#REQ-825}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The Provisa FederationEngine is a pluggable, abstract execution substrate behind a single contract. It is one of four primitives composed as a data-flow pipeline: DATASOURCES → FEDERATION ENGINE → SEMANTIC LAYER → EXECUTION PIPELINE. DATASOURCES are external systems with a capability profile. The FEDERATION ENGINE (this requirement) federates those datasources into one raw unified queryable surface; the method per source is the federation strategy ([REQ-826](#REQ-826)). The SEMANTIC LAYER is the governed virtual model projected over that surface — registered tables/columns, relationships, domains, RLS, masking — engine-independent; it is where governance is DEFINED and the thing queries target. The EXECUTION PIPELINE turns a query into results as a sequence of total IR→IR passes (nanopass discipline), bookended by transport: (1) TRANSPORT ingress — receive the query over the client protocol (pgwire, Bolt, Arrow Flight, GraphQL, SQL); (2) TRANSPILE TO IR — frontend parse of the source language into a dialect-neutral logical IR (plural frontends converge here); (3) TRANSPILE TO GOVERNED IR — apply the semantic layer: inject RLS predicates, masking expressions, and relationship joins (engine-independent; the IP pass); (4) PLAN — the one IMPURE stage: not a pure transpile but a planner that (a) chooses the ROUTE (direct native-driver execution for a single reachable source vs handing the query to the federation engine), (b) resolves RESIDENCY PREREQUISITES — for each source whose federation strategy is MATERIALIZED and stale, emit a side-effecting prep step that loads/refreshes it into the DB cache before execute ([REQ-826](#REQ-826)), and (c) CODEGENs the engine-dialect SQL via SQLGlot for the chosen route. The output is not a string but an ordered plan: a prep phase (0..n cache materializations, may be empty) feeding a single terminal step (DIRECT driver or federation-engine execute); this is where today's router (DIRECT/TRINO/API) and the cache tiers (hot/warm/MV) converge. (5) EXECUTE — the federation engine runs the terminal step and does its own opaque physical planning (NOT Provisa's plan); (6) TRANSPORT egress — stream results back over the same client protocol. Stages 1–3 are pure and engine-independent; stage 4 reads mutable freshness state and emits effects, and stages 4–5 vary per engine — so governance-parity conformance ([REQ-827](#REQ-827)) must PIN residency (pre-warm the prep phase) and then test only the codegen + execute. No governance or compilation logic is duplicated per engine — governance is defined in the semantic layer and ENFORCED at stage 3; the engine only executes. There is no "federation vs warehouse" dichotomy: EVERY engine federates; only the METHOD of federation changes with capability (see [REQ-826](#REQ-826)). An engine advertises a capability profile with two axes: (a) REACH — for each datasource, which federation strategies it can support; and (b) SCALE — single-node/memory-bound vs distributed. The profile RANKS transports per source (native > ADBC > ODBC), not a has_connectors boolean. Concrete engines: DuckDB is a full federation engine whose reach nearly matches Trino's — native ATTACH for postgres/mysql/sqlite; ODBC (community nanodbc odbc_attach, or the DuckDB-team odbc/odbc_scanner functions) for any ODBC source; ADBC (community adbc_scanner, Arrow-native, preferred over ODBC); NoSQL via community extensions (mongo ATTACH TYPE MONGO, cassandra scan/query/ATTACH, elasticsearch_query); and zero-copy file scans. Its ONLY differentiator vs Trino is SCALE (single-node, memory-bound, no distributed shuffle), making it ideal on a developer desktop with no Trino dependency. Trino has broad reach and distributed scale. Snowflake/Databricks/Dremio are reused as engines to leverage a customer's existing infrastructure. A pure RDBMS (e.g. SQL Server) has near-zero reach and federates only by the materialized strategy. The engine is selected at deployment via top-level config (engine: duckdb | trino | snowflake | databricks | dremio | ...), not per-source. All query dispatch that today reads the hardcoded Trino singleton (provisa/api/app.py, provisa/api/data/endpoint.py, provisa/executor/trino.py) must route through the FederationEngine contract. The IR->physical mapping is intentionally thin — the divergence is confined to dialect transpilation and the per-source federation strategy ([REQ-826](#REQ-826)). DONE (2026-07): the PLAN stage OUTPUT is modelled — provisa/federation/plan.py::build_execution_plan composes the pure part of stage 4 into an ordered Plan: a prep phase (0..n PrepStep residency loads/refreshes, one per source whose federate() strategy is MATERIALIZED and stale) feeding a single terminal Route (DIRECT for a single VIRTUAL source, ENGINE otherwise). This is the four-primitive flow's planner surface built on the connector/federate contract ([REQ-840](#REQ-840)/826). REMAINING (kept in-progress): the other pipeline stages as engine-agnostic passes (transport ingress/egress, transpile to IR, transpile to governed IR, codegen, execute) and routing the hardcoded-Trino dispatch through the FederationEngine contract.

**Use case:** Decouples Provisa's governed semantic layer from any single execution vendor. The same semantic layer, execution pipeline, governance, IR, and multi-protocol query surface run on an embedded desktop engine (DuckDB, no Trino), the Trino federation engine, or a customer's existing warehouse (Snowflake/Databricks/Dremio) — making the engine a swappable deployment choice and concentrating Provisa's IP in the semantic layer, pipeline, governance, and federation-strategy logic rather than the engine. Intended model: develop on DuckDB (laptop, zero infra, full governance and full source reach), then scale out to Trino/Snowflake/Databricks in production, unchanged.

**Code:** `provisa/federation/plan.py`, `provisa/api/app.py`, `provisa/api/data/endpoint.py`, `provisa/executor/trino.py`, `provisa/transpiler/router.py`, `provisa/core/config_loader.py`

**Tests:** `tests/unit/test_federation_plan.py`

### REQ-826 · Caching {#REQ-826}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Federation strategy and freshness management — the datasources → federation engine arrow of [REQ-825](#REQ-825). The binding between a datasource table and the engine is a FEDERATION STRATEGY: the method by which that source joins the engine's unified surface. federate(datasource, table) resolves each table to one of three strategies and returns which was chosen, because freshness differs per strategy: (1) VIRTUAL — the engine reaches the source live (a connector, or DuckDB ATTACH to postgres/mysql/sqlite, or a Trino connector); no copy, always fresh, cache_ttl irrelevant. (2) SCAN — the source is a file/object the engine reads in place, exposed as a CREATE VIEW with no data moved; freshness follows the underlying file, cache_ttl irrelevant. For DuckDB this spans read_parquet/read_csv/read_json, httpfs over S3/HTTP/GCS/Azure, Iceberg/Delta via extension, ATTACH of external DuckDB/SQLite, and other scanner extensions; the engine declares what it can scan. (3) MATERIALIZED — no virtual or scan representation (live APIs, NoSQL, or an RDBMS deliberately cached for latency); data is INSERTed/COPYed in. MATERIALIZED is the ONLY strategy where cache_ttl means a reload interval and the load-scheduling/invalidation logic runs; VIRTUAL and SCAN are effectively free. The federation strategy is STATEFUL — it owns residency, reload state, and freshness; the execution pipeline (transpile → transport) is stateless per query and must not carry this state. The chosen strategy depends on BOTH datasource capability and engine capability, so the same source may federate by different strategies on different engines. This unifies today's three separate mechanisms — OpenAPI PG response cache (provisa/openapi/pg_cache.py), NoSQL mapping catalogs, and warm-tables/MV materialization — into one federate() operation. NoSQL scanners (mongo, cassandra, elasticsearch) infer schema on read; because governance in the semantic layer binds to columns, federate() MUST pin the schema on the CREATE VIEW from the semantic layer's mapping ([REQ-251](#REQ-251)), never trust re-inference. The router must not attempt a live/VIRTUAL route when federate() reports only a MATERIALIZED strategy for that source; it uses the strategy federate() returns. Only the TENANT MATERIALIZATION STORE (durable — API/GraphQL/gRPC results, warm tables, MVs; [REQ-830](#REQ-830)) is a true MATERIALIZED strategy: the data becomes a real relation the engine reads, so it PARTICIPATES in federation (joinable/scannable like any source). The TENANT HOT CACHE is NOT a federation strategy and is orthogonal to virtual|scan|materialized — it is a codegen-time VALUES-CTE injection inlined into the SQL at the PLAN stage, never an engine relation and never federated; tier exclusivity ([REQ-241](#REQ-241)) chooses hot over materialization store. DONE (2026-07): federate(source, engine) is implemented (provisa/federation/strategy.py), resolving each source to VIRTUAL | SCAN | MATERIALIZED and returning which was chosen. The choice depends on BOTH source capability and engine capability: an ATTACH connector on a live DB → VIRTUAL, an ATTACH connector on a file/object type (csv/parquet/iceberg/...) → SCAN, a LAND connector (warehouse-native) → MATERIALIZED, and an API/NoSQL source with no connector → MATERIALIZED (loaded into the store). The same source therefore federates differently per engine (csv SCANs on DuckDB but is UnreachableSource on a Trino build without a csv connector). prefer_materialized forces MATERIALIZED for a live source deliberately cached for latency. requires_residency(strategy) is the [REQ-825](#REQ-825) stage-4b signal (only MATERIALIZED needs a prep step). REMAINING (kept in-progress): the STATEFUL side — wiring residency / reload-scheduling / invalidation and unifying the three existing mechanisms (openapi pg_cache, NoSQL mapping catalogs, warm-tables/MV) behind this one federate() operation, and having the router consume the returned strategy — lands with the materialization store ([REQ-844](#REQ-844)-848/830) and the PLAN stage.

**Use case:** Concentrates the real complexity of federation — strategy selection, residency, reload scheduling, and invalidation — in one stateful federate() operation, so any datasource (RDBMS, files, NoSQL, APIs) joins the engine's surface by the best available strategy with predictable freshness, independent of which engine is deployed.

**Code:** `provisa/federation/strategy.py`, `provisa/federation/engine.py`, `provisa/executor/trino.py`, `provisa/transpiler/router.py`, `provisa/openapi/pg_cache.py`, `provisa/mv/registry.py`, `provisa/cache/warm_tables.py`, `provisa/cache/hot_tables.py`, `provisa/core/config_loader.py`

**Tests:** `tests/unit/test_federation_strategy.py`

### REQ-827 · Execution & Routing {#REQ-827}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Governance-parity conformance across federation engines. Any engine admitted to the FederationEngine contract ([REQ-825](#REQ-825)) MUST produce byte-identical governed results — RLS filtering, column masking, relationship-join enforcement, and row/aggregate output — as the reference engine for the same query and same identity. Because governance is defined in the semantic layer and enforced when the query compiles to IR, only the transpile → transport stages differ per engine; SQLGlot transpiles syntax, but semantic edges still differ per engine (NULL ordering, type coercion, decimal/numeric precision, regex and collation semantics, timezone and timestamp handling, string comparison case sensitivity), and these must not change which rows are visible, which cells are masked, or which joins are enforced. An engine is certified only by passing a shared governance-parity conformance suite run against a fixed golden dataset and identity matrix; results are diffed against the reference engine and any divergence fails certification. This is a hard gate: an uncertified engine MUST NOT be selectable in production config ([REQ-825](#REQ-825)). The suite is authoritative for DuckDB, Trino, Snowflake, Databricks, Dremio, and any future engine. DONE (2026-07): the comparator + certification GATE are implemented — provisa/federation/conformance.py:: compare_governed_results diffs a candidate engine's governed rows against the reference as an order-INDEPENDENT multiset (NULL ordering is an allowed semantic edge) but visibility/masking-EXACT: a row visible in exactly one side (an RLS leak or over-filter) or a differently-masked cell is a Divergence, and any divergence fails certification. ConformanceRegistry tracks certified engines (the reference is always certified) and require_certified raises UncertifiedEngineError — the hard gate that an uncertified engine must not be selectable. REMAINING (kept in-progress): the conformance SUITE itself — the fixed golden dataset + identity matrix and the runner that executes each query on every candidate engine and feeds the diff — which needs the live multi-engine execution paths.

**Code:** `provisa/federation/conformance.py`, `provisa/security/`, `provisa/compiler/`, `provisa/executor/trino.py`, `provisa/transpiler/router.py`

**Tests:** `tests/unit/test_conformance.py`, `tests/integration/test_governance_integration.py`, `tests/integration/test_governance_parity_e2e.py`

## 11. Platform, Infrastructure & Delivery

### REQ-828 · Deployment & Infrastructure {#REQ-828}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Pluggable admin/metadata store, decoupled from Postgres, selected via a SQLAlchemy-compliant URI. The control-plane store (sources, domains, registered tables, columns, naming rules, governance config) plus the OpenAPI/GraphQL cache substrate must run on an embedded engine (DuckDB or SQLite) with zero external infra on a developer desktop, and on Postgres in production — same schema, same behavior. This is required for the desktop deployment model in [REQ-825](#REQ-825) (PG today requires Docker, which breaks the zero-infra laptop story). Three couplings must be removed: (1) DRIVER — repositories take asyncpg.Connection directly (provisa/core/repositories/source.py) and the pool factory is asyncpg-only (provisa/core/db.py); these move behind a store abstraction driven by a SQLAlchemy URI. (2) DIALECT — schema.sql uses Postgres-specific DDL (SERIAL, JSONB); metadata DDL must be dialect-neutral. (3) META-GOVERNANCE RLS — the hard gate: _init_meta_rls (provisa/api/app.py, [REQ-041](#REQ-041)/[REQ-402](#REQ-402)) enforces control-plane governance via Postgres row-level security, which DuckDB/SQLite lack. Meta-RLS enforcement MUST move from the database into the app layer so control-plane governance is store-independent (the same parity principle as [REQ-827](#REQ-827), applied to the admin DB). Until meta-RLS is app-enforced, embedded stores cannot be certified for production. On desktop the OpenAPI/GraphQL cache lands in the embedded store rather than a Docker Postgres; where Postgres exists (prod) the existing pg_cache path is unchanged and DuckDB may ATTACH it ([REQ-826](#REQ-826)). This store is the TENANT CONTROL PLANE REGISTRY ([REQ-830](#REQ-830)) — per-tenant, distinct from the PLATFORM CONTROL PLANE REGISTRY (tenant directory, federation-engine registry, platform governance). The meta-RLS move generalizes to a single app-layer tenant-isolation boundary enforced across all tenant-scoped stateful components, since swappable media (Redis, DuckDB, S3) have no native RLS.

**Use case:** Makes the full Provisa deployment — control plane and cache, not just the query engine — infra-free on a developer laptop, completing the develop-on-DuckDB / scale-out-in-prod model in [REQ-825](#REQ-825) without requiring Docker or an external Postgres.

**Code:** `provisa/core/meta_rls.py`, `provisa/core/db.py`, `provisa/core/database.py`, `provisa/core/duckdb_async.py`, `provisa/core/repositories/source.py`, `provisa/core/schema.sql`, `provisa/core/schema_org.py`, `provisa/api/app.py`, `provisa/openapi/pg_cache.py`

**Tests:** `tests/unit/test_admin_store.py`

### REQ-829 · Deployment & Infrastructure {#REQ-829}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Pluggable Redis medium with a lightweight, code-identical embedded realization for desktop. Sibling to [REQ-828](#REQ-828): where [REQ-828](#REQ-828) makes the SQL-store medium of the TENANT CONTROL PLANE REGISTRY pluggable (Postgres -> DuckDB/SQLite), this makes the Redis medium pluggable (Redis -> embedded fakeredis), so both halves of the [REQ-830](#REQ-830) desktop collapse (stateful components 1-4 into one process, zero infra) are covered. Redis is [REQ-830](#REQ-830)'s named production medium for the TENANT HOT CACHE (component #4, hot tables [REQ-230](#REQ-230), provisa/cache/hot_tables.py) and additionally backs the fast result/APQ caches (provisa/cache/store.py, provisa/apq/cache.py) and the Redis-runtime governance state that is orthogonal to the five data-flow components — the sliding-window rate limiter and concurrency gauge (provisa/api/rate_limit.py) and the table-based cache-invalidation index (provisa/cache/store.py). When no REDIS_URL is configured, every one of these MUST transparently use an embedded fakeredis (FakeAsyncRedis) client that is API-identical to redis.asyncio.Redis. All four current connection factories (rate_limit.py:120, cache/store.py:138, apq/cache.py:97, cache/hot_tables.py:174) route through a single make_redis(url, decode_responses) factory that returns redis.asyncio.from_url when a URL is set and FakeAsyncRedis otherwise, sharing one process-wide fakeredis.FakeServer so the four clients (which differ in decode_responses) see the same in-memory store. Zero changes to existing call sites. TENANT ISOLATION for the Redis medium is enforced in the app layer via key namespacing (existing tenant_id key prefixes: provisa:cache:<tenant_id>:..., provisa:apq:<tenant_id>:...), NOT any store-native RLS — the same app-layer isolation boundary [REQ-828](#REQ-828)/[REQ-830](#REQ-830) mandate for all swappable media (Redis, DuckDB, S3); the single shared FakeServer is acceptable because desktop is single-tenant. The stub is not a KVS: it MUST support the full command surface actually invoked — get/set/delete, expire/TTL, sorted sets (zadd/zcard/zremrangebyscore/zrange for the sliding-window rate limiter), sets (sadd/smembers/scan_iter for the invalidation index), incr/decr (concurrency gauge), and pipelines. An aiocache/cashews-style KVS abstraction is explicitly rejected because it cannot express the sorted-set and set operations without rewriting rate limiting and invalidation, and the pure get/set caches it could serve already degrade to no-ops without Redis, so it buys almost nothing. Rationale — desktop already runs without Redis today, but silently: build_rate_limiter(None) returns NoopRateLimiter (rate_limit.py:118) and NoopCacheStore no-ops invalidate_by_table/invalidate_by_pattern (store.py:93-96), so rate limiting and cache invalidation are simply disabled and those code paths are never exercised on the laptop. Yet both surfaces are live in production: rate limiting runs on every HTTP request (rate_limit_middleware.py:28) plus Arrow Flight, subscriptions, and NL endpoints (flight/server.py:248, data/subscribe.py:404, nl_router.py:58), and table invalidation fires after every write (cypher_router.py:560, data/endpoint.py:3307, admin/schema.py:2458). The purpose is therefore NOT to add missing Redis features but to make desktop exercise the SAME hot-cache, result-cache, and rate-limit code paths as production instead of silently falling back to no-ops — so that behavior is testable on a laptop with zero infra. Production with a real REDIS_URL is unchanged; in production the same medium scales out to Redis ([REQ-830](#REQ-830)). Completes the infra-free desktop model ([REQ-825](#REQ-825)/828/830) by removing Redis as the last mandatory Docker dependency alongside the SQL store.

**Use case:** Lets a developer run the full Provisa backend on a laptop with no Redis server and no Docker, collapsing the Redis medium of the TENANT HOT CACHE and Redis-runtime state into an embedded fakeredis while exercising the identical hot-cache, result-cache, APQ, rate-limit, and invalidation code paths that production runs against a real Redis — rather than the current silent no-op fallback — so behavior is the same, and testable, in both environments, with tenant isolation enforced in the app layer.

**Code:** `provisa/core/redis_factory.py`, `provisa/api/rate_limit.py`, `provisa/cache/store.py`, `provisa/apq/cache.py`, `provisa/cache/hot_tables.py`

**Tests:** `tests/unit/test_redis_factory.py`

### REQ-830 · Deployment & Infrastructure {#REQ-830}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Stateful-component topology. The four data-flow primitives (datasources -> federation engine -> semantic layer -> execution pipeline; [REQ-825](#REQ-825)) describe a STATELESS per-query flow. Orthogonal to them, a deployment holds FIVE PLUGGABLE STATEFUL COMPONENTS, each independently swappable and — except the platform registry — TENANT-ISOLATED: (1) PLATFORM CONTROL PLANE REGISTRY — platform-scoped, shared: tenant directory, federation-engine registry, platform-level governance ([REQ-041](#REQ-041)/[REQ-402](#REQ-402)), and per-tenant routing (which registry/caches/engine each tenant uses). (2) TENANT CONTROL PLANE REGISTRY — per-tenant: persists that tenant's datasource definitions and semantic layer (registered tables/columns, relationships, domains, RLS, masking, naming); the store [REQ-828](#REQ-828) makes pluggable. (3) TENANT MATERIALIZATION STORE — per-tenant durable store realizing the MATERIALIZED federation strategy ([REQ-826](#REQ-826)): API/GraphQL/gRPC result caches, warm tables, MVs; disk/object-backed. It is a real relation the federation engine reads, so materialized/MV data PARTICIPATES in federation (joinable/scannable like any source). (4) TENANT HOT CACHE — per-tenant fast tier realizing hot tables ([REQ-230](#REQ-230)): small, low-latency, ephemeral. STRICTLY a CTE-injection mechanism — its data is inlined as a VALUES CTE into the generated SQL at codegen and NEVER becomes an engine relation, so it does NOT participate in federation (cannot be joined or scanned by the engine as a table) and is orthogonal to the virtual|scan|materialized strategies. Tier exclusivity ([REQ-241](#REQ-241)) chooses it over the materialization store. (5) FEDERATION ENGINE — the execution substrate ([REQ-825](#REQ-825)); tenant-dedicated or shared. Each is independently pluggable: on a developer desktop components 1-4 collapse into one embedded engine (DuckDB/SQLite) and #5 is DuckDB (zero infra); in production they scale out independently (platform registry in Postgres, tenant registries sharded, materialization store in Iceberg/S3, hot cache in Redis, engine in Trino/Snowflake). Tenant isolation for components 2-5 MUST be enforced in the app layer ([REQ-828](#REQ-828)), not delegated to any single store's native RLS, so isolation holds across every medium.

**Use case:** Names the stateful components multi-tenancy must isolate and deployment must place, separating them from the stateless execution pipeline, and makes each independently pluggable so the same platform collapses to zero-infra on a laptop and scales each tier independently in production with tenant isolation enforced uniformly in the app layer.

**Code:** `provisa/api/app.py`, `provisa/core/db.py`, `provisa/core/repositories/source.py`, `provisa/openapi/pg_cache.py`, `provisa/cache/hot_tables.py`, `provisa/cache/warm_tables.py`, `provisa/mv/registry.py`

**Tests:** `tests/unit/test_arch_invariants.py`

### REQ-831 · Deployment & Infrastructure {#REQ-831}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Demo datasources run as host processes with no Docker. The demo GraphQL service (Strawberry + Starlette ASGI, demo/graphql_server/server.py, port 4000) and the demo OpenAPI petstore service (Starlette ASGI, demo/petstore_server/server.py, port 18080) are launched as background uvicorn subprocesses by demo/run-demo-servers.sh {start|stop}, replacing the former docker-compose services (swaggerapi/petstore3 'petstore-mock' and the built 'graphql-demo' container in docker-compose.demo.yml). start-ui.sh --demo starts and stops them and exports PETSTORE_BASE_URL=http://localhost:18080/api/v3 and GRAPHQL_DEMO_URL=http://localhost:4000/graphql; config/provisa.yaml reads those env vars with defaults that preserve the old Docker service names for containerized runs. This removes the demo-source Docker dependency for local/desktop development, advancing the zero-infra deployment model ([REQ-829](#REQ-829), [REQ-830](#REQ-830)).

**Use case:** Lets a developer run the full demo (governed GraphQL + OpenAPI sources) on a laptop without Docker, consistent with the embedded-engine / embedded-store zero-infra desktop model.

**Code:** `demo/run-demo-servers.sh`, `demo/graphql_server/server.py`, `demo/petstore_server/server.py`, `start-ui.sh`, `start-ui-install.sh`, `config/provisa.yaml`

**Tests:** `tests/unit/test_demo_servers.py`

### REQ-832 · Data Model & Persistence {#REQ-832}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Control-plane database is accessed through a SQLAlchemy-Core abstraction (provisa/core/database.py: Database/Connection/Row) backed by an AsyncEngine, replacing raw asyncpg. Connection pool replaces asyncpg.create_pool with equivalent behavior.

**Use case:** Enables multi-database support (PostgreSQL Tier-1, SQLite/MySQL Tier-2) by centralizing SQL dialect translation and connection management in one module.

**Code:** `provisa/core/database.py`

**Tests:** `tests/unit/test_admin_database.py`

### REQ-833 · Data Model & Persistence {#REQ-833}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Control-plane data model is split into two SQLAlchemy MetaData registries—platform control plane (orgs, user_profiles, user_org_memberships, local_users, org_invites, tenants, tenant_config) and per-org tenant control plane (sources, registered_tables, table_columns, relationships, roles, rls_rules, domains, materialized_views, kafka_*, api_*, creation_requests, tracked_functions/webhooks, live_query_state, node_ids, rel_ids, query_audit_log, etc.). Exposed at runtime as state.admin_db and state.org_db.

**Use case:** Separates multi-tenant platform metadata from per-tenant schema, enabling per-tenant state isolation and independent schema evolution.

**Code:** `provisa/core/schema_admin.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_schema_metadata_parity.py`

### REQ-834 · Data Model & Persistence {#REQ-834}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Portable-type policy for cross-backend support maps PG-specific types to portable equivalents (JSONB/TEXT[] → JSON, SERIAL/BIGSERIAL → Integer/BigInteger identity, TIMESTAMPTZ → DateTime(timezone), UUID → Uuid, BYTEA → LargeBinary). Dialect-aware upsert() and insert_returning() helpers replace PG-only ON CONFLICT/RETURNING.

**Use case:** Ensures Tier-1 PostgreSQL, Tier-2 SQLite ≥3.35, and Tier-2 MySQL 8 can all execute the same control-plane schema without migration.

**Code:** `provisa/core/database.py`

**Tests:** `tests/unit/test_admin_database.py`

### REQ-835 · Data Model & Persistence {#REQ-835}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A Capabilities flag object gates database features by dialect—LISTEN/NOTIFY, advisory locks, arrays, append-only RULEs, RETURNING are PG-only and must not be used on SQLite or MySQL backends.

**Use case:** Prevents runtime errors when control plane is deployed on non-PostgreSQL backends by enforcing compile-time feature gating.

**Code:** `provisa/core/database.py`

**Tests:** `tests/unit/test_admin_database.py`

### REQ-836 · Docker Composition & Deployment {#REQ-836}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Trino FTE exchange spooling directory (provisa_exchange Docker named volume, mounted at /data/provisa/exchange) must be owned by uid 1000 before Trino starts. A trino-exchange-init busybox one-shot service in docker-compose.core.yml chowns the volume and runs before both trino coordinator and trino-worker via condition: service_completed_successfully.

**Use case:** Trino runs as uid 1000 but freshly-created named volumes are owned root:root, causing write failures and GENERIC_INTERNAL_ERROR on every FTE query (retry-policy=TASK). Pre-initialization via service dependency ensures filesystem readiness.

**Code:** `docker-compose.core.yml`

**Tests:** `tests/unit/test_trino_exchange_init.py`, `tests/integration/test_trino_fte_exchange.py`

### REQ-837 · Data Model & Persistence {#REQ-837}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Platform control plane and tenant control plane use two independent SQLAlchemy engines and connection pools (state.admin_db for platform registry; state.org_db for per-tenant data). Platform database is configured by PLATFORM_DATABASE_URL env var (full SQLAlchemy URI supporting any async backend: postgresql+asyncpg, sqlite+aiosqlite, mysql+aiomysql); required at startup with no fallback.

**Use case:** Enables deployment models where platform registry and per-tenant data live in separate databases on different backends (platform on SQLite/MySQL, tenant on PostgreSQL, or vice versa), allowing independent lifecycle and scaling.

**Code:** `provisa/core/database.py`, `provisa/core/schema_admin.py`, `provisa/api/app.py`, `provisa/auth/wiring.py`, `provisa/auth/middleware.py`, `provisa/auth/providers/basic.py`

**Tests:** `tests/unit/test_platform_tenant_engines.py`

### REQ-838 · Data Model & Persistence {#REQ-838}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Tenant control-plane isolation is a CONTRACT, not a mechanism: within a tenant-scoped control-plane connection (state.org_db bound to one org_id), no other org's rows are reachable — reads and writes see only the connecting org's data, and cross-org access requires explicitly binding a different org. The ENFORCEMENT MECHANISM is backend-relative, selected by the control-plane store's dialect and never assumed to be PostgreSQL-specific: PostgreSQL realizes it with a per-connection SET search_path to org_<org_id> plus a per-org role (physical schema isolation); single-tenant portable backends (SQLite / MySQL bootstrapped from schema_org) realize it as the sole default schema of a per-tenant database; a backend offering row-security may realize it via a tenant predicate. The invariant is identical across backends — only the realization differs, and unavailable mechanisms degrade explicitly, never silently ([REQ-889](#REQ-889)).

**Use case:** Per-tenant state is isolated without row-level filtering logic on every query, on whatever control-plane store the deployment chooses. The mechanism is chosen from the backend's capabilities (PG schema+role, single-tenant default schema, or row-security predicate) so the same isolation guarantee holds when the control-plane store is swapped ([REQ-850](#REQ-850)).

**Code:** `provisa/core/database.py`, `provisa/core/db.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_org_isolation.py`, `tests/unit/test_tenant_isolation_contract.py`

### REQ-839 · Data Model & Persistence {#REQ-839}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Platform registry tables (orgs, user_profiles, user_org_memberships, local_users, org_invites, tenants, tenant_config) are created from portable SQLAlchemy metadata (schema_admin.init_registry_schema via metadata.create_all), not raw SQL schema files. No cross-database foreign key constraints between platform and tenant planes (FKs from domains.org_id, roles.org_id, etc. dropped to plain columns). id/token generation for local_users and org_invites moved app-side (uuid4) instead of PG-specific gen_random_uuid() server defaults.

**Use case:** Ensures platform registry can run on any SQLAlchemy-supported async backend (PostgreSQL, SQLite ≥3.35, MySQL 8) without migration or schema.sql maintenance. Plain columns (no FK enforcement) allow platform and tenant planes to exist in separate physical databases with independent schemas.

**Code:** `provisa/core/schema_admin.py`, `provisa/api/auth_router.py`, `provisa/api/setup_router.py`, `provisa/api/admin/orgs_router.py`, `provisa/api/admin/local_users_router.py`, `provisa/api/admin/invites_router.py`

**Tests:** `tests/unit/test_admin_database.py`, `tests/integration/test_sqlalchemy_dialect.py`

## 4. Source Connectors

### REQ-840 · Federation Engine Abstraction {#REQ-840}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

The federation engine is a pluggable abstraction with per-engine drivers. Three driver classes exist: broad federators (Trino, reaches many external source types), partial federators (DuckDB, attaches a subset of external databases via extensions/scanners), and self-only warehouse-native engines (Snowflake, Oracle, Databricks — reach only their own store). Reachability is connector-presence-defined and binary: reachable(table) is true iff a connector exists for (selected_engine, table.source.type). The connector's mechanism — attach (reference in place) or land (materialize into the engine's reachable store) — is an internal property of the connector, not a reachability tier; warehouse-native engines have only land mechanisms (into self), broad/partial federators use attach where a connector supports it and land otherwise. A federation engine INSTANCE owns its connector collection — a map keyed by source_type (connectors: dict[source_type -> Connector]). Reachability is a lookup into that collection: reachable(table) == (table.source.type in engine.connectors). The three driver classes are defined by collection contents — broad federator (Trino: many source_types), partial federator (DuckDB: postgres/csv/json/parquet where an extension/scanner exists), self-only warehouse-native (Snowflake: own store only). Swapping the engine swaps the connector collection; planner/cache/freshness-gate logic is unchanged. Each Connector projects CatalogEntry rows into the engine catalog. DONE (2026-07): provisa/federation/engine.py::FederationEngine owns a connector collection keyed by source_type; reachable(source_type) is the binary connector-presence lookup; swapping the engine swaps the collection. driver_class() classifies by collection contents — build_trino_engine (postgres/mysql/sqlserver → BROAD), build_duckdb_engine (postgres + csv scanner → PARTIAL), build_snowflake_engine (land-into-self only → SELF_ONLY).

**Use case:** Support deployments whose engine ranges from a broad federator to a self-only data warehouse without changing planner or cache logic.

**Code:** `provisa/federation/engine.py`, `provisa/federation/connector.py`

**Tests:** `tests/unit/test_federation_engine.py`

### REQ-841 · Federation Engine Abstraction {#REQ-841}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

A reachable source (one with a connector for the selected engine) is exposed by that connector's mechanism — attach (reference the source in place, no data movement) or land (physically materialize the source into the engine's reachable store). The mechanism is fixed by the connector, not chosen per query; a source with no connector is rejected as unreachable.

**Code:** `provisa/federation/engine.py`, `provisa/federation/connector.py`

**Tests:** `tests/unit/test_federation_engine.py`

### REQ-842 · Connector Abstraction {#REQ-842}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

A Connector, indexed by (federation_engine, source_type), encapsulates all engine-specific catalog operations — capability(), catalog_add(asset), catalog_remove(asset), land(source), typemap(source->engine). For Trino, catalog_add updates the Trino catalog; for DuckDB it issues ATTACH/scanner setup plus view DDL; for warehouse-native engines it is a no-op (the asset is already a native table). Adding a source type, a federation engine, or a cache backend is a connector registration; planner, cache-role, and freshness-gate logic are unchanged. The connector's derived engine-catalog entry has an explicit persisted row shape: (name: text, engine/platform: text, source_type: text, mechanism: attach|land, details: jsonb). The connector exposes catalog_add, catalog_update, and catalog_remove operations over individual entries, plus a catalog_refresh operation that re-projects the entire engine catalog from the asset registry ([REQ-843](#REQ-843) reconcile). The details jsonb field is engine+source_type specific — e.g., Trino postgresql = connection-url/user /password .properties; DuckDB postgresql = ATTACH dsn; DuckDB csv/json = read_csv_auto/read_json_auto view DDL over file paths+columns; warehouse -native (Snowflake) = no-op land-into-self. Per the V1 no-migrations constraint, the catalog-entry table is derived/rebuildable engine state, not a migrated Postgres table. DONE (2026-07): provisa/federation/connector.py defines the Connector ABC keyed by (engine, source_type) with capability() and details(source), projecting the persisted CatalogEntry shape exactly — (name, engine, source_type, mechanism: attach|land, details: dict/jsonb). Concrete connectors realize the three engine families: Trino postgres/mysql/sqlserver (attach, connection-url/user/password .properties details), DuckDB postgres (attach dsn) + csv (read_csv_auto view DDL), and a warehouse-native land-into-self no-op. catalog_add/remove/refresh are the EngineCatalog.add/remove/refresh operations over these projected entries. REMAINING (kept in-progress): typemap(source→engine) per connector, and the live APPLICATION of a catalog entry to a running engine (writing Trino .properties / issuing DuckDB ATTACH+view DDL) — wired with the executor in the federate() work ([REQ-825](#REQ-825)-827).

**Use case:** A single extensibility seam so new engines/sources/backends require only a connector, not core changes.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_federation_engine.py`

### REQ-843 · Connector Abstraction {#REQ-843}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The asset registry is the source of truth for asset existence; the engine catalog is derived, rebuildable state. Connectors project registry state into the engine catalog on asset create/drop and on a full startup reconcile. A missing or stale catalog entry triggers re-projection from the registry — never a fallback or error-and-continue. DONE (2026-07): EngineCatalog is derived state (name → CatalogEntry). FederationEngine.reconcile rebuilds the whole projection from the source registry (only reachable sources project); ensure_entry re-projects when the entry is missing or stale (!= the fresh projection) — never keeps stale, never falls back. on_asset_create/on_asset_drop keep the catalog in step with registry mutations.

**Code:** `provisa/federation/engine.py`, `provisa/federation/connector.py`

**Tests:** `tests/unit/test_federation_engine.py`

### REQ-844 · Materialization Store {#REQ-844}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

materialization_store — materialized (landed physically by ANY write means: engine-native CTAS/load, SQLAlchemy upsert, or app-side land; [REQ-848](#REQ-848)) into a durable STORE. Configured through the same ${env:...} secrets chokepoint as ControlPlaneConfig. Its members are characterized by three properties that together exclude every other cache: FEDERATABLE (the engine reads it as a real relation it can join/scan — [REQ-826](#REQ-826); excludes hot tables, which are VALUES-CTE plan injections, and the APQ/result memos, which are not relations); DURABLE (survives restart, disk/object-backed — excludes the ephemeral tier); and REFRESHABLE (actively kept in sync by the one centralized freshness gate — [REQ-855](#REQ-855)/856/857/858; excludes a passive scanned source, which is federatable and durable but unmanaged). It holds TWO ROLES on ONE categorical axis — whether the entry has a semantic-SQL definition: (1) REPLICA — undefined verbatim copies of source rows, no standalone identity; governance/derivation applied at query time on top, never baked in. Two population profiles: reactive (source is UNREACHABLE — landed so it is queryable at all; on-miss, single-flighted pull-through; [REQ-845](#REQ-845)/847; formerly api_cache, renamed as the role is source-agnostic) and warm (source is REACHABLE — an optional latency copy; usage-promoted/demoted by query frequency; [REQ-238](#REQ-238)/239). (2) VIEW — mv: governed, named datasets each defined by semantic SQL (view_sql; implicit relationship-generated or explicit view materialize:true). The definition IS the identity and the basis for lineage, provenance ([REQ-862](#REQ-862)), and definition-versioning. Two orthogonal, ENGINE-RELATIVE attributes apply to any entry and are NOT roles: PLACEMENT — where the entry lands, chosen relative to engine capability (object store, a low-latency fast-read tier, engine-native store, or Iceberg); warm's "fast tier" is just placement=engine's-low-latency-store (Trino realizes it as the Iceberg results catalog + local-SSD file cache, other engines differently), never a Trino-specific definition. FRESHNESS POLICY — TTL / probe / transitive ([REQ-855](#REQ-855)). Population trigger and eviction are NOT roles: both roles are lazy and use the one freshness method; the view background refresh loop and warm's frequency promotion are optional warm-ahead only. SUBSTRATE is selected independently per role (roles may occupy distinct schemas of one backend OR distinct backends), engine-capability-gated ([REQ-846](#REQ-846)): replica → a relational/engine-native store suffices (undefined copies have no forensic value); view → SQLAlchemy DB OR engine-native Iceberg, defaulting to Iceberg whenever a reachable Iceberg catalog exists, because the derived dataset's snapshot history is the forensic/bitemporal value (time-travel, provenance, restatement; [REQ-372](#REQ-372)/862). Iceberg view refresh MUST use overwrite semantics — one clean snapshot per refresh (not delete+insert) — so time-travel maps exactly one snapshot to one refresh state. When no Iceberg catalog is reachable the substrate degrades to the relational/native store as an explicit, surfaced capability signal (never a silent fallback; [REQ-847](#REQ-847)).

**Use case:** One engine-agnostic materialization store replacing per-protocol hardcoded caches, defined by three properties (federatable, durable, refreshable) that cleanly separate it from the ephemeral hot/APQ/result tiers and from passive scan sources. The replica-vs-view axis drives substrate: cheap relational storage for undefined source copies (reactive/warm), snapshot-capable Iceberg for the governed derived datasets whose history underpins audit, provenance, and bitemporal restatement. Supports security-conscious tenants pointing the store at their own data plane.

**Code:** `provisa/federation/materialization.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_federation_materialization.py`, `tests/integration/test_materialization_store_lifecycle_e2e.py`

### REQ-845 · Materialization Store {#REQ-845}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The set of sources served by reactive replicas is engine-relative — reactive = { source : reach(source) == land } for the configured federation engine — generalizing the previously hardcoded REST/GraphQL/gRPC pull-through set (renamed from api_cache; the role is source-agnostic). Pull-through is TTL/freshness-gated (per the [REQ-855](#REQ-855) freshness policy) and single-flighted to prevent upstream stampede on concurrent misses. DONE (2026-07): the engine-relative reactive set is derived — provisa/federation/materialization.py:: reactive_sources returns { source.id : federate(source, engine) is MATERIALIZED }, so the pull-through set is a function of engine reach (a VIRTUAL/SCAN source is never a reactive replica; an unreachable source is excluded). REMAINING (kept in-progress): the TTL/freshness-gated, single-flighted pull-through execution that lands rows into the store on miss — lands with the write face ([REQ-848](#REQ-848)) and the freshness gate ([REQ-855](#REQ-855)).

**Code:** `provisa/federation/materialization.py`, `provisa/federation/strategy.py`

**Tests:** `tests/unit/test_federation_materialization.py`, `tests/integration/test_materialization_store_lifecycle_e2e.py`

### REQ-846 · Materialization Store {#REQ-846}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The engine's usable materialized-store set is DERIVED from its connectors — a `materialized_store` flag on a Connector marks a reachable backend that can also be a store; FederationEngine.materialize_stores is a computed property over the connectors so flagged. PG-only today. Native→IR type translation for MATERIALIZED source landing occurs at the landing boundary in `resolve_landing_args` (provisa/federation/residency.py), not in the shared config/repository write path. Each column's stored engine-normalized native type is translated to a canonical IR name via `provisa.core.ir_types.to_ir(native, platform)`, where `platform` is the federation engine's dialect. This containment is necessary because stored `data_type` across the system is engine-physical and consumed as such by GraphQL/SQL generation, masking, and filters. Unmapped native types raise (IR vocabulary gap), never silent varchar defaults. DONE (2026-07): provisa/federation/connector.py (materialized_store flag on PG connectors), provisa/federation/engine.py (materialize_stores property), provisa/federation/residency.py (resolve_landing_args with to_ir translation).

**Code:** `provisa/federation/materialization.py`, `provisa/federation/engine.py`, `provisa/federation/residency.py`

**Tests:** `tests/unit/test_federation_materialization.py`, `tests/unit/test_residency.py`

### REQ-847 · Materialization Store {#REQ-847}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Read-through and write-back discipline. On a pull-through read, a pull failure with no fresh cached data is a hard error; serving stale data is permitted only under an explicit per-source freshness policy ([REQ-855](#REQ-855)), never as a silent fallback. Mutations on a pulled-through relation target the upstream source of truth and invalidate the cache entry; the cache is never written as if it were the system of record. DONE (2026-07): the discipline is implemented as pure policy — provisa/federation/read_through.py:: resolve_read returns SERVE_FRESH (re-pull ok, or cache still fresh), SERVE_STALE (only when stale cache exists AND an explicit per-source stale policy allows it), or HARD_ERROR (pull failed with no fresh data — never a silent stale fallback); plan_mutation states a write targets "upstream" and invalidates the cache entry, never the cache as system of record. REMAINING (kept in-progress): wiring these decisions into the live pull-through read path and the mutation executor.

**Code:** `provisa/federation/read_through.py`

**Tests:** `tests/unit/test_read_through.py`, `tests/integration/test_materialization_store_lifecycle_e2e.py`

### REQ-848 · Materialization Store {#REQ-848}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The federation engine NEVER writes a materialization store; it merely attaches/links and READS the landed replica. ALL data-source landing goes through ONE write-face abstraction (store_writer.land / ensure_table / reconcile_table), vanilla SQLAlchemy Core today, with pluggable platform branches behind it (pyiceberg for an Iceberg store). The "collapse into the engine" applies only to an engine writing its OWN native store, never a separate/attached one. Read-only engines (Databricks/Snowflake external-linking Iceberg) are first-class readers. DONE (2026-07): provisa/federation/store_writer.py, provisa/federation/materialize_exec.py, provisa/api_source/engine_cache.py (land_api_cache), provisa/federation/duckdb_runtime.py.

**Code:** `provisa/federation/materialization.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_federation_materialization.py`, `tests/integration/test_materialization_store_lifecycle_e2e.py`

## 3. Source Registration & Data Modeling

### REQ-849 · Views (Governed Computed Datasets) {#REQ-849}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

After SQL compilation, the query rewriter inspects FROM/JOIN clauses and transparently rewrites the query to read from a materialized view target table when a matching MV exists — without the query author referencing the MV by name. Rewrites are gated by MV enabled status, freshness, and TTL. Match selection is coverage-optimal and independent of candidate ordering: a full match (all query JOINs covered → entire query redirected to the MV) is always preferred over any partial match, and among partial matches the widest coverage (most MV-covered JOINs replaced, remainder preserved) wins. A fully-matching MV is never preempted by an earlier partially-matching one.

**Use case:** Transparent MV substitution improves query performance without requiring application changes; TTL gating ensures stale data is not served without explicit policy.

**Code:** `provisa/mv/rewriter.py`, `provisa/mv/registry.py`

**Tests:** `tests/unit/test_mv_partial_rewrite.py`

## 11. Platform, Infrastructure & Delivery

### REQ-850 · Control-Plane Store Portability {#REQ-850}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Control-plane store is SQLAlchemy-URI-pluggable across dialects, bootstrapped from portable SQLAlchemy Core metadata (provisa/core/schema_org.py) via metadata.create_all, replacing PG-only raw schema.sql (which uses SERIAL, JSONB, DO $$/pg_advisory_lock, CREATE SCHEMA, search_path). PostgreSQL retains the raw schema.sql path unchanged.

**Use case:** Allows control plane to deploy on non-PostgreSQL backends (SQLite, MySQL, DuckDB, Oracle) without migration, supporting infra-free deployments while maintaining backward compatibility with Postgres.

**Code:** `provisa/core/db.py`, `provisa/core/schema_org.py`, `provisa/audit/query_log.py`

**Tests:** `tests/unit/test_schema_metadata_parity.py`

### REQ-851 · Multi-Tenant Org Isolation {#REQ-851}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Org isolation is polymorphic on a schema-capable vs not-schema-capable axis. Schema-capable backends (PostgreSQL, MySQL, Oracle) scope an org as a namespace on a single shared connection via SQL commands (SET search_path TO org_<id>, USE org_<id>, ALTER SESSION SET CURRENT_SCHEMA). Not-schema-capable backends (SQLite, DuckDB) carry org in the database file; org selection happens at engine construction via per-org file paths.

**Use case:** Provides unified multi-tenant abstraction across heterogeneous databases while respecting native schema isolation capabilities, enabling schema-based multi-tenancy on Postgres/MySQL and file-per-org isolation on SQLite/DuckDB.

**Code:** `provisa/core/database.py`

**Tests:** `tests/unit/test_org_isolation.py`, `tests/unit/test_tenant_isolation_contract.py`

### REQ-852 · Multi-Tenant Org Isolation {#REQ-852}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

OrgRouter (provisa/core/database.py) is the multi-tenant mechanism for not-schema-capable backends, mapping org_id → Database and lazily caching one engine per org_<id> database file. OrgRouter guards against construction on schema-capable backends and enforces that single-tenant SQLite uses the default namespace with no router. Cross-org queries within a single not-schema-capable connection are impossible (file boundary).

**Use case:** Enables transparent multi-tenant routing for file-based backends (SQLite, DuckDB) while preventing misconfiguration on schema-capable systems.

**Code:** `provisa/core/database.py`

**Tests:** `tests/unit/test_org_router.py`

### REQ-853 · Control-Plane Store Portability {#REQ-853}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

NOT NULL JSON columns in schema_org.py carry server_default values ('[]' for list, '{}' for dict) so raw-text INSERTs that omit them succeed on portable backends (previously only Python-side Core defaults existed, causing raw inserts to fail NOT NULL constraints on SQLite).

**Use case:** Ensures compatibility with raw SQL inserts across all backends, preventing NOT NULL violations when JSON columns are omitted in direct SQL statements.

**Code:** `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_schema_metadata_parity.py`, `tests/unit/test_admin_database.py`

### REQ-854 · Release Artifact Packaging {#REQ-854}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Core runtime image set (docker-compose.core.yml + OVA/Lima bundles) must include busybox as a core dependency; no third-party billing mock service is bundled in any compose (dev/core/airgap); python:3.12-slim is build-time only and excluded from packaged/airgapped runtime images.

**Use case:** Busybox is required at runtime (trino-exchange-init service uses it to chown /data/provisa/exchange before Trino starts). Lemon Squeezy is called over the public REST API; tests stub it via HTTP fixtures (LEMONSQUEEZY_BASE_URL), not bundled mocks. Python:3.12-slim exists only as a build base for OVA/Lima source builds. This separation ensures airgapped deployments include all runtime dependencies and excludes unnecessary dev/mock components.

**Code:** `docker-compose.core.yml`, `docker-compose.dev.yml`, `.github/workflows/build-dmg.yml`

**Tests:** `tests/unit/test_release_packaging.py`

## 4. Source Connectors

### REQ-855 · Materialization Store {#REQ-855}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

materialization_store freshness policy — the ONE centralized freshness gate referenced by [REQ-845](#REQ-845) (reactive replica pull-through) and applicable to [REQ-844](#REQ-844)'s view role; both roles evaluate it lazily on access. Per cache entry (per-table, keyed like the existing replica/view targets, NOT per-query-slice) the steward selects one of three modes: (1) TTL only — bounded staleness, current behavior (reactive-replica _mem_fresh expiry / view refresh interval); (2) probe — re-probe the upstream on access and rebuild only on change, ignoring TTL; (3) TTL+probe — probe only after a TTL floor elapses, capping probe frequency. The probe returns an OPAQUE token; Provisa never interprets it, only compares stored vs. fresh by equality. Unchanged token → keep the materialized rows, skip the re-pull/rebuild (zero lag at probe cost); changed token → invalidate the entry and re-pull, then store the new token. MUTUAL EXCLUSIVITY with delta ([REQ-874](#REQ-874)): Probe (opaque-token change detection) and delta (monotonic-cursor incremental reload) are MUTUALLY EXCLUSIVE PER ENTRY, selected by token type. Opaque-token entries ⇒ REQ-855 probe + full re-pull, no delta. Monotonic entries ⇒ delta-as-probe (delta query IS the freshness evaluation), no separate probe runs. Probe transport is source-type-agnostic behind a single dispatch (freshness_token(source, table) -> str | None); None means the source cannot produce a token and the entry degrades to TTL (a capability signal, not a silent fallback — [REQ-847](#REQ-847)). Transports: graphql_remote posts a steward GQL query (generalizing the [REQ-673](#REQ-673) count_query/graph_counts prototype); openapi uses ETag/Last-Modified or a steward-configured endpoint; DB-SQL sources run a steward SQL query via the engine; file-based sources use file mtime.

**Use case:** Lets stewards trade probe cost against TTL lag per cached source/table — zero-lag freshness for volatile pull-through sources and materialized views where lag is unacceptable, cheap TTL for static ones — inside the one materialization_store gate ([REQ-845](#REQ-845)) rather than a per-protocol bolt-on. view targets ([REQ-844](#REQ-844) derivation role) may carry both a TTL/refresh interval and a freshness gate; the gate suppresses needless CTAS rebuilds when the upstream token is unchanged. DONE (2026-07): the centralized gate is implemented — provisa/federation/freshness_gate.py:: evaluate_freshness(mode, now, last_refresh_at, ttl, stored_token, probe) returns a FreshnessDecision (fresh + new_token) for the three modes: TTL (bounded staleness), PROBE (probe every access, ignore TTL), TTL_PROBE (probe only past the TTL floor). The probe yields an opaque token compared by equality — unchanged keeps the rows (zero lag), changed marks not-fresh with the new token to persist — and a probe returning None degrades the entry to TTL (a capability signal, never a silent stale fallback, [REQ-847](#REQ-847)). Pure and reused by both the reactive-replica ([REQ-845](#REQ-845)) and view ([REQ-844](#REQ-844)) roles. REMAINING (kept in-progress): the per-source probe TRANSPORTS behind freshness_token(source, table) -> str | None (graphql query, openapi ETag/Last-Modified, DB-SQL steward query, file mtime) and wiring the gate into the store's lazy-on-access path + delta mutual-exclusivity ([REQ-874](#REQ-874)).

**Code:** `provisa/federation/freshness_gate.py`

**Tests:** `tests/unit/test_freshness_gate.py`, `tests/integration/test_materialization_store_lifecycle_e2e.py`

### REQ-856 · Freshness Module {#REQ-856}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Freshness gate ([REQ-855](#REQ-855): three modes TTL/probe/TTL+probe, source-agnostic freshness_token dispatch, transports incl. DB-SQL steward query and file mtime) is generalized into a standalone, context-agnostic module answering one question: given a subject and its observable state, is it fresh? Returns fresh/stale/failed + reason. Pure decision function with no triggering or refresh semantics (caller responsibilities). Enables MV, Source, cache, pgvector all consume one implementation.

**Use case:** Unifies freshness logic currently scattered across MV (TTL-based, provisa/mv/models.py:94-102), Source introspection, and ad-hoc cache checks (provisa/openapi/pg_cache.py:49). Single implementation enables consistent freshness gating across all data paths.

**Code:** `provisa/freshness/__init__.py`, `provisa/freshness/decision.py`, `provisa/freshness/predicate.py`

**Tests:** `tests/unit/test_freshness.py`

### REQ-857 · Freshness Module {#REQ-857}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

FreshnessSubject protocol exposes: (1) last-refresh timestamp; (2) last-refresh outcome (success/failure); (3) upstream-subject handles for transitive freshness; (4) content-query capability — this IS [REQ-855](#REQ-855)'s freshness_token(source,table) -> str | None dispatch (opaque token; None = source cannot produce one, degrades to TTL per [REQ-847](#REQ-847)).

**Use case:** Enables FreshnessPredicate to evaluate any subject uniformly—MV, Source, cache all conform to the same interface. Transitive strategy can check upstream freshness by following handles.

**Code:** `provisa/freshness/subject.py`

**Tests:** `tests/unit/test_freshness.py`

### REQ-858 · Freshness Module {#REQ-858}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

FreshnessPredicate with pluggable strategies that unify [REQ-855](#REQ-855)'s three modes plus a boolean content predicate: TTL (time-based, exists today); PROBE ([REQ-855](#REQ-855) opaque- token change detection via freshness_token — the same mechanism as [REQ-855](#REQ-855)'s DB-SQL steward-query transport, do not treat as a separate "SQL strategy"); TRANSITIVE (fresh iff own predicate holds AND all upstream subjects are transitively fresh — the one net-new capability beyond [REQ-855](#REQ-855), which is per-entry only); composable (TTL+PROBE floors probe frequency). Caller supplies subject and strategy; predicate returns decision without side effects.

**Use case:** Decouples freshness evaluation from refresh logic. Strategies are swappable and composable, enabling rich policies (e.g., TTL for cost-sensitive tables, PROBE for zero-lag accuracy, TRANSITIVE for derived caches/medallion chains).

**Code:** `provisa/freshness/predicate.py`

**Tests:** `tests/unit/test_freshness.py`

### REQ-859 · Freshness Module {#REQ-859}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

MV and the ad-hoc API/pg cache freshness checks (provisa/openapi/pg_cache.py) become FreshnessSubjects (via the StateSubject adapter) and delegate their TTL decision to the one unified FreshnessPredicate. MVDefinition.is_fresh_at keeps its FRESH-status lifecycle gate and delegates the TTL comparison; pg_cache._is_fresh delegates likewise. Behaviour-preserving unification of the scattered TTL logic. Source-level conformance is the gating work in [REQ-860](#REQ-860) (its interesting PROBE mode depends on the [REQ-855](#REQ-855) probe transport).

**Use case:** Consistency: the same freshness semantics and strategies apply to every data path, and there is one place to change freshness policy instead of per-consumer copies.

**Code:** `provisa/freshness/adapters.py`, `provisa/mv/models.py`, `provisa/openapi/pg_cache.py`

**Tests:** `tests/unit/test_freshness_adapters.py`

### REQ-860 · Freshness Gating {#REQ-860}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Freshness gating is now available at the Source level (provisa/core/models.py), not just on MV. A source can declare a freshness predicate; queries reading from that source are gated by freshness before execution.

**Use case:** Prevents serving stale data from pull-through sources. Stewards can enforce freshness without materializing into an MV, reducing latency and compute overhead.

**Code:** `provisa/core/models.py`, `provisa/freshness/source_gate.py`, `provisa/federation/plan.py`, `provisa/federation/native_backend.py`

**Tests:** `tests/unit/test_source_freshness_gate.py`

### REQ-861 · Freshness Gating {#REQ-861}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

File-based sources (CSV, Parquet, etc.) may carry an optional producer command (argv) that runs on-stale (gate-triggered) before the file is read. Allows refresh without modifying the file path or materializing into an MV. DONE (2026-07) - a file source (csv/parquet/sqlite/files) carries an optional producer_command argv (provisa/core/models.py); provisa/freshness/producer.py runs it with shell=False (injection-safe) and fails loud on a non-zero exit. The read path (provisa/events/source_loader.py SourceRowLoader.load) is invoked only after the [REQ-860](#REQ-860) source gate reports stale, so it runs the producer BEFORE the file read - freshening the file in place without touching path or defining an MV.

**Use case:** Supports workflows where stale file detection should trigger a script to refresh the file in place. Example: fetch latest data from an upstream API and write to the same CSV path when staleness is detected. Pairs with [REQ-855](#REQ-855): [REQ-855](#REQ-855) file mtime DETECTS staleness, this producer command is the ACT-on-stale response.

**Code:** `provisa/core/models.py`, `provisa/freshness/producer.py`, `provisa/events/source_loader.py`

**Tests:** `tests/unit/test_file_producer_command.py`

## 5. Data Lineage & Provenance

### REQ-862 · Column-Level Trace Instrumentation {#REQ-862}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

MV creation/refresh is instrumented with detailed column-level trace records: for each refresh, the emitted OTel spans capture the resolved per-output-column derivation (upstream source column(s) and transform, from the view_sql), the input snapshot ids/watermark epochs consumed, the active definition-version, and the refresh trace_id. These records land in the governed telemetry substrate (OTel→Parquet, otel catalog). The system does not provide a lineage-resolution engine; lineage is DERIVED BY THE USER querying these trace records (optionally materialized as an MV), which — combined with Iceberg time-travel — yields point-in-time, column-level lineage as a query result. DONE (2026-07): the mv.refresh.column_lineage span now carries all four store-independent stamps — per-output-column derivation (provisa/lineage/columns.py, sqlglot), the definition-version content hash (provisa/lineage/versions.py), the resolved input-version + fidelity kind, and the trace_id. The final piece — actual per-source signal gathering — is wired: at refresh (provisa/mv/input_signals.py::gather_input_signals) each source table is asked, through the same Trino connection, for the strongest signal it offers — an Iceberg snapshot id from "<table>$snapshots", else an RDB watermark MAX(watermark_column) with the column discovered from the provisa_admin registry catalog ([REQ-260](#REQ-260)) — so input_version reflects the real data version rather than always degrading to the refresh epoch. Best-effort telemetry: sources offering neither contribute nothing and any query error is skipped, never failing the refresh; resolve_input_version grades iceberg_snapshot > watermark > freshness_token > refresh_epoch. Enables audit, impact analysis, and compliance workflows without a bespoke lineage service: because the trace substrate is itself queryable and time-travelable, users answer "where did this column value come from, as of time T" by querying the column-level MV-refresh traces rather than by invoking a dedicated lineage API.

**Code:** `provisa/lineage/columns.py`, `provisa/lineage/versions.py`, `provisa/mv/refresh.py`, `provisa/mv/input_signals.py`, `provisa/mv/registry.py`, `provisa/transpiler/`, `provisa/mv/aggregate_catalog.py`

**Tests:** `tests/unit/test_lineage_columns.py`, `tests/unit/test_mv_input_signals.py`

## 6. Execution, Routing, Caching & Performance

### REQ-863 · Query Planning Pipeline {#REQ-863}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Query planning phases are ordered so that source-set-mutating transforms complete before routing. The pipeline is: query language -> semantic SQL (IR) -> governance (which may ADD sources via RLS subquery predicates) -> a post-governance optimization stage (which may REMOVE sources, e.g. hot-table VALUES-CTE inlining, API-cache rewrites, union-branch pruning) -> route decision + physical lowering -> execute. Routing (extract_sources / decide_route) MUST consume the output of the post-governance optimization stage, never the pre-optimization governed SQL. Consequence: when hot-CTE inlining collapses a federated (multi-source) query to a single remaining live source, routing observes one source and selects DIRECT execution instead of Trino federation; the VALUES CTEs are then lowered in the chosen route's dialect (source dialect for direct pushdown, Trino for federated).

**Use case:** Direct single-source execution is materially faster than Trino federation. Today the route is decided on the governed SQL BEFORE hot-CTE inlining runs (inlining is nested inside the Route.TRINO branch), so a query whose second source is fully inlined still executes federated. Correct phase ordering makes the DIRECT collapse fall out naturally instead of requiring special-case logic, and unblocks the pushdown of inlined lookups on the direct path. The optimization stage sits after governance because hot inlining depends on the governed logical table reference ([REQ-233](#REQ-233): governance binds to the table name, then inlining swaps in the CTE, and the governed wrapper filters/masks the CTE rows).

**Code:** `provisa/pgwire/_pipeline.py`, `provisa/transpiler/router.py`, `provisa/compiler/sql_gen.py`, `provisa/cache/hot_tables.py`, `provisa/compiler/stage2.py`, `provisa/compiler/sql_rewrite.py`

**Tests:** `tests/unit/test_pipeline_order.py`

### REQ-864 · Query Result Cache {#REQ-864}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The query result cache key is computed from a NORMALIZED IR — a canonical, AST-normalized form of the GOVERNANCE-NORMALIZED IR (the post-Stage-2 governed SQL — [REQ-263](#REQ-263) — after RLS, masking, and column visibility are applied with per-identity values resolved, [REQ-041](#REQ-041)/[REQ-402](#REQ-402)) — rather than the serialized compiled-SQL string it hashes today (provisa/cache/key.py hashes compiled.sql as text). Normalization canonicalizes cosmetic variation that does not change results (whitespace, alias naming/ordering, literal formatting, commutable predicate/JOIN ordering, projection ordering) so two semantically-identical queries resolve to the SAME cache entry instead of missing. Because the keyed form is the governed IR, the resolved persona — role, RLS predicate values, masking, and column visibility — is ALREADY embedded in it (no separate concatenation), and the per-tenant prefix ([REQ-595](#REQ-595)) is applied on top; the RLS-unresolved guard (key.py ValueError) is retained. CRITICAL: normalization canonicalizes cosmetic/structural form ONLY and MUST preserve every identity-encoding element — the RESOLVED RLS predicate values, role, masking, and column visibility — so it can never collapse two personas onto one key (the hypercritical isolation constraint [REQ-866](#REQ-866)). Refines, does not replace, the existing result cache ([REQ-544](#REQ-544)).

**Use case:** Raises cache hit-rate and eliminates duplicate entries for queries that differ only in text, making the result cache robust to cosmetic query variation from different clients, formatters, or GraphQL shapes that compile to equivalent IR.

**Code:** `provisa/cache/key.py`, `provisa/api/data/endpoint.py`

**Tests:** `tests/unit/test_cache_key.py`

### REQ-865 · Query Routing {#REQ-865}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The route decision includes CACHED as a first-class route alongside DIRECT ([REQ-027](#REQ-027)) and FEDERATED/Trino ([REQ-028](#REQ-028)): the Route enum gains Route.CACHE and decide_route (provisa/transpiler/router.py) evaluates the result cache — keyed per [REQ-864](#REQ-864)/[REQ-544](#REQ-544) — as the first candidate route. A cache hit resolves to Route.CACHE and serves the stored result with no direct or federated execution; a miss falls through to DIRECT/FEDERATED as today. This elevates the current outer-wrapper cache check (which runs before decide_route in endpoint.py and is invisible to routing) into the routing model, so every query is served by exactly one uniformly-decided route: {cached, direct, federated}. CACHED belongs in the route decision precisely BECAUSE the routing point is where the query is GOVERNANCE-NORMALIZED: routing consumes the post-Stage-2 governed IR ([REQ-263](#REQ-263), persona-resolved via [REQ-041](#REQ-041)/[REQ-402](#REQ-402)), so the cache key derived there ALREADY encodes the requester's persona and a cache serve is inherently isolated ([REQ-866](#REQ-866)) — a pre-governance wrapper cannot guarantee that structurally. The no-cache bypass (@noCache / no_cache / @cached(ttl:0); [REQ-544](#REQ-544)) removes CACHED from the candidate set for that query. Routing consumes the post-optimization IR ([REQ-863](#REQ-863)), so cache-key computation and route selection operate on the same normalized form.

**Use case:** Unifies the serving model — one decision point owns how every query is served — makes cache participation observable in the route/plan (traceable like any route), and lets routing policy reason about the cache alongside direct and federated execution instead of it being a hidden pre-step.

**Code:** `provisa/transpiler/router.py`, `provisa/api/data/endpoint.py`, `provisa/cache/store.py`

**Tests:** `tests/unit/test_routing.py`

## 1. Access Governance & Security

### REQ-866 · Cache Isolation {#REQ-866}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

HYPERCRITICAL isolation invariant for the query result cache. The cache key MUST fully partition by the RESOLVED governance identity of the requester, so a cache hit can NEVER serve rows computed for a different persona. This matters because personas can switch on the same device, connection, or pooled session; a mis-scoped key would leak one user's data to another. The key MUST bind to: the per-tenant prefix ([REQ-595](#REQ-595)), role_id, and the fully-RESOLVED RLS predicates plus masking and column visibility — i.e. RLS current_setting/session values substituted to their actual per-identity values ([REQ-041](#REQ-041)/[REQ-402](#REQ-402)), NOT the unresolved rule template (identical across users, which would collapse distinct personas onto one key). This isolation is STRUCTURAL when the cache is a first-class route ([REQ-865](#REQ-865)): the route decision operates on the governance-normalized IR, so the key intrinsically encodes the persona. IR/SQL normalization ([REQ-864](#REQ-864)) canonicalizes cosmetic and structural form ONLY and MUST preserve every identity-encoding element. FAIL-CLOSED: if any identity dimension is unresolved or unknown, the cache MUST NOT be read or written for that query (the key.py RLS-unresolved guard), and CACHED ([REQ-865](#REQ-865)) is removed from the candidate routes — never a silent fallback that could serve another persona's data.

**Use case:** Prevents cross-persona / cross-user data leakage through the result cache when identities change on a shared device, connection, or pooled session — a hard multi-tenant and RLS isolation guarantee, consistent with the two-layer governance model ([REQ-038](#REQ-038)) and Stage-2 RLS/masking enforcement ([REQ-263](#REQ-263)).

**Code:** `provisa/cache/key.py`, `provisa/api/data/endpoint.py`

**Tests:** `tests/unit/test_cache_key.py`

## 2. Authentication & Identity

### REQ-867 · Authorization {#REQ-867}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Table-scoped mutation registration with per-mutation grants across all remote schema adapters (GraphQL remote, OpenAPI, gRPC, Hasura). Mutations are registered as first-class sub-resources, each with its own writable_by list. Empty writable_by = default-deny. The authorization model is protocol-agnostic; shared/universal layer includes table-scoped mutation sub-resources, per-mutation writable_by, global WRITE capability, execute-time enforcement, read-statement taint, ranked-candidate suggestion interface, admin-confirm, and default-deny. What is per-adapter is only the signals used for classification and association.

**Use case:** Enables fine-grained authorization of table mutations independent of the table's read visibility across all query sources (GraphQL, OpenAPI, gRPC, Hasura), supporting mutation execution as UDFs in the semantic layer.

**Code:** `provisa/security/mutation_authz.py`, `provisa/security/rights.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_mutation_authz.py`

### REQ-868 · Authorization {#REQ-868}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Global WRITE (alias EXECUTE_MUTATION) capability added to the Capability enum. Acts as a coarse master switch; role must hold WRITE AND appear in the specific mutation's writable_by to invoke a mutation.

**Use case:** Two-gate authorization model: coarse capability check (does role have WRITE at all?) plus fine-grained mutation-level grant check (is role in this mutation's writable_by?). Analogous to Postgres role-level rights + per-table GRANT.

**Code:** `provisa/security/rights.py`

**Tests:** `tests/unit/test_mutation_authz.py`

### REQ-869 · Authorization {#REQ-869}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Mutations are classified as WRITES by contract (not caller declaration) via protocol-specific classifiers. Per-adapter read/write classifiers: GraphQL uses operation type (type Mutation), OpenAPI uses HTTP method (POST/PUT/PATCH/DELETE = write; GET = read) plus x-provisa-kind override, gRPC uses MethodOptions.idempotency_level (NO_SIDE_EFFECTS = read; IDEMPOTENCY_UNKNOWN and IDEMPOTENT = write; AIP naming and server-streaming are non-normative heuristics only), Hasura uses exposed_as/action_type. Universal default: unknown/unclassifiable → treat as write (default-deny). Any read statement (SELECT) referencing a kind=mutation UDF is promoted to write and subject to both WRITE capability + writable_by check. Execute-time enforcement in _execute_action_field validates role_id ∈ mutation.writable_by.

**Use case:** Prevents callers from invoking mutations by disguising them as SELECT statements. Protocol-agnostic classification with per-adapter signals ensures mutations cannot bypass write-level authorization regardless of source.

**Code:** `provisa/security/mutation_authz.py`, `provisa/api/data/endpoint.py`

**Tests:** `tests/unit/test_mutation_authz.py`

### REQ-870 · Authorization {#REQ-870}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Admin-only reclassification: legitimate reclassification of a mutation to read-safe is gated by ACCESS_CONFIG capability and recorded as a governance decision. No per-operation read_only opt-out for callers; only admin acts can demote a mutation. Re-introspection of discovered mutations registers with empty writable_by; existing grants preserved by name. DONE (2026-07): two enforceable behaviors. (1) reclassify_kind (provisa/security/ mutation_authz.py) is the single reclassification gate — only a role holding ACCESS_CONFIG (ADMIN bypasses, per has_capability) may demote a mutation to read-safe; the reverse (promoting a read to a write) is rejected for everyone, and there is no caller-supplied read_only flag anywhere in the request path, so classification stays under governance control. A no-op transition is idempotent. (2) upsert_function (provisa/core/repositories/ function.py) now preserves an existing writable_by on re-introspection: a discovered mutation upserts with an empty writable_by that keeps the current admin grant (grants preserved by name), while an explicit non-empty grant still applies. Proven with unit tests for the gate and a live-Postgres test for grant preservation across re-introspection.

**Use case:** Ensures authorization policy remains under control of governance (admin), not individual callers. Prevents callers from overriding mutation classification, supports safe discovery/re-registration of mutations without accidentally granting them.

**Code:** `provisa/security/mutation_authz.py`, `provisa/core/repositories/function.py`

**Tests:** `tests/unit/test_mutation_authz.py`, `tests/integration/test_mutation_writable_by_preservation.py`

### REQ-871 · Authorization {#REQ-871}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Mutation↔table association suggestions at registration time via protocol-specific association-suggesters. Universal rule: adapter emits ranked (mutation → table) candidates; suggestions are hints only; admin-confirmed; confirmed associations register with empty writable_by (default-deny); any mutation may bind to any table (manual override always); unmatched mutations remain registerable. GraphQL: walk mutation return type's fields, unwrap NON_NULL/LIST, match leaf types against type_to_table; LIST-valued field = changed-records collection; prioritize single object > list-of-table-type; ignore scalar/stats fields. OpenAPI: align via URL path template, operationId stem, and tags (e.g., POST /users → users table); response schema is tiebreaker only, not primary signal. gRPC: response-message repeated field type + method-name entity stem (weakest signal). Fallbacks (all adapters): operation name affixes (create*/update*/delete*/upsert*) and input-type stem for operations whose response carries no queryable type. False negatives expected; no suggestion auto-binds. DONE (2026-07): the suggester engine (provisa/security/association_suggester.py) implements all three protocols and the shared fallbacks exactly as specified, returning a ranked list of TableCandidate(table, score, reason) deduplicated per table (strongest signal wins, corroborating signals add a small tiebreak). Signal strengths encode the spec order: GraphQL single-object return (1.0) > list-of-table-type (0.8); OpenAPI path resource (0.9) > operationId stem (0.7) > tag (0.6) with response schema as a 0.05 tiebreak; gRPC repeated response-field type (0.6) > method-name stem (0.4); name-affix / input-type stem fallback (0.3). Nothing auto-binds — the functions are pure and return hints; an empty list is the honest false-negative case. The GraphQL remote mapper (provisa/graphql_remote/mapper.py) invokes it during registration so every mapped mutation entry carries suggested_associations = [{table, score, reason}] computed from its unwrapped return-type leaves + input-type stem, with writable_by left empty (default-deny). Consuming these hints into an admin confirmation surface is [REQ-870](#REQ-870); projecting/binding across surfaces is [REQ-872](#REQ-872).

**Use case:** Registration of mutations across heterogeneous sources requires protocol-aware association signals. Per-adapter suggesters reduce admin friction by ranking likely alignment; unmatched mutations remain registerable as global functions or require manual binding.

**Code:** `provisa/security/association_suggester.py`, `provisa/graphql_remote/mapper.py`

**Tests:** `tests/unit/test_association_suggester.py`, `tests/unit/test_graphql_remote_mapper.py`

### REQ-872 · Authorization {#REQ-872}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Registered remote-schema mutations/functions (tracked_functions, tracked_webhooks) MUST be projected into each query surface's native function catalog and be invocable from each surface, routing through the shared executor and enforcing per-mutation writable_by authorization. Single source of truth: tracked_functions + tracked_webhooks registry. Per-surface fidelity: pgwire populates _pg_proc and supports SELECT fn(...), SQL/Explore projects information_schema.routines and supports table-valued-function invocation, Cypher/Bolt binds CALL <registeredFn>(args) YIELD ... with optional discovery. DONE (2026-07) — the SQL-surface DISCOVERY projection: the pgwire catalog builds pg_proc and information_schema.routines + information_schema.parameters from the tracked_functions registry (provisa/pgwire/catalog.py::_populate_functions, called from _build_catalog_db), so psql \\df, DBeaver and Explore now see registered functions. The registry is the single source of truth; projection is role-scoped by visible_to (unrestricted or the role is granted); table-valued functions (a return_schema) are set-returning (proretset/record); arguments become information_schema.parameters rows in ordinal order with SQL data types; the same function registered under multiple keys (bare + domain-prefixed alias) projects once. DONE (2026-07) — the SHARED EXECUTOR + Cypher INVOCATION: the tracked-function execution core is extracted from the GraphQL action path into invoke_tracked_function(name, args, state, role_id) (provisa/api/data/endpoint.py) — surface-agnostic, enforcing per-mutation writable_by by contract ([REQ-869](#REQ-869)) then running SELECT * FROM "schema"."fn"(args) through the function's source pool. The GraphQL path now routes through it, and the Cypher surface binds CALL <registeredFn>(args) YIELD col [AS alias] to it (provisa/api/rest/cypher_router.py::_detect_registered_call / _handle_registered_call): the call is intercepted before parse, args (literals / $params) are coerced positionally, writable_by is enforced in the executor (403 on deny), and YIELD projects the returned columns. DONE (2026-07) — the pgwire / SQL INVOCATION: execute_pgwire_sql intercepts a bare SELECT-of-a-registered-function before governance/routing and runs it through the same executor (provisa/pgwire/function_call.py). Both the table-valued form SELECT * FROM fn(args) and the scalar form SELECT fn(args) are recognized via sqlglot (an Anonymous func whose name is a registered tracked function), literal args are coerced positionally, and the executor's row dicts are adapted back to a pgwire QueryResult — so psql / DBeaver / Explore invoke registered functions exactly like native ones, with writable_by enforced. All surfaces (GraphQL, Cypher CALL, pgwire/ SQL SELECT) now discover AND invoke registered functions through the one shared executor.

**Use case:** UDF registry currently exposed only through GraphQL; pgwire, SQL/Explore, and Cypher surfaces cannot discover or invoke registered functions. Cross-surface projection and execution binding unifies function access across all query surfaces and enforces consistent authorization.

**Code:** `provisa/pgwire/catalog.py`, `provisa/pgwire/function_catalog.py`, `provisa/pgwire/function_call.py`, `provisa/pgwire/_pipeline.py`, `provisa/api/data/action_exec.py`, `provisa/api/rest/registered_call.py`, `provisa/api/rest/cypher_router.py`

**Tests:** `tests/unit/pgwire/test_function_projection.py`, `tests/unit/pgwire/test_function_call.py`, `tests/unit/test_tracked_function_invocation.py`

## 9. Live Data & Events

### REQ-873 · Subscriptions {#REQ-873}

**Status:** ✗ rejected · **Priority:** MUST · **Type:** behavioral

SUPERSEDED BY [REQ-874](#REQ-874). This requirement duplicated existing watermark/probe change-signal work already captured in [REQ-855](#REQ-855) (probe transport, opaque token comparison), [REQ-260](#REQ-260) (RDB watermark_column polling), and [REQ-282](#REQ-282) (Live Query Engine polling). Its only net-new capability (incremental delta fetch/append to materialized replicas instead of full re-pull) is now carried by [REQ-874](#REQ-874), which generalizes delta-fetch to any federated dataset and separates WATERMARK (existing change-signal) from DELTA (the new incremental append). This entry is retired; use [REQ-874](#REQ-874) and [REQ-855](#REQ-855) instead.

**Code:** —

**Tests:** —

## 4. Source Connectors

### REQ-874 · Materialization Store {#REQ-874}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Delta fetch for materialization_store REPLICA incremental refresh: an INCREMENTAL RELOAD strategy for datasets whose federation strategy is MATERIALIZED ([REQ-826](#REQ-826)) — data is landed/cached, so full re-pull is the cost delta avoids. Remote schemas (graphql_remote/ openapi/grpc) are the PRIMARY DRIVER but not the sole case; also RDB cached-for-latency, NoSQL, live APIs landed via MATERIALIZED. VIRTUAL and SCAN strategies EXCLUDED (always-fresh / read-in-place; nothing to delta-refresh per [REQ-826](#REQ-826)). KEY SIMPLIFICATION: PROBE == DELTA for monotonic entries. For a monotonic-cursor entry the DELTA QUERY IS THE FRESHNESS EVALUATION. Run the delta query: non-empty result ⇒ changed ⇒ apply the rows; empty ⇒ fresh ⇒ no-op. No separate watermark/probe query. No scalar_path. DEFINITION: ONE authored query (delta_query) — authored, source-native, fully-formed filter with two PLACEHOLDERS that Provisa substitutes: (a) watermark placeholder ($wm bind variable) bound to the stored CURSOR VALUE, (b) fields placeholder ({{fields}}) injected from the table's registered selection set. Provisa substitutes only; never parses the filter. CURSOR FIELD IMPLICIT: the cursor field is the field $wm filters on (single query, single field); after applying delta rows, the stored cursor advances to max(cursor-field) over returned rows (delta_query ordered by cursor field). Same-field alignment therefore trivially guaranteed; no separate field declaration required. Cursor and monotonicity are the registrant's responsibility; Provisa does no dedup or boundary-inclusivity logic. SOURCE-TYPE-SPECIFIC AUTHORING AND EXECUTION: delta_query is AUTHORED IN THE SOURCE'S NATIVE QUERY LANGUAGE, syntax and filter/cursor binding PER-SOURCE-TYPE: GraphQL `where: {field: {_gt: $wm}}` + `{{fields}}`; OpenAPI query-param `?updated_since=$wm` + selection headers; gRPC request-message filter + `$wm` + selection; SQL `WHERE field > $wm` + projection. Only $wm/{{fields}} placeholder substitution and cursor-advance logic are uniform across types. DELTA EXECUTION ROUTES THROUGH THE ADAPTER'S NATIVE CALLER, NOT THE FEDERATION ENGINE: GraphQL delta POSTs the native query to the source (predicate pushed down AT source) and lands rows into materialization_store; does NOT execute as Trino `WHERE field > x` scan over the federated relation. Native-pushdown path is the efficiency win and reason delta exists rather than relying on Trino watermark_column poll (which re-scans with no pushdown). Per source type: (1) delta template = native query syntax + filter/cursor binding (author-supplied, source-specific); (2) delta execution = routes through source type's existing native caller (GraphQL/HTTP/gRPC/SQL), bypassing federation engine. Pattern mirrors per-adapter mutation classification/suggestion ([REQ-869](#REQ-869)/871): uniform contract with per-source-type native implementation. APPLY: upsert returned rows on the replica's ALREADY-REGISTERED PRIMARY KEY (entity identity from table registration) to replace prior row state; degrades to plain insert for append-only/immutable sources. REPLICA-ONLY: VIEW/mv role is delta-ineligible—[REQ-844](#REQ-844) Iceberg overwrite-snapshot semantics (one clean snapshot per refresh) for time-travel/provenance/ bitemporal ([REQ-372](#REQ-372)/862) would be violated by upsert. Mutable relational substrate only. RELATIONSHIP TO FRESHNESS: Freshness ([REQ-855](#REQ-855)/856) is pure stale/fresh DECISION with no refresh semantics ([REQ-856](#REQ-856)). Delta is REFRESH ACTION under [REQ-826](#REQ-826)'s MATERIALIZED reload — incremental sibling of full re-pull. But for monotonic entries delta query serves as freshness evaluation, so [REQ-855](#REQ-855)'s opaque probe is NOT also run: PROBE (opaque token) and DELTA (monotonic cursor) are MUTUALLY EXCLUSIVE PER ENTRY, selected by token type. Opaque-token entries ⇒ [REQ-855](#REQ-855) probe + full re-pull, no delta. Monotonic entries ⇒ delta-as-probe, no separate probe. Validation gates: (a) delta_query contains both $wm and {{fields}} placeholders; (b) parses; (c) $wm bind-var type is orderable/monotonic-compatible; (d) injected selection valid against row type. DONE (2026-07): the UNIFORM part is implemented — provisa/federation/delta.py: delta_applies gates delta to MATERIALIZED (VIRTUAL/SCAN excluded); render_delta_fields substitutes {{fields}} textually (Provisa never parses the source-native filter) leaving $wm for native binding; has_wm_placeholder is the $wm validation gate; delta_is_fresh is PROBE==DELTA (empty result ⇒ fresh/no-op, non-empty ⇒ changed/apply); advance_cursor advances the stored cursor to max(cursor-field) over the returned rows (empty keeps it). REMAINING (kept in-progress): the per-source-type delta authoring + native execution (GraphQL/HTTP/gRPC/SQL callers, predicate pushed down at source), keyed-upsert apply on the replica, and the delta/opaque-probe mutual-exclusivity wiring with [REQ-855](#REQ-855).

**Use case:** For REPLICAS (any MATERIALIZED-strategy dataset whose federation strategy lands data), full re-pull on every refresh is expensive and unnecessary. Delta fetch via monotonic watermark keeps replicas fresh with zero lag ([REQ-855](#REQ-855) probe) without repeated full upstream fetches. Primary use: remote-schema sources (graphql_remote/openapi/gRPC) avoid full-relation re-fetch and re-materialize. Also benefits RDB sources presently using watermark_column only for subscriptions ([REQ-260](#REQ-260)) — they gain incremental replica refresh: keyed upsert instead of full re-pull, no full-CTAS + mutation-triggered staleness cycle. Strategy-agnostic scope avoids VIRTUAL (live) and SCAN (in-place read) per [REQ-826](#REQ-826). Replicas in mutable stores absorb deltas; views cannot.

**Code:** `provisa/federation/delta.py`, `provisa/subscriptions/`, `provisa/materialization/`, `provisa/materialization_store/`

**Tests:** `tests/unit/test_delta.py`, `tests/integration/test_materialization_store_lifecycle_e2e.py`

## 6. Execution, Routing, Caching & Performance

### REQ-875 · Query Routing {#REQ-875}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cheap-count route: a count(*)-shaped query over an UNMATERIALIZED source that exposes an EXACT native count ([REQ-673](#REQ-673) cardinality capability) is routed to the native count call instead of materializing the full dataset to count it. Parallels Route.CACHE ([REQ-865](#REQ-865)) — a query shape that resolves to a first-class route bypassing the expensive path. Today SELECT count(*) over a cold API source pulls the entire dataset (endpoint.py _materialize_api_to_trino_cache) to count it; with an exact native count it becomes a single call. THREE GUARDS, all required. (1) SHAPE: a bare count(*) with no other projection and no WHERE the native call cannot honor; a filtered count the source cannot filter falls back to materialize. (2) EXACTNESS: fires only when the cardinality Estimate is exact; approximate statistics stay in the planner/human-hint lane and are never served as the answer to count(*). (3) GOVERNANCE: the route MUST NOT bypass Stage-2 RLS/masking ([REQ-263](#REQ-263)) — when an RLS predicate applies to the table for the requesting persona a native total count would over-count the visible subset, so the route is disabled and falls back to materialize+count. Fail-closed, the same discipline as cache isolation ([REQ-866](#REQ-866)). DONE (2026-07): the three-guard route decision is implemented — provisa/federation/cardinality.py:: can_route_cheap_count(is_bare_count_star, estimate, rls_applies) fires only when the shape is a bare count(*), the [REQ-673](#REQ-673) estimate is EXACT with a value, and NO RLS predicate applies to the persona; any guard failing returns False (fail-closed → materialize+count). REMAINING (kept in-progress): wiring it into the endpoint count(*) path so an exact-native-count source answers with one call instead of _materialize_api_to_trino_cache.

**Use case:** Turns SELECT count(*) over a cold remote/API table from a full-dataset HTTP pull into a single cheap call, while preserving exact-answer semantics and governance, so the volume question is answerable interactively without warming the cache.

**Code:** `provisa/federation/cardinality.py`

**Tests:** `tests/unit/test_cardinality.py`

## 11. Platform, Infrastructure & Delivery

### REQ-876 · Deployment & Infrastructure {#REQ-876}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Compose host-port parameterization for stack coexistence. Every published host port in docker-compose.core.yml (and the dev/observability overlays) MUST be expressed as ${VAR:-default} so the default preserves current behavior while a second stack can bind a different host port. This lets the dev (provisa), e2e (provisa-e2e), and integration (provisa-test) compose projects run simultaneously instead of colliding on 5432/8080/ 9000/6379/8480 — the failure mode observed when a second project's minio could not bind 9000, started detached from its network, and its minio-init could not resolve minio, hanging the stack at N-1/N. THREE parts: (1) parameterize the published ports in core.yml/dev.yml/observability.yml (left side only; container ports fixed). (2) each test harness project (tests/conftest.py integration, tests/e2e/conftest.py) sets its own offset port set (e.g. PG_PORT/TRINO_PORT/REDIS_PORT/MINIO_PORT/ZAYCHIK_PORT) before compose up. (3) the in-process app under test connects to the matching ports — TRINO_PORT/ZAYCHIK_PORT are already env-driven (app.py:1119/1160); the PROVISA_CONFIG fixture's Postgres host:port must become env-interpolated (or the harness must template it) so the app reaches the offset postgres. Isolation of the compose PROJECT (‑p) already exists; this closes the host-port dimension so lifecycle-independent stacks are also bind-independent.

**Use case:** A developer or CI can run the dev stack, the e2e stack, and an integration test run at the same time without tearing one down — each provisions its own services on its own host ports, so no run blocks another and no half-attached container corrupts a shared network.

**Code:** `docker-compose.dev.yml`, `docker-compose.observability.yml`, `docker-compose.demo.yml`, `docker-compose.dev-install.yml`, `docker-compose.mvmvp.yml`

**Tests:** `tests/unit/test_infra_requirements.py`

## 5. Data Lineage & Provenance

### REQ-877 · Row-Level Delta Capture {#REQ-877}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Opt-in, per-table row-level delta capture for materialized views — the row-level companion to the column-level lineage trace ([REQ-862](#REQ-862)). Each MV refresh can record which rows changed (insert / update / delete) since the prior refresh, so downstream consumers get change-data-capture on a derived dataset and audit gets row-granular provenance. NATIVE-FIRST — hashing is the fallback, never the default: (1) an Iceberg-target MV derives deltas from a snapshot-to-snapshot changelog scan (no hashing, no retained state); (2) a watermarked source ([REQ-260](#REQ-260) watermark_column) derives the delta from the watermark-bounded slice — this IS the output-side mirror of [REQ-874](#REQ-874) delta-fetch; (3) ONLY an RDB target with no watermark falls back to full-MV row hashing. QUERY-GENERATED — MAKE THE DB THE COMPUTE: the row hash and the diff are pushed down as SQL (hash via the engine's md5/sha over the canonical row projection; the delta via a set operation — EXCEPT / anti-join — between the current and the previous hash relation, the previous snapshot persisted as a relation). Provisa issues the query and lands the result (INSERT…SELECT / CTAS in the engine); it MUST NOT SELECT rows out and hash/diff them in Python. The delta is (key, change_type, old_hash, new_hash). COLUMN EXCLUSION: the opt-in specifies which columns are EXCLUDED from the row hash (e.g. volatile audit/updated_at columns), so a change to an ignored column is not reported as a row change — the hash is computed over the included projection only. The hash path is the expensive case and is strictly OPT-IN per table (default off). A SYSTEM-WIDE config sets the row-count ceiling for the hash path — `max_row_hashing` (global default in provisa.yaml, per-table override) — above which hash-based delta capture is SKIPPED and logged (mirrors the existing MV materialization max_rows guard); never sampled (sampling would make an audit delta lossy and untrustworthy). Deltas land in a SEPARATE append-only ledger keyed by (mv_id, definition_version, refresh_trace_id) — the same store-independent principle as mv_refresh_log ([REQ-862](#REQ-862)), never stamped on the target table. NON-BLOCKING: delta capture — especially the full-MV hash path — MUST run off the refresh critical path in a background thread/task; the refresh completes and marks fresh without waiting on delta computation, and a slow or failed delta capture never delays or fails the refresh (best-effort provenance, logged on failure).

**Use case:** Gives audit and incremental-consumption workflows row-level "what changed between refresh N and N+1" without a bespoke CDC service, while bounding cost: capable targets/sources (Iceberg, watermarked) get deltas cheaply from the engine, and the expensive full-MV hash path is opt-in and ceiling-guarded so very large MVs never silently pay for it.

**Code:** `provisa/mv/delta.py`, `provisa/mv/refresh.py`, `provisa/mv/models.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_mv_delta_capture.py`

### REQ-878 · Point-In-Time Reconstruction {#REQ-878}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Point-in-time MV reconstruction from the delta ledger ([REQ-877](#REQ-877)). An append-only delta ledger is a temporal substrate: folding change events up to a refresh version N (or reversing them from the live table back to N) reconstructs the view as of N — time-travel for targets that lack it natively. TWO OPT-IN TIERS by fidelity, matching what the ledger stores: (1) HASH-DELTA ([REQ-877](#REQ-877) default) reconstructs MEMBERSHIP and change history only — which row keys existed as of N, and that/when a row changed — because a hash proves change but does not invert to old values; (2) VALUE-DELTA stores the changed row VALUES, enabling FULL CONTENT reconstruction as of N (true time-travel for an RDB target). Value-delta is a second, heavier opt-in — it reimplements what Iceberg gives natively (checkpoint snapshots + a changelog), so Iceberg targets get content time-travel for free and value-delta buys it for RDB targets at a storage cost. CHECKPOINT-BOUNDED REPLAY: do not fold all history — persist a periodic full snapshot plus deltas between; reconstruct = nearest checkpoint <= N, then apply deltas up to N (Iceberg's model). RECONSTRUCTION IS A QUERY — MAKE THE DB THE COMPUTE (same discipline as [REQ-877](#REQ-877)): as-of-N is a windowed greatest-per-key query over the ledger (last event with version <= N per key, excluding deletes), executed in the engine, never a Python replay. Non-blocking and system-config bounded like [REQ-877](#REQ-877).

**Use case:** Lets a user query "the view as of refresh N / time T" for MVs whose target is not Iceberg, without a bespoke temporal store — membership-and-audit reconstruction cheaply from hashes, or full content reconstruction when the value-delta tier is opted in, with replay cost bounded by checkpoints and the whole reconstruction expressed as a single ledger query.

**Code:** `provisa/mv/delta.py`, `provisa/mv/refresh.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_mv_delta_capture.py`

## 4. Source Connectors

### REQ-879 · Materialization Store {#REQ-879}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Cross-instance MV refresh coordination for a load-balanced fleet sharing one materialization store. Today the live refresh state is per-instance in-memory (MVRegistry._mvs; mark_refreshing flips an in-process status), so instance A's REFRESHING is invisible to B and both refresh the same MV — redundant compute plus concurrent writers racing on the same relation. FIX: promote the materialized_views catalog into the AUTHORITATIVE SHARED refresh-state and drive the refresh loop off it (not the per-instance registry). Add columns: writer, lease_until, materialized_definition_version, materialized_input_version, snapshot_id. PROTOCOL: (1) ATOMIC CLAIM — a single conditional UPDATE that BOTH dedups by version (skip when materialized_input_version == target, the [REQ-862](#REQ-862) stamp is the already-written key) AND excludes concurrent writers (skip when status=refreshing AND lease_until>now). 0 rows = skip; 1 row = this instance owns the refresh. This is the per-MV election — DECENTRALIZED, no global leader; a heartbeat renews lease_until during a long refresh, and lease expiry reclaims after a crash. (2) FENCED COMMIT — finalize (set materialized_version/status/ snapshot_id) only WHERE writer=me AND lease_until>=now; 0 rows means the lease was lost (slow/crashed-then-revived) so the result is DISCARDED — prevents a stale writer clobbering a newer refresh (fencing token). (3) SOURCE OF TRUTH — the store's committed snapshot (atomic commit, e.g. Iceberg) is authoritative for what is materialized; the catalog is coordination metadata reconciled to the store on lease expiry, so a crash mid-write leaves no partial data. NOT held pessimistic locks (SELECT FOR UPDATE over a long refresh holds a transaction open and a crash leaves a stuck lock) — leases + fencing. The catalog is the same structure that holds freshness ([REQ-855](#REQ-855)) and the version/lineage stamps ([REQ-862](#REQ-862)). CONSISTENCY KNOB: a per-MV `consistency` setting selects the tier — `shared` (default) engages this coordination catalog (one snapshot-consistent copy); `distributed` per-instance-materializes and does NOT engage it. The distributed tier is only eventually consistent under TWO preconditions: (a) a DETERMINISTIC view_sql — enforced at registration (reject now()/random()/uuid/unordered-LIMIT; provisa/mv/determinism.py); and (b) source QUIESCENCE — the source must stop changing long enough within a refresh cycle for all instances to materialize the same source state. A never-settling high-churn source never converges (each instance perpetually reflects a different snapshot), so the distributed tier is inappropriate for it — determinism is necessary but not sufficient. Quiescence is a workload property, not statically checkable, so it is an operational constraint on choosing `distributed`, whereas determinism is validated.

**Use case:** Lets an arbitrary number of control-plane/engine instances run behind a load balancer against one shared materialization store without redundant refreshes, write races, or a single-leader bottleneck — each instance races to claim due MVs, exactly one wins per MV, and crashes are handled by lease expiry and fenced commits.

**Code:** `provisa/mv/registry.py`, `provisa/mv/refresh.py`, `provisa/mv/coordination.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_mv_fleet_coordination.py`

### REQ-880 · Materialization Store {#REQ-880}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The materialized-view store (MV cache target for materialized relationships and views) must be configurable as a SQLAlchemy URI instead of pinned to a specific database type. Today it is hardcoded to the Trino catalog literal "postgresql", preventing the materialization backend from using any SQLAlchemy-supported database.

**Use case:** Supports flexible deployment scenarios where the materialization backend may be a dedicated data warehouse, distributed datastore, or cloud analytics system independent of the source connectors.

**Code:** `provisa/api/app.py`, `provisa/mv/refresh.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_federation_materialization.py`

### REQ-881 · Materialization Store {#REQ-881}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Probe-based freshness gate on the MV REFRESH path, opt-in per MV — the refresh-time application of [REQ-855](#REQ-855)'s centralized freshness gate to the materialized-view role. MOTIVATION: relationship-based (join-pattern) MVs are expensive to rebuild — a full Trino CTAS / DELETE+INSERT of the join every cycle. Today the background refresh loop gates purely on TTL (MVRegistry.get_due_for_refresh compares now - last_refresh_at >= refresh_interval), so a due MV pays the full rebuild every interval EVEN WHEN NO SOURCE CHANGED. FIX: add a per-MV freshness_mode (ttl [default, current behavior] | probe | ttl_probe). For probe / ttl_probe, before the expensive materialization the refresh computes an opaque per-MV change token from the [REQ-862](#REQ-862) per-source input signals (Iceberg snapshot id / RDB watermark MAX, already gathered by provisa/mv/input_signals.py::gather_input_signals) and compares it by equality to the token stored at last refresh: unchanged → skip the rebuild, reset the TTL, keep the materialized rows and FRESH status (zero lag at probe cost); changed → rebuild and store the new token. ttl_probe probes only after the TTL floor elapses (caps probe frequency); probe re-checks every loop ignoring TTL. SAFETY: a token is usable only when EVERY source produces one (len(signals) == len(source_tables)); if any source is token-less the entry degrades to TTL and rebuilds normally — never skip on partial signal ([REQ-855](#REQ-855) None-degrades-to-TTL, [REQ-847](#REQ-847) capability-signal-not-silent-fallback). Opt-in is per MV so cheap MVs keep plain TTL and only expensive relationship MVs pay the probe. Mutual exclusivity with delta ([REQ-874](#REQ-874)) and the shared coordination catalog ([REQ-879](#REQ-879)) follow [REQ-855](#REQ-855): the same materialized_input_version stamp is the probe token and the dedup key.

**Use case:** Lets a steward gate an expensive relationship-based MV so its costly join is rebuilt only when a source actually changed, trading a cheap per-source probe (snapshot id / watermark) for skipping a full re-materialization every interval — zero-lag freshness where rebuild cost is high, plain TTL where it is not.

**Code:** `provisa/mv/models.py`, `provisa/mv/registry.py`, `provisa/mv/refresh.py`, `provisa/mv/input_signals.py`

**Tests:** `tests/unit/test_mv_probe_freshness.py`

### REQ-882 · Materialization Store {#REQ-882}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Wire the aggregate MV rewrite path into the live query execution path. Populate AggregateMVCatalog from MVRegistry on startup, invoke aggregate rewrite in the endpoint query path, and enforce filter subset-safety before rewriting. DONE (2026-07): (1) POPULATE — MVRegistry.register/unregister now sync the process aggregate-MV catalog (provisa/mv/aggregate_catalog.get_aggregate_catalog()), so every MV registered at startup is visible to rewriting; register() is a no-op for non-aggregate MVs. (2) INVOKE — the endpoint query path calls rewrite_if_aggregate_match after the join-MV rewriter (provisa/mv/rewriter.py); only when the join rewrite did not already fire. It extracts a single-table aggregate query via sqlglot (base table, columns inside aggregate functions, top-level AND filter fragments; queries with a JOIN, multiple tables, or no aggregate are left alone) and, on a covering MV, rewrites FROM to the MV target and sets sources to the MV catalog. (3) SUBSET-SAFETY — find_aggregate_mv now enforces it (the prior code skipped the check its docstring promised): a new MVDefinition.filters records the predicates the aggregate MV was pre-computed with, and an MV is usable only when set(mv.filters) is a SUBSET of the query's filters (an unfiltered MV is always safe); the query's remaining filters (query − MV) are re-applied on the MV at rewrite so no rows the query wants are silently dropped.

**Use case:** Enables query acceleration via pre-computed aggregates stored in materialized views, allowing queries requesting SUM/AVG/etc over MV-backed tables to read pre-aggregated results instead of materializing raw rows and computing aggregates at query time.

**Code:** `provisa/mv/aggregate_catalog.py`, `provisa/mv/registry.py`, `provisa/mv/rewriter.py`, `provisa/mv/models.py`, `provisa/api/data/endpoint.py`

**Tests:** `tests/unit/test_aggregate_mv_catalog.py`

### REQ-883 · Federation Engine Abstraction {#REQ-883}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa's pgwire server must satisfy the capability set that DuckDB's `postgres` extension exercises when attaching Provisa via `ATTACH ... AS x (TYPE postgres)`, enabling DuckDB federation engines and external DuckDB clients to mount a governed Provisa instance with RLS/masking intact. Concrete capabilities enumerated: (1) libpq connection + auth handshake; (2) catalog introspection queries for schemas/tables/ columns/types/PKs over pg_namespace, pg_class, pg_attribute, pg_type, pg_index, pg_constraint with correct atttypmod and pg_type OIDs; (3) bulk read via `COPY (SELECT ...) TO STDOUT (FORMAT binary)` with real PostgreSQL binary COPY wire encoding per column type; (4) transaction-control statements (`BEGIN`/`COMMIT`/`ROLLBACK`, incl. `SET TRANSACTION READ ONLY` / `REPEATABLE READ`) accepted as no-ops — NO real snapshot isolation is provided or needed; (5) single-stream scan ONLY. Snapshot export, server-side cursors, and ctid-range parallelism are explicit NON-GOALS, validated empirically against DuckDB in the sibling pgwire-calcite project (PGW-004 / PGW-023): DuckDB's ctid parallelism splits a table by physical page range, but Provisa exposes no physical row address, so honoring a ctid range would force a `ROW_NUMBER() OVER (ORDER BY ...)` partition that scans+sorts the whole table per stream — strictly worse than one stream. The server reports relpages/reltuples ~ 0 in pg_class so DuckDB never elects parallelism and reads a single correct binary-COPY stream. Server version >= 14 (DuckDB's gate) is already met (server_version '14.0.provisa'). DONE (2026-07) capability (3) — BINARY COPY output: COPY ... TO STDOUT (FORMAT binary) now emits a real PostgreSQL binary COPY stream (provisa/pgwire/copy_handler.py): the 11-byte PGCOPY signature + flags + header-extension, per-row int16 field count with length-prefixed per-field bodies (int32 -1 for NULL), and the int16 -1 trailer. Per-column encoding is selected from the column's advertised type (Arrow schema on the Trino Flight bulk path; DuckDB type strings via QueryResult.column_types on the direct path) — bool/int2/int4/int8/float4/float8/bytea encode to their PG binary representations, everything else to UTF-8 text matching the text OID — and the CopyOutResponse overall-format flag is set to 1 (binary). Capabilities (1) libpq handshake and (2) catalog introspection are served by the existing pgwire server + catalog; (4) transaction-control statements are accepted as no-ops by the txn-tag path. DONE (2026-07) the COPY parser now accepts DuckDB's actual `(FORMAT "binary")` syntax — optional WITH and a quoted format word — which the older regex rejected (all three COPY regexes in provisa/pgwire/copy_handler.py); and a live end-to-end DuckDB 1.5.3 `ATTACH ... TYPE postgres` test drives all five capabilities against a running server (tests/integration/test_duckdb_attach_pgwire.py): ATTACH + catalog discovery, binary-COPY row/type/NULL round-trip, transaction no-ops via postgres_execute, and relpages=0 single-stream. Capability (5) is confirmed a non-goal, not remaining work. COMPLETE.

**Use case:** Lets DuckDB (as the [REQ-825](#REQ-825) federation engine or any external DuckDB/BI client) mount a governed Provisa instance via `ATTACH ... TYPE postgres` and query it with RLS/masking intact, completing the develop-on-DuckDB story where DuckDB reaches Provisa itself the same way it reaches any Postgres.

**Code:** `provisa/pgwire/server.py`, `provisa/pgwire/copy_handler.py`, `provisa/pgwire/catalog.py`, `provisa/pgwire/_pipeline.py`

**Tests:** `tests/unit/pgwire/test_copy_binary.py`, `tests/integration/test_duckdb_attach_pgwire.py`

## 3. Governance & Observability

### REQ-884 · Observability {#REQ-884}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

All Provisa internal logs—starting with the query audit log ([REQ-074](#REQ-074)) and extending to any other operational/observability logs—must be materialized and exposed as first-class registered tables within a dedicated "ops" domain in Provisa's federated catalog.

**Use case:** Makes operational telemetry queryable through the governed pipeline (pgwire/SQL/GraphQL/Cypher) subject to role + domain access control, instead of only via the Python export path (export_audit_log) or raw control-plane JDBC access which bypass governance. Enables self-observability/dogfooding where an MCP/AI planner can read observability signals through the same governed surface as business data. Closes the gap that query_audit_log is not currently exposed in the federated pgwire catalog.

**Code:** `provisa/api/_meta_views.py`, `provisa/api/startup_seed.py`, `provisa/api/app_loaders.py`, `provisa/audit/query_log.py`

**Tests:** `tests/unit/test_ops_domain.py`

## 5. Query Languages, Compilation & Operations

### REQ-885 · Extensible Functions {#REQ-885}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Generalize tracked functions ([REQ-205](#REQ-205)–208: source-resident stored procedures exposed as GraphQL mutations) to support multiple implementation kinds: source procedure (existing), script (local subprocess), http, grpc, and optionally python (in-process). All transports now functional: gRPC uses proto-less generic JSON unary bridge (_grpc_call in function_dispatch.py) via grpc.aio insecure/secure channels, with method path normalization and tls/timeout_s binding keys honored. Implementation kind becomes a selectable dimension at registration; addressing (catalog name) is decoupled from binding (transport+location, swappable). Provisa-hosted functions take relation arguments (kinds: table_ref=lazy, result_set=eager materialized Arrow, column_value=scalar row-wise) and return Arrow buffers materialized as relations, governed at the read-time boundary. Identity model (DEFINER/admin vs INVOKER/user) selected by materialize flag: true ⇒ admin/output-governed; false ⇒ user/input-governed. Composition via view_sql + materialize; no new MV subsystem—reuse MV refresh ([REQ-879](#REQ-879)), CTAS sink, governance, lineage primitives.

**Use case:** Enables diverse computation platforms (hosted scripts, external services, Python libraries) to act as governed data transformations, returning relations (not just scalar functions). Maintains config-as-truthful-model property with URI addressing and spec-driven discovery (OpenAPI/proto/ script-manifest). Preserves existing source-procedure tracked-function contract while extending to Provisa-hosted and external implementations.

**Code:** `provisa/executor/function_dispatch.py`, `provisa/api/data/action_exec.py`, `provisa/api/app.py`, `provisa/core/models.py`, `provisa/core/repositories/function.py`, `provisa/core/schema_org.py`, `provisa/api/admin/actions_router.py`, `provisa-ui/src/pages/commands/CommandFormFields.tsx`, `provisa-ui/src/pages/commands/CommandsPage.tsx`, `provisa-ui/src/pages/commands/types.ts`, `provisa-ui/src/pages/commands/api/actions.ts`

**Tests:** `tests/unit/test_extensible_functions.py`

## 3. Governance & Observability

### REQ-886 · Observability {#REQ-886}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Engine-emitted I/O tracing for UDF/transformer invocations must be mandatory and non-bypassable, recording UDF name, transport type (script/http/grpc/sql/python), identity context (admin/definer vs user/invoker), declared input refs, output cardinality/size, duration_ms, and status. A trace/correlation ID must be stamped into the UDF's minted pgwire session so audit rows join to the invocation trace for precise lineage reconstruction. UDF-emitted tracing (via W3C traceparent context propagation) is encouraged but optional, allowing UDF interiors and egress calls to nest under the engine-side invocation span.

**Use case:** Establishes a non-bypassable observability floor (relied on for governance and audit) while encouraging voluntary UDF-emitted interior/egress tracing as the ceiling for debugging and performance analysis. Enables precise lineage reconstruction by correlating UDF invocation traces with pgwire data-access audit rows.

**Code:** `provisa/otel_compat.py`, `provisa/executor/function_dispatch.py`

**Tests:** `tests/unit/test_udf_tracing.py`

## 4. Source Connectors

### REQ-887 · RDBMS Stored Procedure Auto-Discovery {#REQ-887}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Extend database source introspection to auto-discover stored procedures and routines (via information_schema.routines, pg_proc, or vendor equivalents) and auto-register them, rather than requiring hand-registration. Classify discovered procedures as read-returning (registered as parameterized relations; proc arguments become query arguments) or side-effecting (registered as mutations/tracked functions). Discovered proc results flow through Stage-2 governance identically to tables.

**Use case:** Meet legacy enterprise estates (Oracle/SQL Server shops with large proc inventories) at their current state, ensuring nothing in the estate is invisible to the catalog. Mirrors the OpenAPI discovery pattern ([REQ-316](#REQ-316)/317): introspection + auto-registration avoiding hand-registration. Value scales with proc inventory; low build cost because it reuses existing introspection and OpenAPI-style registration.

**Code:** `provisa/api/admin/introspect.py`, `provisa/discovery/catalog_cache.py`

**Tests:** `tests/unit/test_stored_proc_discovery.py`

## 6. Execution, Routing, Caching & Performance

### REQ-888 · Execution & Routing {#REQ-888}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Federation engine contract is wired into the live query path via EngineRuntime, bound mandatorily at AppState construction as state.federation_engine. Engine-specific consumer transports are capability-gated via EngineCapability enum (ROWS/ARROW/ARROW_STREAM); missing capabilities fail closed via UnsupportedCapabilityError with no fallback.

**Use case:** Mandatory EngineRuntime binding and capability-gating defines the defensible abstraction boundary for platform-specific escape hatches. Named, advertised capabilities prevent arbitrary injected connections and enforce consistent routing (DIRECT → native driver, ENGINE → federated execution) across all consumer paths (pgwire, GraphQL, Arrow Flight, COPY handler, NL executor).

**Code:** `provisa/federation/runtime.py`, `provisa/api/graphql/`, `provisa/bolt/`, `provisa/arrow_flight/`, `provisa/query_helpers.py`

**Tests:** `tests/integration/test_engine_runtime_binding.py`, `tests/unit/test_engine_capabilities.py`

## 8. Deployment & Infrastructure

### REQ-889 · Deployment Tier Architecture {#REQ-889}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The metadata/config/roles store MUST be the embedded single source of truth in every deployment tier. The embedded tier (SQLite or DuckDB) is the metadata home; container tier adds Trino + observability strictly as COMPUTE, never as metadata home. Tier changes MUST be additive and reversible: Trino can be toggled on/off without migrating the catalog.

**Use case:** Tier changes must be reversible without catalog migrations. If Postgres becomes the metadata home in a container tier, it reintroduces a container dependency and creates a migration problem on every tier toggle, breaking the design invariant that compute is pluggable while metadata stays embedded.

**Code:** `provisa/core/db.py`, `provisa/core/database.py`, `provisa/core/desktop_profile.py`

**Tests:** `tests/unit/test_arch_invariants.py`

## 4. Source Connectors

### REQ-890 · Federation Engine Abstraction {#REQ-890}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Pgwire auth must be a pluggable provider interface selected at launch (trust | local | oidc), superseding today's fixed MD5/cleartext wiring while keeping trust/cleartext as the baseline. (a) A local-accounts provider stores SCRAM-SHA-256 verifiers at rest (RFC 5802 — random salt, iterations, StoredKey/ServerKey; no cleartext stored or required on the wire, secure without TLS) with a CLI to add/remove/list users, and offers SCRAM-SHA-256 as a wire auth mechanism. (b) An external-token provider accepts an OIDC ID token (JWT) presented as the password and verifies it against the issuer's JWKS (signature, exp, aud, iss); issuer-generic (Firebase/Auth0/Google/Entra) via configured issuer URL + audience.

**Use case:** Pgwire connections to Provisa today use MD5 or cleartext auth, neither of which meet enterprise security standards. SCRAM-SHA-256 (salted, iterated, server-side verification) secures local accounts without requiring TLS. OIDC integration allows enterprises to reuse their identity provider, eliminating separate credential management for database access.

**Code:** `provisa/pgwire/server.py`

**Tests:** `tests/unit/test_pgwire_requirements.py`, `tests/unit/test_auth_providers.py`

### REQ-891 · Federation Engine Abstraction {#REQ-891}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Pgwire per-role authorization maps an authenticated account to Calcite/source schema+table visibility, enforced in BOTH discovery and execution by construction: the catalog intercept (provisa/pgwire/catalog.py) filters pg_catalog/information_schema discovery to the role's allowed objects, and query execution rejects access to objects outside the role's grant. Complements existing RLS/masking (which govern rows/columns) with object-level visibility over the wire.

**Use case:** Pgwire clients (DBeaver, DataGrip, DuckDB) discover database schemas via pg_catalog and information_schema queries. Without object-level authorization filtering, a Pgwire client will display all accessible Calcite schemas/tables to the user even if the user's role should not see them. Enforcing role visibility in both discovery and execution prevents information leakage and aligns object access with existing RLS/masking governance.

**Code:** `provisa/pgwire/catalog.py`, `provisa/pgwire/_pipeline.py`

**Tests:** `tests/unit/pgwire/test_catalog.py`

### REQ-892 · Connector Abstraction {#REQ-892}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Pgwire may present popular PostgreSQL extension surfaces backed by the federation engine, as a pluggable opt-in-per-deployment mechanism: each surface (a) advertises itself in pg_extension so clients see it "installed", (b) declares its types/OIDs in the single normalization module, (c) maps its operators/functions to an engine function or rejects them explicitly (convert-or-reject-loudly — a surface must NOT claim a capability, e.g. index acceleration, it does not implement). Concrete surfaces: pgvector-compatible (vector type + distance operators <-> L2, <=> cosine, <#> inner product, including ORDER BY emb <-> $1 similarity search, NOT advertising ivfflat/hnsw index acceleration); PostGIS-subset (geometry/geography + ST_Distance/ST_Contains/ST_DWithin/ST_Intersects/&&); JSON operators (-> ->> #> #>> mapped to JSON_VALUE/JSON_QUERY); and compatibility functions (pg_trgm similarity()/%, gen_random_uuid(), pgcrypto digest()).

**Use case:** Pgwire clients assume PostgreSQL extensions are available (pgvector for ML/embedding workloads, PostGIS for geospatial, JSON operators for document processing). Making these surfaces available over the federation engine eliminates the need to add specialized backends or custom UDFs; a single Provisa pgwire endpoint can serve diverse analytics workloads (ML, geospatial, document) via compatibility operators mapped to federation engine functions.

**Code:** `provisa/pgwire/ext_surfaces.py`, `provisa/pgwire/catalog_data.py`, `provisa/pgwire/catalog.py`, `provisa/pgwire/catalog_populate.py`, `provisa/pgwire/_pipeline.py`

**Tests:** `tests/unit/test_pgwire_ext_surfaces.py`

### REQ-893 · Federation Engine Abstraction {#REQ-893}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

PostgreSQL can act as a pluggable FederationEngine (DriverClass.PARTIAL) via postgres_fdw, enabling Postgres servers to attach foreign tables from other Postgres servers (ATTACH mechanism — no data movement, references in place). FDW types (postgres_fdw, mysql_fdw, file_fdw, mongo_fdw, tds_fdw, etc.) are provisioned at install time via CREATE EXTENSION; per-source connections are created at runtime via CREATE SERVER + CREATE USER MAPPING + IMPORT FOREIGN SCHEMA / CREATE FOREIGN TABLE. This mirrors Trino's architecture: connector types bundled at provision, catalogs dynamically created at runtime ([REQ-251](#REQ-251)).

**Use case:** Enables Tier 1 deployment: a single PostgreSQL instance serves as both the metadata home and the compute engine for PG-to-PG and cross-FDW federation, eliminating the need for Trino until heterogeneous/broad federation is required. Maintains containerless deployment spirit and completes the multi-tier deployment story. Queries pass through the semantic→IR layer (RLS/masking applied before transpile), ensuring Postgres engine passes governance-parity conformance suite ([REQ-827](#REQ-827)) by construction.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_postgres_federation_engine.py`, `tests/integration/test_postgres_federation_engine_e2e.py`, `tests/integration/test_duckdb_federation_engine_e2e.py`

## 8. Deployment & Infrastructure

### REQ-894 · Deployment Tier Architecture {#REQ-894}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

DuckDB federation engine is a DESKTOP/EDGE/DEV tier embedded federator, not a production multi-user backend. DuckDB is single-process with no server-side connection pooling and no HA; one Provisa process owns one DuckDB file and serves as the multi-user multiplexer (concurrent readers cheap via MVCC snapshots; writers/materialization serialize under Provisa coordination). The "multiple Provisas behind a load balancer with DuckDB" pattern works only for fully independent/partitioned workloads because DuckDB provides no shared state and is single-writer.

**Use case:** DuckDB is recommended for zero-install single-node desktop/edge/dev deployments with one Provisa instance per DuckDB file. Production multi-user federation at scale requires either Trino (MPP, distributes across a worker cluster) or PostgreSQL with postgres_fdw (single-node shared backend, operational hardening). The tier boundary is SCALE (single-node vs MPP), not just federation breadth.

**Code:** `provisa/federation/engine.py`

**Tests:** `tests/unit/test_arch_invariants.py`

## 4. Source Connectors

### REQ-895 · Federation Engine Abstraction {#REQ-895}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

RDBMS-as-single-node-federation-engine pattern generalizes across DB vendors via native foreign-data mechanisms (SQL/MED or equivalent). Faithful SQL/MED: PostgreSQL (most complete, postgres_fdw and ecosystem FDWs) and IBM Db2 (federation via CREATE NICKNAME). Equivalent proprietary mechanisms: Oracle (database links + Heterogeneous Gateways), SQL Server (linked servers + PolyBase external tables), MySQL/MariaDB (FEDERATED / MariaDB CONNECT engine). All are single-node ATTACH-mechanism connectors and belong to the same federation-engine tier as PG/DuckDB; Trino remains the only MPP/broad federator.

**Use case:** Pluggable "warehouse/RDBMS-native" engine family: a customer's existing warehouse (Db2/Oracle/ SQL Server/PG) could itself be the federation engine via its native foreign-data mechanism, requiring no new infrastructure. These engines are OPTIONAL/MAY and operations-driven, not scale-driven. Unifies the single-node federator family under the SQL/MED abstraction and enables ecosystem partnerships (e.g., Db2, Oracle, SQL Server connectors as plugins).

**Code:** —

**Tests:** —

## 0. Architecture & Design Principles

### REQ-896 · Federation Architecture {#REQ-896}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Provisa's defensible value is the GOVERNED-FEDERATION MODEL and consistent governance philosophy, not any compute engine or infrastructure. Federation engines (Trino, DuckDB, Postgres, Snowflake, warehouse-native) are interchangeable commodity COMPUTE substrates; governance (RLS, column masking, relationship-join enforcement) is applied in the semantic→IR layer BEFORE any engine sees the query, ensuring governance-parity across all engines regardless of architecture.

**Use case:** Engine independence allows Provisa to swap compute substrates (Trino ↔ DuckDB ↔ Postgres) with the entire governed federation model unchanged. The connector, CatalogEntry, materialization-store abstractions and single-node→MPP tier graduation remain engine-agnostic; governance invariants hold across deployment tiers and engine boundaries.

**Code:** `provisa/compiler/stage2.py`, `provisa/pgwire/_pipeline.py`

**Tests:** `tests/unit/test_arch_invariants.py`

### REQ-897 · Federation Architecture {#REQ-897}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Federation engines are declared-capability objects; capabilities are planner inputs. Each FederationEngine carries a set of DECLARED capability traits used to make routing/planning/tier decisions. Established traits: driver_class (reach: BROAD/PARTIAL/SELF_ONLY) and mpp (scale-out; orthogonal to reach). Candidate additional traits: file_native, pooled, transactional, streaming (Arrow transport modes), and connector-level pushdown (predicate/join/aggregate).

**Use case:** Consolidating engine capabilities into one coherent descriptor enables planner decisions such as: push a large federated join to an MPP engine; land API/remote sources into a pooled/transactional store; prefer a file_native engine when sources are files; gate single-node→MPP tier graduation. Capabilities become planner inputs rather than being inferred from incidental plumbing (e.g., connector count or driver class alone).

**Code:** `provisa/federation/engine.py`, `provisa/federation/connector_base.py`, `provisa/federation/plan.py`, `provisa/transpiler/router.py`

**Tests:** `tests/unit/test_engine_capability_traits.py`

## 3. Federation Engines & Data Sources

### REQ-898 · PostgreSQL Deployment {#REQ-898}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Curated, prebuilt catalog of PostgreSQL extensions (sqlite_fdw, parquet_fdw/parquet_s3_fdw,  pg_lake, pg_duckdb, pg_analytics) indexed by pinned PostgreSQL major version and target OS/platform  (manylinux, macOS, Windows), layered onto the pip-installed pgserver runtime to avoid runtime  build dependencies.

**Use case:** Prebuilt extension artifacts eliminate runtime compilation and external build toolchains,  preserving pgserver's "no root, pip install" deployment model. Enables sqlite_fdw and Iceberg-oriented  FDWs (pg_lake, pg_duckdb, pg_analytics) to work out-of-box with Provisa's Postgres federation engine  without requiring libsqlite3, libarrow/libparquet, or DuckDB compilation by the end-user.

**Code:** `provisa/pg_extensions/catalog.py`, `config/pg_extension_catalog.yaml`

**Tests:** `tests/unit/test_pg_extension_catalog.py`

## 4. Source Connectors

### REQ-899 · Federation Engine Abstraction {#REQ-899}

**Status:** ✅ complete · **Priority:** MAY · **Type:** structural

Catalog of DuckDB extensions eligible to back a federation-source connector, each providing either a relation-producing table function usable as CREATE VIEW ... SELECT * FROM fn(...), or an ATTACH catalog of tables. Only these two mechanisms qualify; scalar/index-only extensions are excluded.

**Use case:** Enables connector authors to identify which DuckDB extensions can materially contribute federated sources via scanner table-function views or ATTACH catalogs, excluding pure utility/scalar/index extensions that cannot produce relations.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_duckdb_community_connectors.py`

## 3. Federation Engines & Data Sources

### REQ-900 · PostgreSQL Deployment {#REQ-900}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Postgres federation engine's curated v1 connector set: prebuilt FDW/extension bindings for csv, sqlite, sqlserver, oracle, duckdb, and parquet/iceberg/delta_lake (via pg_duckdb), indexed by (pg_major, platform, libc) triple and layered onto the embedded pgserver runtime. Catalog resolution fails closed: no artifact for a triple means the source is unreachable on that deployment (no silent fallback).

**Use case:** A deployed Postgres federation engine connects to CSV (file_fdw), SQLite (sqlite_fdw), SQL Server (tds_fdw), Oracle (oracle_fdw), and lakehouse sources (parquet/iceberg/delta via pg_duckdb) without runtime build or external toolchain dependencies. The fixed v1 set avoids sprawl; future connectors (mysql_fdw, mongo_fdw) or write-heavy operations route via separate engines or deferred tiers.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`, `docs/arch/fdw_catalog.md`

**Tests:** `tests/integration/test_postgres_federation_engine_e2e.py`

### REQ-901 · PostgreSQL Deployment {#REQ-901}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

pg_duckdb v1.0.0 (bundles DuckDB v1.3.2) is the compatible build for embedded PostgreSQL 16.2; later pg_duckdb versions require backend symbols absent from PG16.2. Extension built from pg source against matching major version and layered into pgserver install via build script.

**Use case:** Pinned PostgreSQL 16.2 in pgserver 0.1.4 runtime requires pg_duckdb v1.0.0 to avoid symbol resolution failures at extension load. Build process (scripts/build_pg_duckdb.sh) compiles against PG16 source, and the resulting .dylib/.so is dropped alongside other core FDWs (file_fdw, postgres_fdw).

**Code:** `scripts/build_pg_duckdb.sh`, `provisa/federation/engine.py`

**Tests:** `tests/integration/test_embedded_pg_duckdb_engine_e2e.py`

### REQ-902 · Query Compilation {#REQ-902}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

pg_duckdb's transparent execution path cannot emit PostgreSQL 16 nested-JSON syntax (JSON_OBJECT with colon separators); nested-relationship GraphQL federation through pg_duckdb requires a pg_duckdb-specific JSON emission in the transpiler to collapse JSON_OBJECT into flat json_build_object.

**Use case:** Governed federation of nested GraphQL pipelines (with JOIN-enforced relationships) through pg_duckdb. Today pg_duckdb engine handles flat federated joins and governance filtering; nested-relationship queries route to other engines until JSON output is adapted.

**Code:** `provisa/transpiler/transpile.py`, `provisa/federation/pg_backend.py`

**Tests:** `tests/unit/test_transpile.py`

### REQ-903 · PostgreSQL Deployment {#REQ-903}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Postgres federation engines validate connector availability and fail explicitly when sources require unavailable connectors. Sources whose connector is unavailable resolve to UnreachableSource (explicit error, never silent fallback). Superseded by [REQ-904](#REQ-904) (probe-based discovery with runtime functional validation).

**Use case:** Operators need clear diagnostics when sources are unreachable due to missing connectors, rather than silent downgrade to alternative engines.

**Code:** `provisa/federation/engine.py`

**Tests:** `tests/unit/test_federation_engine.py`

### REQ-904 · PostgreSQL Deployment {#REQ-904}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Federation connectors are self-describing prebuilt definitions; each has an async probe() reporting FUNCTIONAL truth (FDW/extension actually loads/works), not mere presence. build_pg_engine() registers all prebuilt Postgres connectors as candidates in precedence order (postgres_fdw; pg_duckdb_csv preferred over file_fdw; pg_duckdb_parquet; pg_duckdb_json). At startup, FederationEngine.discover(fetch, disabled=frozenset(keys)) probes each candidate and sets the active set: a connector struck by the override (key in disabled) is not probed; every other candidate is probed and only kept if available; per source_type the first available candidate wins (richer preferred, stock fallback). A source with no available connector is UnreachableSource (explicit, no silent fallback). The override config is now only a strike-list, not a presence declaration. Runtime probe correctly reports pg_duckdb unavailable until preloaded (shared_preload_libraries check), where a static flag would get it wrong.

**Use case:** Runtime discovery ensures only functionally available connectors are registered, avoiding silent failures from misconfigured deployments. The strike-list override provides fallback control without requiring static presence declarations.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_federation_engine.py`, `tests/integration/test_embedded_pg_duckdb_engine_e2e.py`

### REQ-905 · Query Execution {#REQ-905}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cost-based promotion of VIRTUAL/SCAN sources to MATERIALIZED when the source connector cannot push down row-reducing operators (filter/aggregate) the query requires, and the estimated scan size exceeds the threshold (default 1,000,000 rows). Promoted sources delegate reduction to the engine store, amortized across the freshness TTL. UNKNOWN cardinality never promotes (fail-open).

**Use case:** When source connectors lack required row-reducing pushdown capability but scans are large, promoting to MATERIALIZED allows the engine store (with full pushdown) to handle reduction efficiency, amortizing the cost across the source's freshness window. Join-only gaps do not promote (non row-reducing).

**Code:** `provisa/federation/promote.py`, `provisa/federation/strategy.py`, `provisa/federation/plan.py`

**Tests:** `tests/unit/test_promote.py`

### REQ-906 · Source Connectors {#REQ-906}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Seven DuckDB community-extension federation connectors implemented: MSSQL, MongoDB, Snowflake, BigQuery, Firebird, Google Sheets (gsheets), and Airport. Each connector references an external source via ATTACH mechanism and includes a load-only startup probe that verifies the extension loads and registers its scanner/attach symbol without opening a live connection to a source.

**Use case:** Extends federation to support additional cloud and enterprise source types (MSSQL, MongoDB, Snowflake, BigQuery) plus specialty connectors (Firebird, Google Sheets, Arrow Flight via Airport), each verified at startup via extension probe to ensure engine capability is installed and reachable offline.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`, `provisa/federation/duckdb_extensions.py`, `provisa/core/models.py`, `scripts/stage_duckdb_extensions.py`

**Tests:** `tests/unit/test_duckdb_community_connectors.py`

### REQ-907 · Source Connectors {#REQ-907}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

SqliteFdwConnector and MysqlFdwConnector enable the embedded/BYO Postgres federation engine to attach SQLite files and remote MySQL servers via sqlite_fdw and mysql_fdw, respectively. Each connector declares runtime_deps (tuple of shared libs tagged system/bundled/operator) documenting the packaging surface; probes verify functional availability at startup.

**Use case:** Federation of SQLite file-based data stores (in-place via sqlite_fdw; libsqlite3 system-provided) and remote MySQL databases (via mysql_fdw; libmysqlclient/mariadb-connector-c bundled) extends the Postgres v1 connector set beyond CSV/Parquet/Iceberg, enabling embedded deployments to query across multiple relational sources without data movement.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`

**Tests:** `tests/integration/test_embedded_pg_sqlite_fdw_e2e.py`, `tests/unit/test_fdw_connectors.py`

### REQ-908 · Source Connectors {#REQ-908}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

PgDuckdbIcebergConnector reads Apache Iceberg tables in-place via pg_duckdb's iceberg_scan (DuckDB iceberg extension) inside the stock embedded Postgres. Extension is STATIC-linked into libduckdb (aws-sdk-cpp, avro-c, roaring) with no extra runtime dylib; build requires USE_MERGED_VCPKG_MANIFEST=1 and iceberg pinned to branch v1.3-ossivalis.

**Use case:** Federation of Apache Iceberg lakehouse tables stored on S3/GCS/Azure without data movement. Iceberg's time-travel and ACID guarantees are preserved through the query; governance predicates filter rows at the scan level before federation.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`, `scripts/build_pg_duckdb.sh`

**Tests:** `tests/integration/test_embedded_pg_duckdb_iceberg_e2e.py`, `tests/unit/test_pg_duckdb_iceberg_connector.py`

## 4. Source Connectors

### REQ-909 · Direct-Route Dialect Expansion {#REQ-909}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Generic SQLAlchemy fallback driver provides direct-route write capability for any source type with a SQLAlchemy dialect, without requiring a bespoke async driver. Writable sources are the intersection of {has_driver} AND {sqlglot_write_dialect}, where has_driver now includes the SQLAlchemy fallback when no native driver exists.

**Use case:** Broadens the writable RDBMS set on the direct route without per-source driver implementations. Any SQLAlchemy dialect (whose DBAPI is installed) + sqlglot write dialect support enables writes to that source type without a native async driver.

**Code:** `provisa/executor/writable.py`, `provisa/executor/drivers/sqlalchemy_driver.py`, `provisa/executor/drivers/registry.py`, `provisa/federation/connector.py`, `provisa/core/models.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_writable.py`, `tests/unit/test_drivers.py`

### REQ-910 · Multi-Route Write Dispatch {#REQ-910}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Preference-ordered write-path resolution routes mutations across three tiers: NATIVE (bespoke async driver, direct), SQLALCHEMY (generic fallback driver, direct), and ENGINE (federation engine writes upstream via a write-capable connector). NATIVE and SQLALCHEMY require both a direct driver and SQLGlot write dialect; ENGINE requires only the connector's Capability.write. ENGINE is the only route that reaches sources with no direct driver and no SQLGlot dialect (e.g., NoSQL sources exposed as writable relations by their connector).

**Use case:** Enables mutation routing to any writable source regardless of per-source driver availability or SQLGlot dialect support. Maximizes write coverage by cascading through direct routes (native first, then SQLAlchemy) before falling back to engine-routed mutations through connectors.

**Code:** `provisa/executor/writable.py`, `provisa/executor/drivers/registry.py`

**Tests:** `tests/unit/test_writable.py`

### REQ-911 · ClickHouse Federation Engine {#REQ-911}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

ClickHouse is a selectable federation engine (PROVISA_ENGINE=clickhouse) alongside trino/pg/duckdb/sqlalchemy. It is a single-node-reach PARTIAL federator, MPP, native_store="clickhouse" that ATTACHes external sources via ClickHouse's native integration engines (Mechanism.ATTACH) rather than an FDW.

**Use case:** Federate PostgreSQL, MySQL, CSV, Parquet, and MongoDB sources as ClickHouse tables without data movement. PostgreSQL/MySQL sources mount as DATABASE engines exposing all remote tables; CSV/Parquet mount as TABLE engines (S3, URL, or File based on path scheme) with columns inferred by ClickHouse; MongoDB mounts as MongoDB engine with columns supplied from registry metadata.

**Code:** `provisa/federation/engine.py`, `provisa/federation/connector.py`, `provisa/federation/clickhouse_backend.py`, `provisa/federation/clickhouse_runtime.py`, `provisa/transpiler/transpile.py`

**Tests:** `tests/integration/test_clickhouse_runtime_e2e.py`, `tests/unit/test_clickhouse_connectors.py`

### REQ-912 · ClickHouse Federation Engine {#REQ-912}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The ClickHouse federation runtime supports three interchangeable execution backends selected by URL scheme at initialization: clickhouse:// (HTTP via clickhouse-connect, port 8123), clickhouse+native:// (native TCP via clickhouse-driver, port 9000), or chdb:// (embedded in-process engine, optional filesystem persistence). All backends expose identical SQL and integration engine surfaces.

**Use case:** Operators can choose between a remote ClickHouse server (HTTP or native protocol), in-process embedded ClickHouse, or persistent embedded store depending on deployment topology and latency requirements.

**Code:** `provisa/federation/runtime.py`

**Tests:** `tests/unit/test_clickhouse_connectors.py`, `tests/integration/test_clickhouse_runtime_e2e.py`

## 0. Architecture & Design Principles

### REQ-913 · AST-Based Transforms {#REQ-913}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Every query transform — governance (RLS, masking, visibility, row-cap), correlated- subquery rewriting, and dialect lowering — MUST operate on a parsed SQL/relational AST, never on SQL text via regex or string substitution. Each surface language (GraphQL, Cypher, SQL) parses to an AST that is carried through governance and planning and serialized to a dialect string only at final emission; no stage may re-derive query structure (SELECT scope, WHERE injection point, alias binding, LIMIT) from text.

**Use case:** String/regex transforms silently mis-apply governance: a scalar subquery in the projection moves the masking boundary (PII leak), a non-t0 alias convention skips RLS qualification (scope-ambiguous predicate), and a parameterized LIMIT bypasses the row-cap ceiling. AST transforms make "governance applied" a structural property of the tree rather than a textual coincidence each frontend must keep satisfying.

**Code:** `provisa/compiler/stage2.py`, `provisa/compiler/rls.py`, `provisa/compiler/mask_inject.py`, `provisa/transpiler/transpile.py`, `provisa/cypher/translator.py`

**Tests:** `tests/unit/test_ast_transform_invariants.py`, `tests/unit/test_ast_transforms.py`, `tests/integration/test_ast_transform_e2e.py`

## 3. Governance & Observability

### REQ-914 · Pipeline Stage Tracing {#REQ-914}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Each query-pipeline boundary (govern.in, govern.rls, govern.mask, govern.out, and transpile.<dialect>) emits an OpenTelemetry span event carrying the canonical SQL, so the transform sequence is inspectable per request and the pre/post-governance pair serves as a compliance receipt. SQL capture is OFF by default; PROVISA_TRACE_SQL=redacted blanks every literal to a placeholder before attaching, and =full attaches raw SQL for dev only. Telemetry never fails the request: a redaction parse error is surfaced as a span attribute. PROVISA_TRACE_AST=1 attaches the SQLGlot AST repr, redacted the same way (literals blanked), so it is safe to enable independently of the SQL mode.

**Use case:** Debugging special-case query transforms previously meant reading regex and guessing where text was mangled; per-stage canonical SQL turns that into a diff. The redaction default prevents literal values (RLS predicate values, PII) from leaking into traces in a governance product.

**Code:** `provisa/observability/stage_trace.py`, `provisa/compiler/stage2.py`, `provisa/compiler/rls.py`, `provisa/compiler/mask_inject.py`, `provisa/transpiler/transpile.py`

**Tests:** `tests/unit/test_stage_trace.py`

## 1. Configuration & Deployment

### REQ-920 · Config Generation {#REQ-920}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The DDN-to-Provisa connector schema converter (ddn/mapper.py, _map_connectors) emits editable placeholder connection strings when an HML subgraph connector lacks a source-override. Placeholder format: localhost/5432/default/postgres/${env:DB_PASSWORD}. Intentional fallback: DDN is a config-generation tool whose output is reviewed before deployment, not a runtime system.

**Use case:** Configuration generation tools output templates for human review and edit; the placeholder guides the operator to fill in the correct connection URI with environment variable substitution.

**Code:** `provisa/ddn/mapper.py`

**Tests:** `tests/unit/test_ddn.py`

## 6. Materialized Views & Caching

### REQ-921 · Promotion Coercion {#REQ-921}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

An unmapped promotion target_type in API JSONB promotions (provisa/api_source/promotions.py, generate_promotion_ddl) yields an empty cast (no explicit JSONB→target coercion). Intentional: only types requiring explicit coercion are in _PG_CAST_MAP; unmapped types are already representable in JSONB without casting.

**Use case:** Promotion coercion maps only the types that need explicit PostgreSQL casts; types not requiring coercion do not need a cast and the absence signals correct representation.

**Code:** `provisa/api_source/promotions.py`

**Tests:** `tests/unit/test_api_promotions.py`

## 7. Connectors & Source Adapters

### REQ-922 · Change Data Capture {#REQ-922}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Debezium CDC provider (provisa/subscriptions/debezium_provider.py) uses a stable sentinel (datetime.min) for missing or unparseable ts_ms, mirroring [REQ-343](#REQ-343) (RSS provider sentinel). Intentional fallback: ts_ms is optional in Debezium envelopes (snapshot/tombstone records may omit it); using now() would advance the watermark and drop later real events.

**Use case:** Change-data-capture watermarks must monotonically advance only with real event timestamps, not clock time; sentinel values preserve event ordering when optional fields are absent.

**Code:** `provisa/subscriptions/debezium_provider.py`

**Tests:** `tests/unit/test_debezium_provider.py`, `tests/integration/test_debezium_cdc.py`

## 2. Query Execution & Planning

### REQ-923 · Trino Introspection {#REQ-923}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Trino column introspection (provisa/compiler/introspect.py, introspect_column_types / _fetch_with_startup_retry) retries with backoff when the coordinator reports SERVER_STARTING_UP, up to the ready timeout. Every other Trino error propagates. Intentional resilience: transient coordinator boot can last minutes; retry captures this operational reality without masking genuine errors.

**Use case:** Coordinator restarts are transient operational events; introspection must tolerate boot time without becoming a silent mask for genuine connection or permission errors.

**Code:** `provisa/compiler/introspect.py`

**Tests:** `tests/unit/test_compiler_introspect.py`, `tests/integration/test_introspect.py`

## 3. Source Registration & Data Modeling

### REQ-924 · Source Watermarking {#REQ-924}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

A source table has at most one watermark column, selected from existing columns only. Sources have no derived/synthetic columns; watermark value is a column name or none.

**Use case:** The watermark column enables incremental refresh and subscriptions; limiting to one existing column simplifies source configuration and avoids synthetic stamping.

**Code:** `provisa/core/models.py`, `provisa/federation/residency.py`

**Tests:** `tests/unit/test_residency.py`, `tests/unit/test_watermark_validation.py`

### REQ-925 · Source Watermarking {#REQ-925}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

If a watermark column is set, it must be monotonic non-decreasing (timestamp or incrementing integer). The column type is validated at selection time; non-monotonic columns are rejected.

**Use case:** Incremental refresh and subscription reads rely on WHERE wm > cursor predicates; non-monotonic values would miss or duplicate rows, violating correctness.

**Code:** `provisa/core/config_loader.py`, `provisa/core/ir_types.py`

**Tests:** `tests/unit/test_watermark_validation.py`

## 9. Live Data & Events

### REQ-926 · Change Subscriptions {#REQ-926}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The watermark column gates refresh mode and subscribability: watermark set → APPEND refresh (incremental, WHERE wm > cursor) and subscriptions allowed; watermark unset → REPLACE refresh (full DELETE+INSERT) and subscriptions forbidden.

**Use case:** Subscribability and incremental refresh are coupled; only watermarked sources can reliably serve incremental reads and change notifications.

**Code:** `provisa/core/change_signal.py`, `provisa/federation/residency.py`, `provisa/events/injector.py`

**Tests:** `tests/unit/test_change_signal.py`, `tests/unit/test_residency.py`, `tests/unit/test_injector.py`

### REQ-927 · Change Subscriptions {#REQ-927}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Subscriptions require a watermark column. No watermark = no subscriptions and no incremental refresh.

**Use case:** Subscriptions deliver rows WHERE wm > cursor; without a watermark, there is no monotonic reference point for incremental delivery.

**Code:** `provisa/core/change_signal.py`, `provisa/core/config_loader.py`, `provisa/live/reconcile.py`

**Tests:** `tests/unit/test_change_signal.py`, `tests/unit/test_injector.py`

### REQ-928 · Change Subscriptions {#REQ-928}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Delivered subscription grain is INSERTS and UPDATES only; HARD DELETES are not supported. An append + (wm > cursor) model cannot observe removed rows; a watermarked landed source is a watermark-ordered append log, not deduplicated current-state.

**Use case:** Subscribers receive a changelog stream ordered by watermark; deletes leave no advancing watermark and cannot be signaled within the append-log model.

**Code:** `provisa/core/change_signal.py`, `provisa/events/injector.py`

**Tests:** `tests/unit/test_injector.py`, `tests/unit/test_change_signal.py`

### REQ-929 · Change Signals {#REQ-929}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The notification/inbound change signal is independent from the watermark column. Allowed values: ttl | probe | probe+ttl | native | debezium | kafka. When a watermark exists, probe defaults to MAX(watermark); otherwise probe is a source-native query or TTL is used. Push variants (native/debezium/kafka) rely on the operator to maintain feed health.

**Use case:** Change signals are orthogonal to watermarking; different sources may use different notification mechanisms (polling vs. push, automatic vs. operator-managed).

**Code:** `provisa/core/change_signal.py`

**Tests:** `tests/unit/test_change_signal.py`

## 6. Materialized Views & Caching

### REQ-930 · Materialization {#REQ-930}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Materialization is derived from federation-engine reachability, not a user knob. A source the engine cannot reach live is materialized/landed (mandatory). A reachable source is queried live, with optional perf-caching (TTL-based). First land of an unreachable source is unconditional; the change signal governs subsequent refresh.

**Use case:** Materialization ensures correctness when live access is unavailable; it is driven by infrastructure capability, not user preference, eliminating misconfiguration.

**Code:** `provisa/federation/strategy.py`, `provisa/federation/plan.py`

**Tests:** `tests/unit/test_federation_strategy.py`

## 9. Live Data & Events

### REQ-931 · Change Subscriptions {#REQ-931}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Derived/synthetic watermarks are manufactured in the SQL/view layer, never at the source. A view can project a computed watermark column (e.g. GREATEST(created_at, updated_at), MAX of inputs' watermarks, ROW_NUMBER) and downstream subscribes to that derived column. Source-level synthetic-watermark stamping is intentionally unnecessary.

**Use case:** Views provide full SQL tooling for composing derived watermarks; sourcing synthetic columns at ingest would duplicate this capability and complicate source registration.

**Code:** `provisa/federation/residency.py`

**Tests:** `tests/unit/test_residency.py`

### REQ-932 · Change Detection Strategy {#REQ-932}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The change_signal→landing-shape mapping (push→CDC upsert/tombstone by PK, poll+watermark→APPEND, else REPLACE) is implemented as store_writer.select_landing_shape + the DB-agnostic Core ops in materialize_exec (land_replace/land_append/apply_cdc via the dialect-agnostic Connection.upsert), NOT engine CTAS in mv/refresh. The named integration tests (test_landing_replace/append_delete/publish_downstream) do not exist; coverage lives in unit tests (test_change_signal, test_materialize_landing, test_subscribe_publish). DONE (2026-07): provisa/federation/store_writer.py, provisa/federation/materialize_exec.py, provisa/core/change_signal.py.

**Use case:** Eliminates redundant encoding of change-detection intent across three independent fields (`change_signal`, `freshness_mode`, `live.strategy`). Centralizes the authoritative source and clarifies orthogonal axes (detection method, append/replace mode, materialization).

**Code:** `provisa/core/change_signal.py`, `api/app.py`, `api/admin/schema.py`, `api/data/subscribe.py`, `live/reconcile.py`, `core/config_loader.py`, `core/models.py`, `mv/refresh.py`

**Tests:** `tests/unit/test_change_signal.py`, `tests/unit/test_materialize_landing.py`, `tests/unit/test_subscribe_publish.py`

### REQ-933 · Execution & Routing {#REQ-933}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

SQLAlchemy generic types are the ONE canonical engine-independent IR type registry (provisa/core/ir_types.py). Sources map native→IR (to_ir), fed engines map IR→physical dialect, the write face maps IR→SQLAlchemy — so the fed engine is never the type authority. to_ir normalizes native/dialect spellings to one canonical IR name (varchar→text, int4→integer, jsonb→text), strips length/precision, and RAISES on an unknown type (no silent varchar default). data_type is an IR name: authoritative for a LANDED table (drives store DDL), informational for a LIVE source (engine's live resolution wins). Types resolved at registration, never lazily backfilled. DONE (2026-07): provisa/core/ir_types.py, provisa/federation/materialize_exec.py; tests: tests/unit/test_ir_types.py.

**Code:** `provisa/core/ir_types.py`, `provisa/federation/materialize_exec.py`

**Tests:** `tests/unit/test_ir_types.py`

## 4. Source Connectors

### REQ-934 · Materialization Store {#REQ-934}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The materialization store is RECONCILED, not blindly created: an existing landing table whose columns match config is KEPT (survives restart, data intact); a config/table/engine drift that changes the column set RECREATES it (drop+create, re-landed on next refresh); absent → create. Convergent + idempotent. Two triggers, one primitive: boot (after config load) and runtime UI (re)registration. Eager-creates landing tables (DDL) + attaches the engine's read view WITHOUT landing data — DDL/DML split so the catalog is complete at startup; landed sources only, live sources attach live. DONE (2026-07): provisa/federation/store_writer.py (reconcile_table), provisa/federation/native_backend.py (reconcile_landed_tables), provisa/federation/duckdb_runtime.py (attach_landed_source), provisa/federation/runtime.py, provisa/api/app.py, provisa/api/admin/schema.py; tests: tests/unit/test_reconcile_landed.py, tests/unit/test_store_writer.py.

**Code:** `provisa/federation/store_writer.py`, `provisa/federation/native_backend.py`, `provisa/federation/duckdb_runtime.py`, `provisa/federation/runtime.py`, `provisa/api/app.py`, `provisa/api/admin/schema.py`

**Tests:** `tests/unit/test_reconcile_landed.py`, `tests/unit/test_store_writer.py`

### REQ-935 · Materialization Store {#REQ-935}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

At land time, incoming rows are mapped to the target columns by NAME (matched land, unmatched target NULL, extra keys dropped — best-effort); the land is REFUSED when the fraction of target columns present is <= a match floor (default 0.0 fails only at 0% overlap), turned by the caller into an error event — never silently land a mangled table. Distinct from config/design drift (which recreates, authoritative). DONE (2026-07): provisa/federation/store_writer.py (check_source_drift); tests: tests/unit/test_store_writer.py.

**Code:** `provisa/federation/store_writer.py`

**Tests:** `tests/unit/test_store_writer.py`

## 9. Live Data & Events

### REQ-936 · Live Data & Events {#REQ-936}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

One event loop over one queue; base sources and MVs are homogeneous nodes. FIVE event kinds in three categories: APPLY {delta, append, replace} → landing shapes; ADVISE {warn} → alert/at-risk/DAG-tint, no enforcement, data still flows; HALT {error} → freeze (landed) / detach (live). Delta is UPSERT by PK (insert/update + tombstone delete); append is INSERT (rows + cursor WHERE watermark > last). Append carries the cursor/rows; a delta payload above the jsonb size ceiling degrades to replace. The emitted kind is set by the node's incremental-maintainability (may differ from consumed; non-maintainable → replace).

**Use case:** Unified event model provides a single, homogeneous abstraction for all data-movement primitives (sources, MVs, subscriptions) with clear semantics for change operations (delta, append, replace), advisory warnings, and error handling.

**Code:** —

**Tests:** `tests/unit/test_event_queue.py`, `tests/unit/test_injector.py`

## 6. Execution, Routing, Caching & Performance

### REQ-937 · Execution & Routing {#REQ-937}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Every node = generate (fetch for a source | SQL for an MV — the only difference) → post-process (enrich and/or validate/gate) → emit one of the 5 events. Post-processing (enrich and/or validate) applies to sources AND MVs alike. Landing is OPTIONAL: a change event can trigger validation/reaction on LIVE data without materializing.

**Use case:** Single-threaded node model (generate→post-process→emit) provides a clear, composable architecture for incremental data flows while keeping all processing layers visible and auditable.

**Code:** —

**Tests:** `tests/unit/test_processor.py`, `tests/unit/test_handlers.py`

### REQ-938 · Execution & Routing {#REQ-938}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Every MV is SQL, authored to REFERENCE its inputs (SQLGlot extracts the dependency edges — reuse the exp.Table extraction already in provisa/api/admin/_table_ops.py) and DECLARE its full output schema; columns the SQL doesn't derive are constant placeholders completed by post-processing. The input reference (count/key/max(watermark)) doubles as the node's change/freshness token. Guarantees lineage is always derivable — no node ever has opaque inputs. DAG is acyclic (cycle detection at registration); re-parsed on registration. One event fans out to many dependents → the exclusive unit of work is (event × dependent-table).

**Use case:** SQL lineage extraction from MV definitions ensures the DAG is always derivable and transparent; combining with change tokens enables correct cascading materialization and versioning.

**Code:** —

**Tests:** `tests/unit/test_processor.py`

### REQ-939 · Execution & Routing {#REQ-939}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

SQL is always the lineage/schema authority; processors are SUBORDINATE, attached to COLUMNS (post-process completing declared columns, or a UDF sugar; external UDFs deferred to a batched post-process). Processor contract: pure function of declared inputs → declared output model, no side effects, deterministic (or a declared best-effort). Tiers by robustness: in-engine SQL → in-process UDF → external env. MV compute resources are OPERATOR-SUPPLIED (the fed engine, sized to need — MPP for big views; Provisa builds no distributed compute and orchestrates the supplied engine). DQ metrics / ML features / enrichment are external processors — bring your compute, Provisa governs the OUTPUT as a first-class governed data product.

**Use case:** Processors provide an imperative escape hatch when SQL is insufficient, while keeping lineage and schema declaration authoritative; tiers enable plugging any compute backend.

**Code:** —

**Tests:** `tests/unit/test_lineage.py`

### REQ-940 · Execution & Routing {#REQ-940}

**Status:** ✅ complete · **Priority:** MAY · **Type:** infrastructure

The external processor tier is a toolkit of transport adapters — shell (stdin/stdout), HTTP (streamed request/response), gRPC (bidi-stream) — all conforming to ONE streaming input→output contract: a stream of schema-declared rows in, a stream of schema-declared rows out; pure; no side effects; schema-validated both ends. NDJSON default framing (Arrow-stream for throughput is a future framing under the same contract). The toolkit owns framing, streaming, and fail-loud schema validation; the author implements only the transform. Extends the in-process [REQ-957](#REQ-957) preprocess hook to external processors reachable over any transport. A declarative spec (transport + address) selects the adapter via provisa.processors.build_adapter.

**Use case:** Pluggable transport adapters allow external processors to be invoked over any mechanism (shell, HTTP, gRPC) while maintaining a single streaming contract and declarative schema validation.

**Code:** `provisa/processors/contract.py`, `provisa/processors/framing.py`, `provisa/processors/shell.py`, `provisa/processors/http.py`, `provisa/processors/grpc.py`, `provisa/processors/factory.py`

**Tests:** `tests/unit/test_processors.py`

## 9. Live Data & Events

### REQ-941 · Live Data & Events {#REQ-941}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A gate is a validate-processor with halt authority; outcome ∈ {pass → data event; warn → advisory warn event (no enforcement, data flows); fail → error event (halt)} — declared per node. Hard-fail enforcement: landed → freeze last-good; live → detach + alert (declared-severe and rare; eventual + catalog-granular, no per-row/in-flight block over federation). error is emitted by ANY hard node failure (source-land failure, hard gate, drift-below-floor, unreachable source, unknown type), propagates blocked-upstream transitively, downstream freezes last-good, vintage stays honest ("as of last good land, known-blocked since T"). Recovery = re-registration through the reconcile controller (no dedicated re-attach event, no default background loop); auto-recovery is an opt-in health-probe policy that owns cadence + hysteresis.

**Use case:** Gates provide declarative data validation and enforcement, with clear semantics for warnings (advisory) and errors (halt), ensuring data correctness while maintaining transparent recovery paths and vintage honesty.

**Code:** —

**Tests:** `tests/unit/test_processor.py`, `tests/unit/test_supervisor.py`, `tests/unit/test_boot.py`, `tests/unit/test_app_wiring.py`

### REQ-942 · Live Data & Events {#REQ-942}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

A shared event substrate is the third fleet-shared pillar (with control-plane DB + shared cache), PLUGGABLE: in-process (single-node/embedded) → durable log (fleet). Default backend is a control-plane EVENT TABLE as a transactional outbox (the event is enqueued in the SAME transaction as the state change → atomic; superior to a broker for correctness); SELECT FOR UPDATE SKIP LOCKED claim + LISTEN/NOTIFY wake + a reaper; graduate to Kafka only for raw-event volume. Two tables: events (posted once) + event_status (one row per event × dependent-table = the fanout work item). Claim by TARGET TABLE (not per-event): cross-table parallel, same-table serialized, coalesce at drain; heartbeat lease, stale → reclaim, idempotent apply = effectively-once. High-volume raw CDC is digested at the injector edge into coalesced node-level events; the table holds only digested events. Three roles: INJECTORS (external → post events, one per modality; delta/append payloads above jsonb ceiling degrade to replace), TABLE PROCESSORS (claim → generate/land → post-process/gate → complete → re-post downstream events; homogeneous/stateless/horizontally-scaled), REPEATERS (cursor-based fanout read for SSE-to-local-clients; forward, never claim). Dispatch rule (the N-boxes test): CLAIM exactly-once for anything whose duplication across the fleet is wrong — landing, side-effecting reactions (warn/error handlers: email/ticket/page), Kafka-sink produce; FANOUT (cursor read-all) ONLY for per-box-local delivery (SSE). In-loop = pure/convergent (constrained); out-of-loop reactions = arbitrary side-effecting automation (free), including error/warn-event automation — but to change the governed DATA you must re-enter the governed loop (inject/re-register/re-ingest), never a side-channel store write.

**Use case:** A shared event substrate provides the scalability and correctness foundation for fleet-wide data propagation, with clear role separation (injector, processor, repeater) and dispatch semantics (claim for mutations, fanout for broadcast).

**Code:** —

**Tests:** `tests/unit/test_event_queue.py`, `tests/unit/test_injector.py`

### REQ-943 · Live Data & Events {#REQ-943}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The event-loop source-land path fetches current rows from a materialized source table through the federation engine's SQL terminal. SourceRowLoader.load(source, table) issues a qualified/quoted SELECT * via engine.execute_engine and returns row dicts. Row-oriented API/push source types (openapi, ingest, websocket, rss, grpc_remote, prometheus, google_sheets) have no engine-scannable table; load() raises UnsupportedSourceFetch (explicit, never silent) for types without a wired adapter (openapi is wired via [REQ-945](#REQ-945); others require per-adapter fetch implementation). The boot wiring lands nothing for that node (logged once) until a per-adapter fetch is wired.

**Use case:** The event-loop source-land path requires a unified interface to fetch current snapshots of materialized source tables through the federated SQL engine, with explicit failure signaling for source types that lack engine-scannable tables and deferred adapter-specific fetch logic.

**Code:** `provisa/events/source_loader.py`, `provisa/events/app_wiring.py`

**Tests:** `tests/unit/test_source_loader.py`

## 10. UI & Admin Surfaces

### REQ-944 · Schema Discovery {#REQ-944}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

The schema-discovery admin UI assigns canonical IR data-types to source columns. The backend exposes the IR vocabulary via GET /admin/schema-discovery/ir-types (returns sorted IR_TYPES from provisa/core/ir_types.py). The SchemaDiscovery UI component fetches that vocabulary to populate per-column type dropdowns instead of hardcoded engine-specific types. New manually-added columns default to IR "text". Discovered native columns are normalized to IR names via toIrType (mirroring backend native→IR aliases) so introspected columns land on valid dropdown values. Steward-assigned IR data-types persist through registration via GraphQL ColumnInput.data_type field, which flows into Column model and is saved via table_repo.save. Type assignments are engine-independent; the canonical IR vocabulary maps every IR name to a GraphQL scalar, and the landing write path maps IR → the store's physical type.

**Use case:** Stewards can standardize column types to a canonical vocabulary independent of source engine during schema discovery, ensuring consistent data modeling across heterogeneous sources.

**Code:** `provisa/api/admin/discovery_schema.py`, `provisa/api/admin/types.py`, `provisa/api/admin/_table_ops.py`, `provisa/compiler/type_map.py`, `provisa/core/ir_types.py`, `provisa-ui/src/components/SchemaDiscovery.tsx`, `provisa-ui/src/api/admin.ts`, `provisa-ui/src/irTypes.ts`

**Tests:** `tests/unit/test_ir_types_endpoint.py`, `tests/unit/test_ir_type_persistence.py`, `provisa-ui/src/__tests__/schemaDiscovery-irtype.test.ts`

## 9. Live Data & Events

### REQ-945 · Live Data & Events {#REQ-945}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The openapi adapter fetch is wired into SourceRowLoader via make_openapi_loader, which accepts injectable per-type adapter_loaders. make_openapi_loader resolves a table's registered ApiEndpoint and its ApiSource (base_url + auth) from live app state (state.api_endpoints / state.api_sources), calls the operation with its default_params via api_source.caller.call_api, and flattens the response pages into row dicts via api_source.flattener.flatten_response — the same call_api → flatten chain the API-cache path uses. app_wiring builds it from state and registers it under "openapi" source type, so openapi source nodes now land real data. A table with no registered endpoint raises UnsupportedSourceFetch (never a silent empty snapshot). Remaining API/push types (ingest, websocket, rss, grpc_remote, prometheus, google_sheets) still raise UnsupportedSourceFetch until each is wired.

**Use case:** Openapi sources need a real-time fetch path that reuses the same authentication and response flattening logic as the API-cache write path, without requiring an engine-scannable table.

**Code:** `provisa/events/source_loader.py`, `provisa/events/app_wiring.py`

**Tests:** `tests/unit/test_source_loader.py`

## 4. Source Connectors

### REQ-946 · Connector Abstraction {#REQ-946}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Trino Connector class is the single source of truth for a source type's Trino catalog and its reach (realizes [REQ-842](#REQ-842)). Each catalogable source type has a Trino Connector (provisa/federation/connector.py) declaring trino_connector (the Trino connector.name for CREATE CATALOG ... USING) and details() (the catalog .properties). build_trino_engine registers the full set via build_trino_connectors(); provisa/core/catalog.py create_catalog/_build_catalog_properties derive name+props from the TRINO_CONNECTORS registry — a source type with no Trino connector gets NO catalog. sqlite/openapi are Mechanism.LAND (data landed into local PG cache, then Trino reads it); all other catalogable types are ATTACH. Consequence: federate() classifies Trino's connector types as VIRTUAL (federated live) rather than MATERIALIZED — engine.connectors is the engine's true reach.

**Use case:** Centralizing connector registration through a single registry eliminates parallel type→name maps and hardcoded property branches, ensuring consistent catalog generation and reach classification across all source types.

**Code:** `provisa/federation/connector.py`, `provisa/core/catalog.py`

**Tests:** `tests/unit/test_federation_strategy.py`, `tests/unit/test_federation_engine.py`, `tests/unit/test_federation_materialization.py`, `tests/unit/test_sharepoint_connector.py`, `tests/unit/test_splunk_connector.py`, `tests/unit/test_trino_catalog_files.py`

### REQ-947 · Connector Abstraction {#REQ-947}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

engine.connectors (the unified connector registry per federation engine) must include a connector for EVERY source type the engine can offer: (a) federated via engine's own connectors, (b) direct-route via Provisa's native async drivers (executor/drivers/registry.py), and (c) adapter-materializable (source_adapters/_ADAPTER_MAP). Only then is the source-creation dropdown a pure projection of engine.connectors; today direct-only sources (e.g., duckdb when used alone) and adapter-only sources (govdata) are absent from engine.connectors and would be dropped from that projection. Introduce a third Mechanism (FETCH) for (c): Provisa's adapter fetches rows; write face lands them; engine reads the replica. This retires the parallel SOURCE_TO_CONNECTOR map and adapter_loaders special-casing.

**Use case:** engine.connectors must be complete — federatable ∪ Provisa-direct ∪ adapter-materializable — so that the source-creation dropdown dynamically reflects exactly what each federation engine can reach, whether through its native connectors, direct drivers, or adapters. Eliminates hardcoded source-type lists and parallel registry maps.

**Code:** `provisa/federation/connector.py`, `provisa/federation/engine.py`, `provisa/events/source_loader.py`, `provisa/executor/drivers/registry.py`, `provisa/source_adapters/_ADAPTER_MAP`

**Tests:** `tests/unit/test_source_registry_contract.py`, `tests/unit/test_engine_reach_faces.py`

### REQ-948 · Driver Management {#REQ-948}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Driver availability is per-connector data, not new machinery: Connector.runtime_deps entries are tagged by who provides the driver — "system" (OS-provided), "bundled" (Provisa ships + relocates it), "operator" (BYO, must install) — and Connector.probe() ([REQ-904](#REQ-904)) reports functional availability + operator remediation. A source is shown in the dropdown but rendered disabled with its remediation when its driver is operator-provided and not installed. Bundled subset Provisa distributes: postgres (libpq), duckdb (libduckdb), sqlite (system libsqlite3), csv, parquet, iceberg, sqlserver, mysql, plus any source reachable via SQLAlchemy fallback driver. Niche drivers (e.g. oracle, some SaaS) stay operator/BYO.

**Use case:** Tagging driver availability per-connector allows stewards to see which sources are available, which require operator installation, and what remediation is needed — without silent failures or hardcoded availability checks.

**Code:** `provisa/federation/connector.py`

**Tests:** `tests/unit/test_connector_reach_modes.py`

### REQ-949 · Connector Abstraction {#REQ-949}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Structural debt: provisa/federation/connector.py and provisa/federation/engine.py are monoliths that do not scale as more federation engines are added. Split them into per-engine modules (e.g. provisa/federation/connectors/<engine>.py and per-engine engine builders) before/as new engines land.

**Use case:** As Provisa adds support for additional federation engines (DuckDB, ClickHouse, etc.), centralizing all engine-specific code in monolithic modules becomes unmaintainable. Per-engine module organization enables independent engine contributions without merge conflicts or coupling.

**Code:** `provisa/federation/trino_connectors.py`, `provisa/federation/clickhouse_connectors.py`, `provisa/federation/connector.py`, `provisa/federation/engine.py`

**Tests:** `tests/unit/test_federation_engine.py`, `tests/unit/test_clickhouse_connectors.py`, `tests/unit/test_connector_reach_modes.py`

### REQ-950 · Wire-compatible RDB Families {#REQ-950}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Wire-compatible RDB reuses its base wire's native async driver, SQLGlot dialect, and Trino connector via registry data only. Implemented for CockroachDB/YugabyteDB/Greenplum (Postgres-wire) and TiDB (MySQL-wire) via SourceType + SOURCE_TO_CONNECTOR + SOURCE_TO_DIALECT + _PG_WIRE_TYPES/_MYSQL_WIRE_TYPES + jdbc_url + _TRINO_JDBC_TYPES + _DRIVER_FACTORIES. Each federates VIRTUAL on Trino and is direct-route capable.

**Code:** `provisa/core/models.py`, `provisa/federation/connector.py`, `provisa/executor/drivers/registry.py`

**Tests:** `tests/unit/test_wire_compatible_rdbs.py`

### REQ-951 · Connector Abstraction {#REQ-951}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Connector declares a CAPABILITY SET of reach modes (mechanisms: frozenset[Mechanism]) instead of a single fixed mechanism. Extend Mechanism to include SCAN and a Provisa-direct materialize member (FETCH). Examples: Trino postgresql = {ATTACH, LAND}; API source (openapi/graphql_remote/grpc_remote) = {FETCH} (Provisa fetches + write-face lands; engine reads replica); lake file on DuckDB = {SCAN}; warehouse-native = {LAND into self}. federate(), strategy.py, CatalogEntry.mechanism, materialization.py, and promote.py read the declared capability set and pick a reach mode per policy/cost instead of a single mechanism plus scattered heuristics (prefer_materialized, cost-based promotion). Materializer's flag (materialize_store) and planner logic are simplified.

**Use case:** Declaring a full set of reach modes per connector enables federate() and the planner to intelligently select reach modes per policy and cost, eliminating ad-hoc heuristics and making reach flexibility explicit and testable.

**Code:** `provisa/federation/connector.py`, `provisa/federation/strategy.py`, `provisa/federation/materialization.py`, `provisa/federation/promote.py`, `provisa/core/catalog.py`

**Tests:** `tests/unit/test_connector_reach_modes.py`

### REQ-952 · Source Type Availability {#REQ-952}

**Status:** ✅ complete · **Priority:** MUST · **Type:** ui

The source-creation type dropdown displays the UNION of all possible source types, gated and annotated against the currently-configured federation engine to educate users on the impact of engine choice. Backend engine_registry() exposes per-engine reach faces: live_source_types (types read LIVE via attach connector — ATTACH_RW/ATTACH_R) and reachable_source_types (every configurable type = live-attach ∪ _MATERIALIZE_ONLY ∪ Provisa-direct drivers/adapters that land a refreshed replica). UI tags each option: LIVE (attach, in place, fresh) or REPLICA (Provisa lands a refreshed copy). An unreachable type is disabled and annotated "LIVE w/ {engine labels}" naming the engines that reach it live.

**Use case:** Users can see at source-creation time which types are available live on the current engine and which would require materialization, enabling informed connectivity choices without trial-and-error or missing sources in the dropdown.

**Code:** `provisa/federation/engine.py`, `provisa-ui/src/pages/SourcesPage.tsx`

**Tests:** `tests/unit/test_engine_reach_faces.py`

## 9. Event Processing

### REQ-953 · Replica Lifecycle {#REQ-953}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Source replicas are built at boot by posting replace events for each source, and refreshed by events based on their cache_ttl cadence or push listeners.

**Use case:** Ensures all source data is available immediately at startup and stays current during runtime through event-driven updates.

**Code:** `provisa/events/boot.py`

**Tests:** `tests/unit/test_boot.py`, `tests/unit/test_app_wiring.py`

## 4. Source Connectors

### REQ-954 · Replica Strategy {#REQ-954}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

To materialize files/sharepoint/splunk sources, Provisa starts the connector's bundled Calcite pgwire server (pgwire-file, pgwire-sharepoint, pgwire-splunk from github.com/kenstott/calcite release engine-v0.28.0), connects to it as a generic PostgreSQL endpoint, and SELECTs from the connector schema to land rows into the materialize store.

**Use case:** Enables replica materialization for connector-backed sources independent of the federation engine's own connectors, making them reachable via pgwire bridge.

**Code:** `provisa/federation/strategy.py`, `provisa/federation/pgwire_replica.py`, `provisa/events/source_loader.py`, `provisa/events/app_wiring.py`

**Tests:** `tests/unit/test_replica_strategy.py`

### REQ-955 · Replica Strategy {#REQ-955}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Each pgwire server is configured via model/model.json in the bundle with source-specific credentials and storage paths: files → directory (or S3 storageType + storageConfig + AWS env vars); sharepoint → siteUrl, tenantId, clientId, clientSecret (or cert/device-code); splunk → url (or host/port/protocol), token or username/password, app. Provisa manages server lifecycle (start/health/stop) per source. Each server is allocated a unique --port (default 5433) and --calcite-child host:port (default 127.0.0.1:5533) to enable concurrent execution without collision; port allocation is handled transparently by the pgwire-<connector> CLI wrapper (./bin/pgwire-<connector> --port <N> copies args into python -m pgwire_calcite.launcher). No PGWIRE_PORT env var exists; only --port CLI flag and PGWIRE_PYTHON/PGWIRE_HOME/PGWIRE_CALCITE_CLASSPATH* env vars.

**Use case:** Centralizes per-source configuration and lifecycle management so each connector's pgwire server is properly authenticated, monitored, and port-isolated for concurrent operation across multiple federated sources.

**Code:** `provisa/federation/strategy.py`, `provisa/federation/pgwire_replica.py`

**Tests:** `tests/unit/test_replica_strategy.py`

### REQ-956 · Replica Strategy {#REQ-956}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Pgwire bundles (pgwire-file, pgwire-sharepoint, pgwire-splunk) are fetched on demand from github.com/kenstott/calcite release engine-v0.28.0 at runtime and cached locally; they are not bundled with Provisa. Version pinning and artifact download are managed via runtime_deps system.

**Use case:** Decouples Provisa release cycles from pgwire connector releases, and keeps the Provisa distribution lean by fetching connector bundles on-demand.

**Code:** `provisa/runtime_deps`, `provisa/federation/pgwire_replica.py`

**Tests:** `tests/unit/test_replica_strategy.py`

## 9. Live Data & Events

### REQ-957 · Table Processor (Shared) {#REQ-957}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A node's processing envelope collapses to ONE optional user hook: preprocess(rows, ctx) -> rows, run after produce (source fetch or MV SQL) and before land. Absent = identity. Returning rows proceeds to land; returning [] is a row-level no-op (nothing lands, no re-post); ctx.warn(reasons) emits an advisory warn event and keeps going (non-fatal); raising emits an error event and short-circuits the land (fatal), fanning the error forward for poison propagation. Reject is not a new outcome — it is an error ([REQ-941](#REQ-941)), just emitted before landing. warn/error severity replaces any separate accept/reject channel. preprocess MUST be deterministic (its output feeds the content hash that gates the re-post; non-determinism ripples forever). Built-ins NOT in the hook: pre-produce poison short-circuit (a claimed upstream error skips produce) and Defer (a lease action, not an emitted event). ctx is read-only: node name/kind, claimed-event summary, columns/schema, per-input frontier, forced flag, last-landed hash/epoch, and window bounds.

**Use case:** Authors express custom data prep, enrichment, validation, quarantine, and rejection through a single deterministic transform that folds into the existing event-type vocabulary (delta/append/replace/warn/error), with no bespoke gate machinery.

**Code:** `provisa/events/processor.py`, `provisa/events/handlers.py`, `provisa/events/boot.py`, `provisa/mv/preprocess.py`, `provisa/mv/models.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_live_core_loop.py`, `tests/unit/test_mv_preprocess.py`

### REQ-958 · Derived / MV Processor {#REQ-958}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Temporal MVs are expressed as completeness-gated processing windows, not a separate snapshot pipeline. A node's required-input set is its declared expected-events list ([REQ-961](#REQ-961)), defaulting to all SQL-lineage inputs (extract_inputs, [REQ-939](#REQ-939)), verified as a freshness contract at fire time (no NO_CHANGE events). A window opens on the first fanned-in event, gathers and coalesces subsequent events (fan-in collapse), and fires at the claim's deadline (window.end + allowed_lateness). produce reads as-of ctx.window.end (never now(), to stay deterministic and replayable); the result lands as an append keyed on the window boundary (accumulating lineage), so a point-in-time snapshot (e.g. daily sales) is a read-time filter over the accumulating MV. Matching "received for this window" REQUIRES a window/epoch tag stamped on each event at emit time. Late data policy is per-node: drop, or emit a correction partition (never rewrite a sealed one). A window pegs by time and aligns time-keyed inputs; joins across inputs with no common time key still need frontier machinery. The accumulating-append + read-time-filter form serves PIT as a VIEW over retained RT history and therefore stays INSIDE the NRT ideal ([REQ-966](#REQ-966)) — it is reconstructible, not a departure; the departure line is reconstructible-vs-frozen, and a PIT dataset that CANNOT be rebuilt from current state plus retained event history is a preserved, sealed snapshot ([REQ-983](#REQ-983)), not this.

**Use case:** Snapshot/temporal MVs (daily sales, period reporting) and torn-read-free joins are served by one primitive — wait for the lineage-required inputs, then compute as-of the window boundary — collapsing the trigger, the as-of peg, the finality proxy, and partial-recompute cost into a single windowing option.

**Code:** `provisa/events/lineage.py`, `provisa/events/processor.py`, `provisa/events/deadlines.py`, `provisa/federation/store_writer.py`

**Tests:** `tests/unit/test_live_temporal.py`

### REQ-959 · Table Processor (Shared) {#REQ-959}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Claim scheduling and failover rest on ONE correctness anchor — a compare-and-set on the claim's processor_name at commit — with every other timer demoted to a heuristic that is safe to be wrong. A claim declares a per-claim deadline (fire-by); there is no global lease. Three independent recovery paths, all resolved by the ownership CAS: (1) reassert-on-restart — a returning owner CASes its remembered claims; matched = resume (rehydrate deadline/window from the row), zero rows = a peer took it, drop; (2) optional heartbeat — a lapsed lease lets a peer take over a CRASHED owner before the deadline (failover-latency knob only; false positives cost a wasted recompute, never a double effect); (3) deadline-miss reclaim — deadline+grace passed and not completed lets a peer take over a STUCK-BUT-ALIVE owner (the failure a heartbeat structurally cannot catch). reclaimable = (heartbeat lapsed) OR (deadline+grace passed AND not completed). Deadline is availability-mode (fire partial at deadline) or consistency-mode (fire only on completeness; deadline is an SLA alert, not a trigger), per node. deadline may be self-declared at claim or derived as min(fixed, time-to-deadline).

**Use case:** Long-held batch/window claims survive worker crash, restart, and stuck-but-alive workers with no coordination service and no false-positive double-processing, because the ownership CAS at commit — not any lease's accuracy — decides who commits.

**Code:** `provisa/events/queue.py`, `provisa/events/supervisor.py`, `provisa/events/processor.py`

**Tests:** `tests/unit/test_live_core_loop.py`, `tests/unit/test_processor.py`

### REQ-960 · Table Processor (Shared) {#REQ-960}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The processing loop's commit ordering and landing MUST be crash-safe under at-least-once resume. post + fan_out + complete run AFTER land and in one ownership-CAS-guarded control-plane transaction (post-before-complete): a crash between land and commit re-claims and re-runs, never losing the downstream ripple. Because land hits the store DB — a different database from the control-plane queue, so no shared transaction — land MUST be idempotent on the window key (replace overwrites; append upserts by window key, never blind-appends), so a re-run or a rare double-land from a reclaim race is harmless. Combined with recompute-to-current/window-scoped generate (idempotent) and the ownership CAS at commit, this yields exactly-one committed effect from any number of resumed attempts.

**Use case:** Guarantees no lost fan-out and no duplicate partitions across worker crash, restart, and failover, without distributed transactions — the two code gaps (complete-before-post ordering; land not keyed for idempotency) are closed as an invariant.

**Code:** `provisa/events/processor.py`, `provisa/federation/store_writer.py`, `provisa/federation/materialize_exec.py`

**Tests:** `tests/unit/test_live_core_loop.py`, `tests/unit/test_processor.py`

### REQ-961 · Derived / MV Processor {#REQ-961}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A periodic MV's processing trigger is the CALENDAR boundary ([REQ-962](#REQ-962)), bounded by the claim deadline (window.end + allowed_lateness) — NOT a completeness barrier and NOT a stream of NO_CHANGE events. There is no NO_CHANGE event type. The DECLARED expected-events list is the report's FRESHNESS CONTRACT: the inputs that must be fresh-through-window.end for the output to be trusted, verified by a PULL against per-input freshness state at fire time — not by receiving events. List length is the trust/latency dial: a fact-table subset = periodic fact-gated (dimensions read as-of window.end, never listed); all lineage inputs = strong; empty = calendar-only (compute the closed period, verify nothing). Undeclared default = all lineage inputs. At fire (the deadline; earlier only if a node opts into completeness firing): produce as-of window.end from landed data, then verify each listed input is fresh-through-window.end (its source successfully refreshed to cover the window, with zero or more rows). All fresh → trusted; zero rows on a fresh input → a trustworthy zero. A listed input NOT fresh-through-window.end at the deadline → outage → warn/hold (expected-but-absent) — the accountability the existence gate provides without a human — UNLESS the calendar says the window should not exist (holiday), which removes the expectation and the alarm. Early completeness firing is an optional optimization with little value for periodic reports; the push-completeness signal it needs is only justified for a continuous always-current MV requiring torn-read-free early firing, out of scope here.

**Use case:** A single declarative freshness contract replaces gating/reference classification and NO_CHANGE punctuation — the author lists which inputs must be current for the report to be trusted, matching a human's expectation of whether an output is due, so an absent output is unambiguously an outage (required input stale) or a legitimate no-op (holiday / fresh zero), decided by input freshness rather than by an event.

**Code:** `provisa/events/processor.py`, `provisa/events/freshness_contract.py`, `provisa/events/deadlines.py`, `provisa/events/lineage.py`

**Tests:** `tests/unit/test_live_temporal.py`

### REQ-962 · Derived / MV Processor {#REQ-962}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Temporal window boundaries come from named, shared, versioned CALENDARS — not fixed intervals. A calendar is a registered definition: base system (Gregorian / fiscal / retail 4-4-5), timezone (DST-aware — days are 23/25h, never a fixed offset), fiscal anchor, and a versioned business-day/holiday set. An MV declares (calendar, grain ∈ daily/weekly/monthly/quarterly/annual/…); the calendar deterministically yields [start,end), a calendar-addressable window id (e.g. 2026-Q1), and the next boundary — driving the as-of peg (window.end), the claim deadline (boundary + per-grain lateness), the scheduler wake, and forced-regen addressing. Grains NEST (day ⊂ week ⊂ month ⊂ quarter ⊂ year) so a coarse roll-up MV's expected set is its constituent sealed sub- windows, and a shared calendar guarantees the nesting is consistent across the DAG. The calendar also GATES WINDOW EXISTENCE: a business-day grain has no window on a non- business-day (holiday) → the MV deterministically does not generate and raises no deadline alarm; a calendar-day grain still opens a window (zero/empty row) — the grain picks which, because "no sales row" and "zero-stock row" are both valid. Holiday data is versioned/immutable-per-window for replay fidelity — never silently drifted. Per-grain: allowed_lateness and the availability-vs-consistency deadline policy ([REQ-959](#REQ-959)).

**Use case:** Quarterly/annual/daily/fiscal/retail MVs get correct, nesting, DST- and holiday-aware boundaries from one shared calendar registry, and holidays become a deterministic "no window" rather than a silent gap or a spurious deadline alarm.

**Code:** `provisa/events/calendars.py`, `provisa/events/deadlines.py`, `provisa/events/processor.py`, `provisa/events/boot.py`

**Tests:** `tests/unit/test_live_temporal.py`

### REQ-963 · Derived / MV Processor {#REQ-963}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Live (always-current) MVs debounce input churn using the SAME claim+deadline primitive as periodic MVs ([REQ-959](#REQ-959)) — only the deadline SOURCE differs. Periodic: deadline = calendar boundary + lateness ([REQ-962](#REQ-962)). Live: deadline = min(last_change + quiet, first_change + max_delay), a trailing-edge debounce with a hard cap. The max_delay cap is MANDATORY, not optional: pure quiet-reset starves under continuous churn (e.g. a report over 10 frequently-changing tables never goes quiet), so the cap guarantees a refresh and IS the staleness SLA — "wait for a quiet lull, but never be more than max_delay stale"; quiet=0 disables debounce (real-time). The first triggering change opens a claim; it absorbs and coalesces subsequent changes (fan-in collapse — a cross-table burst of N events becomes ONE recompute); it fires once at the deadline. Safe ONLY because recompute- to-current is idempotent — debounce deliberately drops intermediate states and recomputes the final state once. Live MV specifics vs periodic: computes as-of now (no window.end peg) and REPLACE-lands (forgetful current view, not append/accumulate); the expected- events list is the TRIGGER SET (which inputs' changes debounce-fire the recompute), not a freshness contract — unlisted inputs are read at recompute time but do not trigger. quiet and max_delay are per-MV. Debounce controls recompute FREQUENCY, not cost — per-fire cost is the orthogonal incremental-maintenance axis. All [REQ-959](#REQ-959) failover (reassert, deadline- takeover, ownership CAS) applies unchanged since the debounce deadline is the claim deadline.

**Use case:** A live report over many frequently-changing tables refreshes without churn — a burst of input changes collapses into a single recompute after a quiet lull, bounded by a max_delay staleness SLA — reusing the periodic MV's claim+deadline machinery with a debounce-derived deadline instead of a calendar-derived one.

**Code:** `provisa/events/processor.py`, `provisa/events/deadlines.py`, `provisa/events/queue.py`

**Tests:** `tests/unit/test_live_temporal.py`, `tests/unit/test_processor.py`

## 0. Architecture & Design Principles

### REQ-964 · Data-Model-Driven Estate {#REQ-964}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The materialization estate is DATA-MODEL-DRIVEN, not process-driven — the key differentiator. You declare the authoritative, governed data you want represented (each MV as SQL) and the RULES for processing/reprocessing (expected-events freshness contract [REQ-961](#REQ-961), calendar/grain [REQ-962](#REQ-962), deadline/debounce [REQ-959](#REQ-959)/[REQ-963](#REQ-963), preprocess [REQ-957](#REQ-957)); the system DERIVES the process — lineage from SQL ([REQ-939](#REQ-939)), fan-out, scheduling, and failover — with nothing hand-orchestrated. This inverts the imperative pipeline model (dbt/Airflow author the process; the data model is emergent) and yields a single, almost- PROVABLE data estate: a deterministic function of (governed sources, SQL definitions, rules). "Almost" is bounded by five explicit PROOF OBLIGATIONS, each pushable toward mechanical enforcement: (1) transform determinism — MV SQL and preprocess must be pure (reject now()/random at registration; sandbox from wall-clock); (2) source freshness honesty — the one irreducible trust boundary, "trusted" is relative to what a source reports; (3) land idempotency on the window key — exactly-one-effect under the store/queue two-database split rests on this + ownership CAS ([REQ-960](#REQ-960)), not a distributed transaction; (4) temporal-join correctness — the barrier guarantees input availability, not point-in- time-correct (SCD-2) joins, which are the SQL author's obligation (lintable); (5) calendar immutability — replay fidelity requires holiday/fiscal data immutable-per-window ([REQ-962](#REQ-962)). Close 1/3/5 mechanically, lint 4, and the estate is provable except where a source lies — the strongest guarantee a federated estate can make.

**Use case:** Governance and reproducibility are structural properties of the estate, not process hygiene: because lineage is derived (cannot drift), outputs are addressable and deterministically replayable, and trust is explicit via the freshness contract, the whole data estate is auditable as a function of its declared inputs and rules.

**Code:** `provisa/events/lineage.py`, `provisa/events/boot.py`, `provisa/events/handlers.py`

**Tests:** `tests/unit/test_live_selfdescribing.py`

## 9. Live Data & Events

### REQ-965 · Derived / MV Processor {#REQ-965}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

An MV has TWO INDEPENDENT operator-declared outcomes, not one: a PERSISTENCE outcome (how the compute result is applied to the MV's own store table — replace / append / upsert, via land_replace / land_append / apply_cdc already in materialize_exec) and a SET of EVENT outcomes emitted downstream: per fire an MV may emit ANY, ALL, or NONE of {replace, append, delta} (alongside warn / error), each shape routed to the dependents that consume it — persistence is a single choice, the event side is a set, not a scalar. Emission is DEMAND- DRIVEN and pay-per-consumer: a shape is produced only if a dependent (or an explicit sink) subscribes to it, so delta's diff cost is paid ONLY when delta has a subscriber. This is what makes operational datasets emergent — a History dataset is the subscriber of the append stream, a DQ dataset the subscriber of the warn/error stream, incremental dependents the subscribers of the delta stream, all from the same MV fire; the estate becomes self- describing (quality, history, freshness, lineage, audit are just more MVs over the streams the machinery already emits). The event outcomes are symmetric with the input side (select_landing_shape / change_signal, [REQ-932](#REQ-932)), so a dependent MV cannot distinguish a base source from an upstream MV — the uniform DAG completed on the output side. The two axes are decoupled along BOTH cost and semantics: (cost) persist=replace + emit=delta recomputes the whole table but diffs new-vs-old ONCE at emit to hand every dependent a precise delta instead of a recompute; persist=upsert + emit=replace maintains precisely but tells downstream only "recompute"; (semantics) persist=replace + emit=append lets a forgetful current-state MV feed a downstream that KEEPS HISTORY — it overwrites itself yet emits its output as an append batch the downstream accumulates. The event outcome is a downstream-facing contract chosen for downstream's needs, not derived from persistence. Emit shapes form a cost/semantics ladder: replace = cheapest, a recompute signal → downstream recomputes; append = emit produced rows as a batch, no diff → downstream accumulates; delta = needs row identity (PK) and possibly a diff at emit if persistence did not already yield the changed rows → downstream applies precisely. Governing rule: event granularity <= (what persistence yields) + an optional emit-diff, whose cost — like the persistence cost — is a DESIGN CHOICE only the operator can make (dataset scale, incrementalizability, prior-state storage, recompute-vs-diff SLA); for very large datasets a diff/delta may be infeasible and the operator declares coarser shapes. The system NEVER infers a shape nor silently downgrades: a declared-but-infeasible outcome on either axis (emit=delta or persist=upsert without a PK, a diff exceeding a declared bound) is an EXPLICIT ERROR surfaced to the operator, never a silent fallback — that would hide churn/cost and break provability ([REQ-964](#REQ-964)). A maintenance mode MAY supply defaults (periodic → append, live → replace) but the operator's declarations are authoritative. Concretely, persist=replace (a limited, performance-sized hot/current table) + emit=append (a downstream history accumulator) keeps PIT DERIVABLE from RT and therefore INSIDE the NRT ideal ([REQ-966](#REQ-966)) — no departure; the frozen, non-reconstructible snapshot is the separate departure ([REQ-983](#REQ-983)).

**Use case:** An MV is a source to its dependents, and how it stores itself is separable from what it tells them: the operator declares a persistence outcome for its own storage economics and an event outcome for the downstream contract — enabling a forgetful view to feed a historical accumulator, or a full recompute to propagate as a precise delta — so incremental (or historical) change flows through the DAG exactly as the operator chose and can afford, or the estate errors explicitly rather than degrading silently.

**Code:** `provisa/events/outcomes.py`, `provisa/events/handlers.py`, `provisa/events/processor.py`, `provisa/events/boot.py`, `provisa/events/app_wiring.py`, `provisa/federation/materialize_exec.py`, `provisa/federation/store_writer.py`, `provisa/core/change_signal.py`, `provisa/mv/models.py`

**Tests:** `tests/unit/test_live_mv_outcomes.py`

## 0. Architecture & Design Principles

### REQ-966 · Data-Model-Driven Estate {#REQ-966}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The zero-config default is the HAPPY PATH of an always-on, real-time enterprise, and it is the model's canonical IDEAL: an MV needs only its SQL — lineage is auto-derived ([REQ-939](#REQ-939)), fan-out auto-wired, failover (ownership CAS + reassert, [REQ-959](#REQ-959)/[REQ-960](#REQ-960)) always-on and config-free — and everything self-organizes into the DAG and reprocesses in near-real-time (event-driven, recompute-to-current, replace-persist, demand-emit). EVERY other primitive is PROGRESSIVE OPT-IN CONFIG that expresses a justified DEPARTURE from that ideal. Every departure traces to exactly TWO ROOT CAUSES, and config SHOULD be tagged with which: (1) RESOURCE constraint — "can't afford NRT" — cost/scale/latency limits behind debounce ([REQ-963](#REQ-963)), deadlines/batch ([REQ-959](#REQ-959)), fact-gating ([REQ-961](#REQ-961)), delta/incremental ([REQ-965](#REQ-965)), and (at the storage level) even pre-cut rollups vs filtering the live stream ([REQ-958](#REQ-958)/ [REQ-962](#REQ-962)); (2) BUSINESS MANDATE — "the biz wants EOD reports and that's the way it is" — period boundaries and history the organization requires ([REQ-958](#REQ-958)/[REQ-962](#REQ-962)/[REQ-965](#REQ-965)). The two have OPPOSITE TRAJECTORIES: resource concessions are DEBT that retires as infrastructure improves (cheaper compute retires debounce, reliable sources retire deadlines, incremental engines make NRT-at-scale free). Business mandates stratify: ENDOGENOUS mandates — internal habits (EOD/monthly cadences) formed because real-time was infeasible — dissolve as NRT normalizes, because capability RESHAPES demand (free live output makes people stop asking for EOD); EXOGENOUS mandates — market close, filing deadlines, counterparty cutoffs — are imposed from outside and no compute or culture-shift retires them (the genuine floor). Defaults are designed, documented baselines, NOT silent fallbacks — the operator always knows the NRT default and departs explicitly. The why-tag makes the config surface a SELF-DESCRIBING LEDGER (itself queryable as a dataset): resource-tagged = retirement backlog / infra-ROI map ("what real-time maturity costs to buy"); endogenous-mandate = adapts as culture catches up; exogenous-mandate = the permanent business-requirement registry ("what you can never buy out of, and who owns it") — and it surfaces FOSSILIZED constraints (EOD because an overnight mainframe batch once required it) as retireable debt masquerading as mandate. But the knobs are PERMANENT for a reason bigger than any floor: business culture and technical capability are two independently-moving frontiers that NEVER FULLY SYNC — tech outruns adoption, biz outruns capability, and the gap is never zero, never static, and differs per dataset. So the config is not merely retireable compromise but the permanent ALIGNMENT LAYER coupling two co-evolving systems — which is WHY it must be declarative (biz expresses intent), granular / per-node (the gap differs per dataset), continuously retunable (the gap moves), and non- destructive (retune re-aligns without re-authoring process). Config measures distance from the ideal (sharpening [REQ-964](#REQ-964)); the knobs keep intent and capability coupled while both move.

**Use case:** Onboarding is SQL-only and the platform degrades from the NRT ideal one justified, why-tagged config at a time — so the sum of an org's config is a queryable map of its distance from always-on real-time, split into retireable-with-investment vs permanent-business-truth, and reducing that distance is a matter of retiring resource-driven config as infrastructure allows (never re-authoring process) down to the irreducible mandate floor.

**Code:** `provisa/events/boot.py`, `provisa/events/lineage.py`, `provisa/events/processor.py`

**Tests:** `tests/unit/test_live_selfdescribing.py`

### REQ-967 · Data-Model-Driven Estate {#REQ-967}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The estate is SELF-DESCRIBING: because every change is an event and every derived dataset is an MV, the system's OWN operational metadata is emergent — each operational concern is just an MV over streams the machinery already emits, with no bespoke subsystem. Catalog: DQ / quality = an MV over the warn/error emissions ([REQ-957](#REQ-957), [REQ-941](#REQ-941)), append-keyed → a scorecard and quality history; HISTORY / SCD-2 = an MV over an append emission from a replace-persist upstream ([REQ-965](#REQ-965)) → a time series, which also RETIRES proof-obligation #4 ([REQ-964](#REQ-964)) — the SCD-2 dataset IS the point-in-time-correct dimension, so temporal joins become a declared dataset instead of an unenforced author burden (this reconstructible SCD-2 view is PIT inside the ideal; a frozen, non-reconstructible snapshot is the [REQ-983](#REQ-983) departure); FRESHNESS / VINTAGE = an MV over the freshness-contract state ([REQ-961](#REQ-961)) → an SLA / as-of dashboard with queryable outage detection; LINEAGE / IMPACT = an MV over the SQL-derived DAG edges ([REQ-939](#REQ-939)) → impact analysis; AUDIT / PROVENANCE = an MV over the durable event log ([REQ-960](#REQ-960) queue) → a replayable operational trail; CHURN / COST = an MV over fire and debounce-collapse events ([REQ-963](#REQ-963)) → a tuning dashboard; CONFIG MATURITY = an MV over why-tagged config ([REQ-966](#REQ-966)) → the distance-from-NRT map. The meta-level (how the estate operates) folds into the object- level (governed MVs) — the estate observes itself with its own machinery, so quality, history, freshness, lineage, audit, and cost are first-class governed data, governed and lineage-tracked like any other dataset. This reflexivity is the COMPLETENESS TEST for the primitive set (you cannot reach outside it to describe it) and the strongest form of the [REQ-964](#REQ-964) thesis: a new operational capability is a DECLARATION — an MV over an existing stream — never a new code path or a separate DQ / history / audit subsystem.

**Use case:** Observability, quality, history, audit, lineage, and cost are not separate subsystems bolted on — they are declared MVs over the streams the platform already emits, so they inherit the same governance, lineage, and reprocessing guarantees as the business data, and new operational views are added by declaration rather than engineering.

**Code:** `provisa/events/handlers.py`, `provisa/events/processor.py`

**Tests:** `tests/unit/test_live_selfdescribing.py`

## 9. Live Data & Events

### REQ-968 · Derived / MV Processor {#REQ-968}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Forced regen / replay recomputes a derived node — or an addressable calendar window (e.g. 2026-Q1, [REQ-962](#REQ-962)) — ON DEMAND, bypassing the token/no-op gate ([REQ-958](#REQ-958)). Scope selection: by SOURCE (regen the roots and let lineage fan-out cascade forward — the natural, always-correct form), by MV NODE (regen a specific derived node without re-landing its sources — for a changed SQL def), or by WINDOW-ID (regen one sealed period). It posts synthetic events marked forced ({forced:true, reason, scope}); a forced event MUST bypass the no-op gate (it recomputes regardless of change) yet still runs preprocess ([REQ-957](#REQ-957)) and honors the output outcomes ([REQ-965](#REQ-965)). The forward cascade is the normal fan-out — dependents recompute in topological order. Idempotent land on the window key ([REQ-960](#REQ-960)) makes replay exactly-once; determinism ([REQ-964](#REQ-964) obligation 1) + calendar-addressable windows ([REQ-962](#REQ-962)) make a window regen EXACT — it recomputes precisely that period from its as-of inputs. Forced events are marked for audit ([REQ-967](#REQ-967)) so a replay is distinguishable from an organic change.

**Use case:** Operators can restate a period, rebuild after a definition change, or recover from a bad load by declaring a scope to regenerate — sources, a node, or a window-id — and the estate replays deterministically and exactly-once through the existing DAG, with no bespoke backfill job.

**Code:** `provisa/events/processor.py`, `provisa/events/injector.py`, `provisa/events/queue.py`, `provisa/events/deadlines.py`

**Tests:** `tests/unit/test_live_regen_snapshots.py`

### REQ-969 · Derived / MV Processor {#REQ-969}

**Status:** ✅ complete · **Priority:** MAY · **Type:** behavioral

Incremental maintenance computes a derived node's output from an upstream DELTA instead of a full recompute — the cost axis that makes emit=delta ([REQ-965](#REQ-965)) worth declaring. When an input arrives as delta/append ([REQ-932](#REQ-932)/[REQ-965](#REQ-965)), an incrementally-maintained MV applies only the changed rows to its prior landed state (persist=upsert/append) and emits the resulting delta, rather than re-running the full SELECT. Feasibility is per-MV and OPERATOR-DECLARED: it is expressible only when the SQL admits an incremental form (many aggregations/joins do via delta rules; some do not) and a PK exists for row identity — a declared-but-infeasible incremental MV is an EXPLICIT ERROR, never a silent fall-back to full recompute ([REQ-964](#REQ-964); a silent fall-back hides cost). Realized either as author-provided incremental SQL or a system delta-rule derivation where the SQL shape permits. Recompute-to-current stays the DEFAULT ([REQ-966](#REQ-966) NRT baseline); incremental is the resource-concession config that lowers per-fire cost for large or hot derived nodes. Conflict to declare around: an incrementally-maintained MV cannot freely drop intermediate deltas the way recompute-to-current can, so it partially conflicts with debounce ([REQ-963](#REQ-963)) — the operator chooses per node.

**Use case:** Large or high-churn derived nodes refresh by applying upstream deltas instead of full recomputation, making delta emission affordable and incremental propagation real — declared per MV where feasible, erroring explicitly where not, never silently degrading.

**Code:** `provisa/events/handlers.py`, `provisa/events/lineage.py`, `provisa/events/boot.py`, `provisa/federation/materialize_exec.py`, `provisa/mv/models.py`

**Tests:** `tests/unit/test_live_regen_snapshots.py`

### REQ-970 · Derived / MV Processor {#REQ-970}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A derived node's store-table schema is DERIVED from its SQL SELECT (output column names + types), not taken from a source — the structural contrast with a replica, whose schema comes from its source ([REQ-846](#REQ-846)). At registration the SELECT is analyzed (SQLGlot + type inference, or a describe against the federation engine) to produce the output columns; the MV store table is created/reconciled to that schema via the existing reconcile_table machinery ([REQ-846](#REQ-846)). A definition change that drifts the output schema → RECREATE (drop + reland on next fire), symmetric to source-drift reconcile. A PK for the derived table — required for persist=upsert / emit=delta / incremental ([REQ-965](#REQ-965)/[REQ-969](#REQ-969)) — is operator-declared or inferred from an unambiguous GROUP BY / distinct key; required-but-absent is an explicit error, never a silent replace. The derived schema flows into lineage ([REQ-939](#REQ-939)) so downstream MVs type-check their inputs against it.

**Use case:** An MV needs only its SQL: its output schema, store table, and PK are derived and reconciled automatically and published into lineage so the DAG is type-checked end to end — no hand- authored DDL, symmetric to but distinct from a replica's source-derived schema.

**Code:** `provisa/federation/store_writer.py`, `provisa/events/lineage.py`, `provisa/events/app_wiring.py`, `provisa/compiler`

**Tests:** `tests/unit/test_live_mv_outcomes.py`

## 6. Execution, Routing, Caching & Performance

### REQ-971 · Data Masking & Expression Pushdown {#REQ-971}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Column-level data masking MUST be expressed as semantic SQL projection expressions within the governed IR so that masking participates in expression/projection pushdown exactly as RLS predicates participate in predicate pushdown. Where a connector can evaluate the mask expression at the source, the mask MUST push down so the raw, unmasked column value never crosses the wire into the middle tier. Any mask that does NOT push down to the source MUST be evaluated in a bounded-memory streaming stage — either engine-side (the fed engine computes the mask expression and streams results) or per-event/per-batch in the middle tier (as the subscription path already does via _mask_row over an async event stream). It is FORBIDDEN to materialize the full unmasked relation in the middle tier to mask it (no fetchall()-style buffer-then-mask). This streaming constraint ensures masking correctness never depends on fed-engine pushdown capability — a capability gap must never become a data leak — and bounds middle-tier memory to O(batch), not O(relation), minimizing the amount of cleartext resident in the tier at any instant.

**Use case:** Masking today injects scalar SQL expressions (REGEXP_REPLACE, DATE_TRUNC, constant) into the governed SELECT projection after RLS injection. This works correctly but lacks first-class pushdown capability: the planner cannot reason about whether a mask pushed to the source, and no Capability flag tracks whether the source can evaluate the mask's functions. Without projection pushdown, the raw cleartext value must transit to the engine even if the source could mask it — a confidentiality gap analogous to forcing RLS evaluation server-side when the source supports it. The design must model masking as a pushdown-capable expression so confidentiality-aware masking (including a governance guard that acknowledges non-pushable masks) becomes explicit in the planner. For masks that cannot push down, the fallback must evaluate them in a streaming stage to prevent the middle tier from buffering the full unmasked relation into memory — rationale: masking correctness must never depend on pushdown capability (a gap is never a data leak), and the middle tier must bound memory to O(batch), not O(relation), with minimal cleartext residence time.

**Code:** `provisa/compiler/mask_inject.py`, `provisa/compiler/stage2.py`, `provisa/federation/connector.py`, `provisa/federation/promote.py`, `provisa/federation/cardinality.py`, `provisa/api/data/subscribe.py`

**Tests:** `tests/unit/test_mask_pushdown.py`

## 11. Platform, Infrastructure & Delivery

### REQ-972 · Desktop Installation {#REQ-972}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Desktop installers default to a self-contained native install: DuckDB federation engine + sqlite control plane + in-memory (fakeredis) cache, with no Docker, VM, Trino, Redis, or MinIO. A `native` preset in config/capabilities.yaml declares this base tier.

**Use case:** Enables zero-dependency installation for lightweight desktop and demo scenarios without external infrastructure.

**Code:** `config/capabilities.yaml`, `provisa/core/desktop_profile.py`

**Tests:** `tests/unit/test_desktop_profile.py`

### REQ-973 · Desktop Installation {#REQ-973}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Federation engine is a wizard choice, not a fixed tier: DuckDB (native, default) | Trino-on-Docker | External engine (URL or host+port). Trino-on-Docker and external engines are opt-in overrides layered on the native base.

**Use case:** Allows users to upgrade from native DuckDB to Trino or an external federation engine without reinstalling the base system.

**Code:** `provisa/core/desktop_profile.py`, `packaging/macos/first-launch.sh`, `packaging/windows/first-launch-native.ps1`

**Tests:** `tests/unit/test_desktop_profile.py`

### REQ-974 · Desktop Installation {#REQ-974}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

desktop_profile.load_profile gains engine/engine_url/materialize_url/trino_endpoint/otlp_endpoint overrides mapping to PROVISA_ENGINE / PROVISA_ENGINE_URL / PROVISA_MATERIALIZE_URL / TRINO_HOST+PORT / OTEL_EXPORTER_OTLP_ENDPOINT environment variables.

**Use case:** Enables container orchestrators and deployment tools to override engine and observability endpoints at runtime without modifying the installer-created configuration.

**Code:** `provisa/core/desktop_profile.py`

**Tests:** `tests/unit/test_desktop_profile.py`

### REQ-975 · Desktop Installation {#REQ-975}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Observability is always-on built-in (self-telemetry in Admin). The optional Docker observability stack (otel-collector + prometheus + grafana) is an external demonstration, not an on/off switch; selecting it only redirects OTLP export to external infrastructure.

**Use case:** Ensures telemetry is always collected and visible in Admin UI; external obs-integration is optional for enterprise monitoring without adding complexity to the base install.

**Code:** `provisa/core/desktop_profile.py`

**Tests:** `tests/unit/test_desktop_profile.py`

### REQ-976 · Desktop Installation {#REQ-976}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

Docker and VMs are provisioned only when needed: engine == Trino-on-Docker, OR the obs-integration demo runs on Docker, OR the demo runs on Docker. Otherwise the install is fully native.

**Use case:** Minimizes resource footprint for users who do not require Trino or external observability, reducing memory and disk overhead on desktop systems.

**Code:** `packaging/macos/first-launch.sh`

**Tests:** `tests/unit/test_desktop_install.py`

### REQ-977 · Desktop Installation {#REQ-977}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

All heavy add-ons (Trino image set, observability stack, demo) are downloadable and resolved local-first: installer-adjacent directory, then mounted volumes, then ~/Downloads, then GitHub release. Enterprise can pre-stage tarballs for airgapped installation.

**Use case:** Enables fully airgapped deployment by allowing enterprises to pre-stage container images and demo datasets alongside the installer.

**Code:** `packaging/macos/first-launch.sh`

**Tests:** `tests/unit/test_desktop_install.py`

### REQ-978 · Desktop Installation {#REQ-978}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Installing the demo is optional and OFF by default. When installed, the launcher opens the UI at ?tour=1 to auto-start the guided tour.

**Use case:** Separates the base product from demo data; guided tour auto-starts for new users who opt into the demo without manual navigation.

**Code:** `packaging/macos/ProvisaLauncher/Sources/ProvisaLauncher/Views/Setup/SetupWizardView.swift`, `packaging/windows/provisa-native.ps1`, `packaging/macos/first-launch.sh`

**Tests:** `tests/unit/test_desktop_install.py`

### REQ-979 · Desktop Installation {#REQ-979}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

The native tier bundles a standalone Python runtime (python-build-standalone + provisa wheel + duckdb/pg_duckdb extensions + aiosqlite) so the base installer runs the app with no Docker or container dependencies. The base installer ships no container images.

**Use case:** Ensures the entire native-tier stack is self-contained and runs on any OS (macOS, Windows, Linux) without requiring external Python environments or container runtimes.

**Code:** `packaging/macos/build-dmg.sh`, `packaging/windows/build-sfx.ps1`, `packaging/windows/first-launch-native.ps1`, `packaging/windows/provisa-native.ps1`

**Tests:** `tests/unit/test_desktop_install.py`, `tests/unit/test_infra_requirements.py`

## 3. Federation Engines & Data Sources

### REQ-980 · Query Compilation {#REQ-980}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Add `json` as a first-class IR type so native JSON in source columns preserves through to materialized stores as native JSON where supported, instead of collapsing to `text`. The write face maps IR `json` to SQLAlchemy's generic JSON (native on Postgres/MySQL/DuckDB, TEXT-affinity on SQLite via the dialect compiler); the land path parses a JSON column's serialized-text value into a Python object before insert so the SQLAlchemy JSON column does not double-encode; and the read path converts an already-parsed dict/list consistently with the string path. Composite/array types (row/array/list/struct/map) remain collapsed to `text`.

**Use case:** Federation queries over JSON columns maintain type information and enable native JSON operations in target stores, rather than forcing serialization fallbacks.

**Code:** `provisa/core/ir_types.py`, `provisa/federation/materialize_exec.py`, `provisa/executor/serialize.py`

**Tests:** `tests/unit/test_ir_types.py`, `tests/unit/test_materialize_landing.py`

## 9. Live Data & Events

### REQ-981 · Live Data & Events {#REQ-981}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The event loop's content-hash output gate — an always-on no-op ripple suppressor for replace-shaped landings. After a source-land or MV-generate produces its result, the node hashes the landed content (canonical, order-stable over the primary key) and compares it to the hash stored from the prior land. Unchanged → the node does NOT re-post its change event, so no downstream MV recompute cascades; the redundant store write is also skipped. Changed → land, persist the new hash, re-post. This makes a change_signal=ttl / probe_type=none poll (which always re-fetches) cheap on the DOWNSTREAM side: the full fetch still happens, but an unchanged result never ripples the DAG. Applies to replace shapes only (append/CDC deltas are new rows by definition and pass through). The hash is the correctness backstop that probe_type (REQ-981) optimizes the fetch in front of.

**Use case:** A TTL-polled source or MV that re-queries on cadence but whose content rarely changes no longer forces its entire dependent MV subtree to recompute on every tick — the DAG only ripples when landed content actually differs, eliminating wasted downstream work.

**Code:** `provisa/events/processor.py`, `provisa/events/handlers.py`, `provisa/events/content_hash.py`, `provisa/events/queue.py`, `provisa/core/schema_org.py`

**Tests:** `tests/unit/test_content_hash.py`, `tests/unit/test_live_change_gates.py`, `tests/unit/test_handlers.py`

### REQ-982 · Live Data & Events {#REQ-982}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

probe_type is the event loop's input-side change-detection axis, orthogonal to the change_signal cadence (ttl | probe | ttl_probe). Values: watermark | hash | count | none. The type both selects the detection transport AND implies the landing shape: watermark -> append (fetch WHERE wm > cursor, needs watermark_column), hash/count/none -> replace. This REPLACES the prior "watermark_column presence implies append" heuristic in select_landing_shape with an explicit probe_type -> shape map. Availability is gated by a per-source capability matrix probe_capabilities(source_type): SQL/engine-scannable sources support {watermark, hash, count, none} (hash = full-scan checksum, expensive, degenerates to the [REQ-980](#REQ-980) output hash); HTTP/API sources support {hash via ETag/Last-Modified, count via total-header, watermark if a sortable cursor, none}; file/object sources support {hash via mtime+size, none}; push/streaming sources are not on the probe axis. Config validation rejects a probe_type outside the source's capability set; when change_signal in {probe, ttl_probe} and probe_type is unset the default resolves per class (SQL -> watermark if watermark_column else count; API/file -> hash). Each transport implements freshness_token(source, table) -> str | None; a None token degrades the node to TTL cadence ([REQ-847](#REQ-847) capability signal, never a silent stale fallback). The token baseline is persisted per node and compared through evaluate_freshness ([REQ-855](#REQ-855)). The real probe factory replaces the always-changed _poll_probe_factory in the event-loop boot wiring.

**Use case:** An operator declares how each source detects change appropriate to its type — a watermark column for an append-only table, an ETag for an HTTP API, mtime for a file — so the loop skips the fetch entirely when the source is unchanged, rather than re-querying every cadence and relying only on the downstream hash gate.

**Code:** `provisa/core/change_signal.py`, `provisa/core/models.py`, `provisa/events/injector.py`, `provisa/events/boot.py`, `provisa/events/app_wiring.py`, `provisa/events/probes.py`, `provisa/events/handlers.py`, `provisa/federation/residency.py`, `provisa/federation/store_writer.py`, `provisa/federation/freshness_gate.py`

**Tests:** `tests/unit/test_probes.py`, `tests/unit/test_injector.py`, `tests/unit/test_probe_type_config_validation.py`, `tests/unit/test_live_change_gates.py`

### REQ-983 · Derived / MV Processor {#REQ-983}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A preserved snapshot is a point-in-time dataset that is MATERIALIZED-AND-SEALED because it is NOT reconstructible from current state plus retained event history — the one PIT form that is a genuine DEPARTURE from the NRT ideal ([REQ-966](#REQ-966)), as opposed to the reconstructible PIT of [REQ-958](#REQ-958)/[REQ-965](#REQ-965)/[REQ-967](#REQ-967) (accumulating append + read-time filter) which stays inside the ideal. It is DECLARED, never inferred, and MUST be why-tagged with its [REQ-966](#REQ-966) root cause: RESOURCE — the deriving history was never retained or the source cannot be replayed (debt that retires as retention/replay capability improves); or BUSINESS MANDATE — the output must be frozen exactly as computed at a moment (a definition or upstream changed and the prior result must be preserved as-is; an external filing/attestation fixes the number). The snapshot is landed as an IMMUTABLE, addressable partition (a [REQ-962](#REQ-962) window-id or an explicit snapshot-id), sealed like a calendar window: never rewritten, corrected only by a new partition, replay-exact via determinism ([REQ-964](#REQ-964)) where the inputs still exist, and otherwise preserved verbatim because re-derivation is impossible. Distinct from [REQ-958](#REQ-958)'s accumulating-append history (a reconstructible view over RT) and from a live replace-persist hot table ([REQ-965](#REQ-965)): a preserved snapshot is chosen precisely when derive-from-RT is NOT available. Never a silent fallback — the operator declares it with its root cause, or the estate serves PIT as a view.

**Use case:** Real-world estates cannot always retain full event history or replay every source, and some outputs must be frozen as-computed for audit/attestation — so the platform offers PIT datasets built and preserved at a moment as an explicit, why-tagged departure, keeping derivable PIT the default and making every frozen snapshot a visible entry in the distance-from-NRT ledger ([REQ-966](#REQ-966)).

**Code:** `provisa/events/snapshots.py`, `provisa/core/schema_org.py`, `provisa/federation/materialize_exec.py`

**Tests:** `tests/unit/test_live_regen_snapshots.py`

## 6. Materialized Views & Caching

### REQ-984 · Materialized View Lifecycle {#REQ-984}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The legacy periodic CTAS materialized-view refresh loop is retired in favor of the event loop as the sole MV compute path. The refresh_loop function is replaced by reclamation_loop, which handles only storage reclamation (drop removed MVs + reap orphaned tables per [REQ-234](#REQ-234)); periodic refresh cadence is now driven by each MV's event-loop poll job, reframing periodic refresh as the [REQ-963](#REQ-963) debounce deadline source.

**Use case:** Consolidating MV compute into a single event-driven path eliminates double-computation and simplifies the refresh lifecycle while preserving operator-triggered refresh mutations.

**Code:** `provisa/mv/refresh.py`, `provisa/api/app_startup.py`, `provisa/events/boot.py`

**Tests:** `tests/unit/test_mv_reclamation_loop.py`

## 6. Execution, Routing, Caching & Performance

### REQ-985 · Cache {#REQ-985}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

When `cache.enabled` is omitted from config, caching defaults to ENABLED. A CacheStore always exists: `RedisCacheStore(None)` falls back to embedded fakeredis when no Redis URL is configured. Operators must explicitly set `cache.enabled: false` to disable caching via `NoopCacheStore`.

**Use case:** Operators have safe defaults where caching is active unless explicitly disabled, eliminating silent no-cache states and ensuring consistent cache behavior across hot tables and response caches.

**Code:** `provisa/api/app.py`, `provisa/cache/hot_tables.py`

**Tests:** `tests/unit/test_cache_default_enabled.py`

## 3. Federation Engines & Data Sources

### REQ-986 · Arrow Flight Transport {#REQ-986}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

ClickHouse federation engine must expose Arrow-native transport through the Provisa Arrow Flight server to honor its declared ARROW and ARROW_STREAM capabilities, enabling columnar data streaming without row materialization.

**Use case:** ClickHouse can deliver Arrow data natively over HTTP and native TCP protocols, but currently Provisa collapses these to row tuples. Exposing this capability via Arrow Flight matches Trino's architecture and enables efficient columnar streaming for high-throughput federated queries.

**Code:** `provisa/federation/engine.py`, `provisa/federation/backend.py`, `provisa/federation/clickhouse_runtime.py`, `provisa/api/flight/server.py`

**Tests:** `tests/unit/test_engine_capabilities.py`, `tests/unit/test_native_arrow_transport.py`, `tests/integration/test_clickhouse_runtime_e2e.py`, `tests/integration/test_arrow_flight_integration.py`

### REQ-987 · Arrow Flight Transport {#REQ-987}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Databricks SQL warehouse must be supported as a first-class Provisa federation engine with Arrow-native transport capabilities for both READ and WRITE paths. Read-side: Arrow columnar data streaming through the Provisa Arrow Flight server via execute_arrow/execute_stream. Write-side: materialization-store landing, residency reconciliation, and CDC snapshot seed must use Arrow-native columnar bulk writes (via ADBC adbc_ingest() or Parquet -> cloud object stage -> COPY INTO / Delta register), never row-oriented INSERT. End-to-end columnar path avoids row materialization overhead in Python and enables Databricks' distributed ingest parallelization. Row INSERT acceptable only for tiny writes where per-file staging overhead dominates.

**Use case:** Organizations using Databricks SQL warehouses require direct federation capability to Provisa without intermediary routing through Trino. Arrow-native transport for both read and write paths avoids row materialization overhead and matches Trino's architecture for efficient federated query execution over Databricks. Materialized views and landed federation outputs must remain columnar end-to-end, enabling downstream queries to execute at warehouse speed.

**Code:** `provisa/federation/engine.py`, `provisa/federation/backend.py`, `provisa/federation/store_writer.py`, `provisa/api/flight/server.py`, `provisa/transpiler/transpile.py`

**Tests:** `tests/unit/test_databricks_store.py`, `tests/unit/test_engine_capabilities.py`, `tests/integration/test_databricks_federation_engine_e2e.py`, `tests/integration/test_databricks_external_link_e2e.py`

### REQ-988 · Arrow Flight Transport {#REQ-988}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Snowflake federation engine must be promoted from a source-only connector (federated through Trino) to a first-class Provisa federation engine peer to Trino/DuckDB/ClickHouse/Postgres, with Arrow-native transport wired through the Provisa Arrow Flight server. Engine declares ROWS, ARROW, and ARROW_STREAM capabilities; backend implements execute_arrow() and execute_stream() to deliver Arrow columnar data and lazy record-batch streaming without row materialization. Physical SQL transpilation must target Snowflake dialect. Governance (RLS, masking, rate-limiting) applies unchanged above the backend.

**Use case:** Snowflake is an MPP warehouse with native Arrow support; exposing its columnar capabilities via Provisa's Arrow Flight server enables direct, efficient federated queries without row materialization and eliminates the current Trino-as-middleman routing. Matches architectural parity with Trino ([REQ-986](#REQ-986) ClickHouse parallel) and serves enterprises whose primary warehouse is Snowflake.

**Code:** `provisa/federation/engine.py`, `provisa/federation/backend.py`, `provisa/api/flight/server.py`, `provisa/executor/trino_flight.py`, `provisa/transpiler/transpile.py`, `provisa/source_registry.py`, `provisa/source_connectors/snowflake_connector.py`

**Tests:** `tests/unit/test_engine_capabilities.py`, `tests/unit/test_transpiler.py`, `tests/integration/test_snowflake_federation_engine_e2e.py`

### REQ-989 · Platform Defaults {#REQ-989}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Zero-config default stack must be fully embedded and in-process with no external dependencies: default federation engine is duckdb (not trino), default materialize store is duckdb's native embedded store (not platform tenant DB via Postgres), and default control-plane store is sqlite (not embedded_pg). External engines and stores (Trino, Postgres, ClickHouse, Snowflake, Databricks) remain selectable via PROVISA_ENGINE / PROVISA_ENGINE_URL / PROVISA_MATERIALIZE_URL / control_plane_store overrides.

**Use case:** Organizations and developers need instant local startup with zero Docker, no Trino cluster, no Postgres dependency — everything embedded for true out-of-the-box instant start. The zero-config path must serve as an approachable entry point while preserving full enterprise engine/store pluggability via configuration.

**Code:** `provisa/federation/engine.py`, `provisa/core/desktop_profile.py`, `provisa/core/capabilities.yaml`

**Tests:** `tests/unit/test_duckdb_store_native.py`, `tests/unit/test_federation_engine.py`

### REQ-990 · Materialization Writes {#REQ-990}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Materialization-store writes MUST use the target store's columnar or bulk-COPY ingest path wherever available, instead of row-by-row INSERT. Row INSERT is permitted only for targets without bulk support or tiny writes where staging overhead dominates, and this choice must be explicit/capability-gated, never a silent fallback.

**Use case:** Row-by-row INSERT serializes writes, defeats vectorization, and forces Python row tuples through the network. Bulk columnar paths (Arrow, COPY, adbc_ingest) maintain end-to-end columnar efficiency, reduce network overhead, and leverage target-store parallelization. This is especially critical for high-volume materialization, residency reconciliation, and CDC snapshot seeding.

**Code:** `provisa/federation/store_writer.py`, `provisa/federation/store_connection.py`, `provisa/federation/backend.py`

**Tests:** `tests/unit/test_store_writer.py`, `tests/unit/test_databricks_store.py`

## 9. Live Data & Events

### REQ-991 · Live Delivery Strategy {#REQ-991}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Live Delivery mechanism (push repeat, append poll, or full-replace poll) is DERIVED SOLELY from the effective change_signal plus watermark column presence. The Live Delivery UI exposes no independent Strategy selector; the delivery mechanism mirrors change_signal.select_landing_shape: push signal → repeat the change stream; poll signal WITH watermark → append; poll signal WITHOUT watermark → full-replace.

**Use case:** Eliminates redundant strategy selection across detection method and delivery mode. Centralizes authoritative control in Change Signal, reducing cognitive load and preventing inconsistent strategy/detection pairs. The watermark presence is the determining factor for append vs. replace, not a separate configuration.

**Code:** `provisa/live/engine.py`, `provisa/live/reconcile.py`, `provisa-ui/src/pages/tables/LiveDeliveryFieldset.tsx`

**Tests:** `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`, `tests/unit/test_live_delivery_config.py`

### REQ-992 · Live Delivery Strategy {#REQ-992}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Poll signals without a watermark column execute a full-replace poll mode: LiveEngine._poll_replace re-scans the full result set at each interval and emits a replace-mode LiveSpec only when an order-independent content hash (stored per output in live_query_state) changes. Unchanged scan results do not emit a snapshot, suppressing ripples in small or non-monotonic tables.

**Use case:** Non-monotonic or small tables scanned via poll (no watermark) need refresh semantics without the churn of repeated identical snapshots. Content-hash-based delivery suppresses ripples while ensuring consistency after real data changes.

**Code:** `provisa/live/engine.py`, `provisa/live/reconcile.py`

**Tests:** `tests/unit/test_content_hash.py`, `tests/unit/test_live_engine.py`, `tests/unit/test_live_engine_full.py`

### REQ-993 · Live Delivery Strategy {#REQ-993}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The content hash used for full-replace ripple suppression is order-independent (e.g., XOR or sorted multiset hash), stored per-output in live_query_state, and is recomputed on each full-table re-scan. A change in hash always triggers a replace snapshot delivery; an unchanged hash always suppresses delivery.

**Use case:** Order-independent hashing ensures that logical content changes (row addition/deletion) trigger snapshots even if rows appear in different order, while avoiding spurious ripples when only row order differs. Persistent storage in live_query_state allows next interval's scan to detect real changes.

**Code:** `provisa/live/engine.py`

**Tests:** `tests/unit/test_content_hash.py`

## 3. Federation Engines & Data Sources

### REQ-994 · Data Sources {#REQ-994}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Trino/Presto clusters are read as first-class remote SOURCES via SQLAlchemy's trino dialect and landed as REPLICA tables, reaching replication parity with GraphQL, REST, and gRPC sources. Source classification is by read mechanism, not data-at-rest residency; Trino qualifies as both a source and a federation engine, mirroring the dual roles of Snowflake, Databricks, and BigQuery.

**Use case:** Users need to query live Trino coordinators without deploying a separate federation engine. Trino-as-source unifies the source registration surface: a remote Trino cluster lands via the standard replication path, identical to web-service sources, enabling analytics workloads to federate without ETL.

**Code:** `provisa/executor/drivers/registry.py`, `provisa-ui/src/pages/sources/constants.ts`

**Tests:** `tests/unit/test_trino_source_driver.py`

### REQ-995 · Federation Engine Capabilities {#REQ-995}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Microsoft Fabric federation engine attaches Apache Iceberg data LIVE via OneLake Delta metadata virtualization (OPENROWSET FORMAT='DELTA' over virtualized/shortcut paths), extending live-attach reach from {parquet, csv, delta_lake} to include iceberg. Azure Synapse serverless lacks OneLake virtualization, so Iceberg tables are NOT attachable there and must land as REPLICA materialization.

**Use case:** Fabric's OneLake Delta virtualization layer enables direct live queries over Iceberg tables without requiring separate Iceberg compute. Synapse's absence of this virtualization necessitates materializing Iceberg data as REPLICA, preserving correctness while leveraging different engine strengths: Fabric for live lakehouse queries, Synapse for post-materialization analytics.

**Code:** `provisa/federation/mssql_warehouse_connectors.py`

**Tests:** `tests/unit/test_mssql_warehouse_connectors.py`

## 6. Cross-Source Data Movement

### REQ-996 · Physical Data Transfer {#REQ-996}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Support one-time physical data move expressed as standard SQL CREATE TABLE {schema}.{table} AS SELECT ... FROM {schema2}.{table2}. This physically creates and populates a table in a writable source. It does NOT create or register a Provisa model entity (no Domain, no Table model, no creation_request) — the new table is invisible to the federated model until separately introspected/registered.

**Use case:** High-speed cross-data-source bulk copy and migration: users express a one-time physical move as a single SQL statement over the federated namespace without external ETL orchestrator, per-source connector code, or intermediate file staging. The CTAS operates at two speed tiers: same-engine zero-copy native CTAS pushdown, and cross-engine bulk-load via the landing face bounded by target bulk-insert driver performance. No semantic model registration required. Primary derived pattern: snapshot/archive via scheduled CTAS with date templating (e.g., CREATE TABLE archive.orders_{{yyyymmdd}} AS SELECT * FROM live.orders on cron). Each run produces distinct table name, so global schema.table uniqueness ([REQ-998](#REQ-998)) holds automatically with no collision/overwrite; snapshots remain unregistered, ideal for time-windowed archives.

**Code:** `provisa/executor/ctas.py`, `provisa/executor/writable.py`, `provisa/federation/store_writer.py`, `provisa/pgwire/server.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-997 · Execution Routing {#REQ-997}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Execution routing for CTAS: resolve the write path via existing resolve_write_path(target_source_type, engine). If source and target are on the same engine, emit engine-native CTAS pushed down (zero-copy). If cross-engine, the engine runs the SELECT and rows are bulk-loaded into the target source via the landing write face (provisa/federation/store_writer.py) — the engine is never on the write path. Writes get no retry budget (provisa/executor/direct.py) to avoid duplicate side effects.

**Use case:** Routing strategy for high-speed bulk migration: same-engine CTAS achieves zero-copy pushdown; cross-engine routes through landing write face to isolate engine from write side effects. No retry budget prevents accidental duplicate table creation. This two-tier routing model is the performance guarantor for federation-wide bulk copy without intermediate staging.

**Code:** `provisa/executor/ctas.py`, `provisa/executor/writable.py`, `provisa/federation/store_writer.py`, `provisa/executor/direct.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-998 · Name Resolution {#REQ-998}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The load-bearing invariant: schema.table must be globally unique across the entire model. This is the language contract. A new CTAS target whose schema.table collides with an existing asset is rejected (name taken model-wide).

**Use case:** Provisa's catalog-free addressing relies on schema.table as the sole global identifier. Allowing collisions would create ambiguity in later reads: schema.table could refer to multiple physical tables across different sources, breaking the semantic model.

**Code:** `provisa/executor/ctas.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-999 · Name Resolution {#REQ-999}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Catalog is optional in the grammar and, when present, always names the source. It is never used for resolution when absent (schema.table uniqueness resolves it). When present, catalog MUST equal the source that schema.table resolves to — matches proceed (self-documenting), contradiction is an error, never an override. Grammar stays standard three-part catalog.schema.table so SQLGlot parses it natively (read placement off exp.Table.catalog); the model itself stays catalog-free.

**Use case:** Catalog in the SQL statement provides optional documentation/validation of placement intent without overriding the model's catalog-free resolution. Explicit catalog allows users to self-document which source they are writing to; mismatch is an error, preventing accidental writes to wrong sources.

**Code:** `provisa/executor/ctas.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-1000 · Name Resolution {#REQ-1000}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Placement disambiguation at CREATE only: a new unique schema.table normally lands in the single writable source its schema maps to. In the rare case a schema maps to more than one writable source, catalog becomes required to disambiguate placement; absent an unambiguous target the CREATE is rejected (unique-match-or-error). Never silently pick a target.

**Use case:** Schemas can map to multiple writable sources in a federation (e.g., a schema with replicas on two different engines). Explicit catalog requirement prevents silent mis-targeting while allowing valid operations when placement is unambiguous.

**Code:** `provisa/executor/ctas.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-1001 · Transaction Safety {#REQ-1001}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

Transactional safety for the move: create-temp → bulk-load → atomic swap so a mid-stream failure leaves no partial target table.

**Use case:** Long-running or unreliable bulk-load operations may fail partway through. Atomic swap ensures target table is either completely populated or not created at all, preventing partial or corrupted data.

**Code:** `provisa/federation/store_writer.py`, `provisa/executor/ctas.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-1002 · Type Coercion {#REQ-1002}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Cross-engine result-schema → target DDL type coercion policy: the SELECT result schema must map to valid target-dialect DDL (handling engine types like ROW/array/DECIMAL scale). Reuse store_writer ensure_table where possible.

**Use case:** Source engines emit result schemas with types that may not be natively supported by target engines (e.g., Snowflake ROW types, complex arrays, variable-precision DECIMAL). Type coercion ensures that the target table DDL is valid and preserves logical semantics.

**Code:** `provisa/federation/store_writer.py`, `provisa/executor/ctas.py`, `provisa/core/ir_types.py`

**Tests:** `tests/unit/test_ctas.py`

### REQ-1003 · Scheduled Execution {#REQ-1003}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Extend the APScheduler-backed ScheduledTrigger (provisa/core/models.py:640) to support execution of a user-supplied SQL statement on its cron schedule, not only webhook POST. When a trigger fires, the SQL statement executes against the federated engine. This enables scheduled CTAS snapshot/archive patterns ([REQ-996](#REQ-996)).

**Use case:** Scheduled CTAS (CREATE TABLE ... AS SELECT) is the primary derived pattern from one-time CTAS ([REQ-996](#REQ-996)), enabling snapshot and archive tables that update on a regular schedule without requiring external ETL orchestrators or manual user intervention. Users express scheduled snapshots as declarative triggers whose SQL statements run on cron.

**Code:** `provisa/scheduler/jobs.py`, `provisa/scheduler/templating.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_scheduled_sql.py`

### REQ-1004 · Template Substitution {#REQ-1004}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Scheduled SQL statements support run-time substitution of date/timestamp tokens (e.g., {{yyyymmdd}}, {{YYYY-MM-DD}}, {{iso8601}}, {{timestamp}}) into the statement text before execution. Each run substitutes tokens with that run's execution date/time, enabling scheduled CTAS to produce distinct date-varied table names (e.g., archive.orders_{{yyyymmdd}}) without collision or overwrite, preserving the global schema.table uniqueness invariant ([REQ-998](#REQ-998)) automatically. Depends on [REQ-1003](#REQ-1003).

**Use case:** The snapshot/archive pattern requires each run to produce a uniquely-named table. Date tokens allow users to express target table names declaratively (e.g., archive.orders_{{yyyymmdd}}) so scheduled CTAS automatically creates archive.orders_20260712, archive.orders_20260713, etc. without manual collision detection or user-supplied name-generation logic. This is essential for autonomous time-windowed archives that preserve the model's global schema.table uniqueness ([REQ-998](#REQ-998)) while remaining unregistered semantic entities.

**Code:** `provisa/scheduler/templating.py`

**Tests:** `tests/unit/test_scheduled_sql.py`

## 11. Platform, Infrastructure & Delivery

### REQ-1005 · Desktop Installation {#REQ-1005}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** infrastructure

Windows native tier first-launch (after setup completes and before/after opening the UI) presents next-steps guidance telling the user that the federation engine (Trino), observability stack, and demo data pack are NOT part of the native tier and are added via separate layered installers: Container installer (Provisa-Container-*.exe → WSL2 + containerd + Trino), then Obs installer (requires container tier), then Demo installer (requires Core + Obs). Guidance includes how to initialize the federation engine by running the Container installer.

**Use case:** The native tier installer provides a standalone local query engine (core + duckdb) but ships no federation engine (Trino), observability stack, or demo data. Users who install native tier receive no indication of how to access these optional features and the app opens directly to the UI without guidance, creating confusion about what Provisa can do. This requirement ensures users understand the tier model and how to extend their installation.

**Code:** `packaging/windows/first-launch-native.ps1`

**Tests:** `tests/unit/test_desktop_install.py`

### REQ-1006 · Infrastructure {#REQ-1006}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The SPA static server routing is deterministic and driven by the browser Sec-Fetch-Dest request header (never by a maintained path allowlist). Requests with Sec-Fetch-Dest: document or GET whose Accept contains text/html are top-level SPA navigation and receive index.html; all other requests (fetch/XHR/EventSource, iframe subresource, non-GET) are proxied to the API and their real HTTP status — including 404 — is surfaced.

**Use case:** The previous path-allowlist heuristic silently served index.html (HTTP 200 HTML) for any unlisted API path, causing JSON parse errors and UI hangs on "Loading...". Deterministic header-driven routing eliminates both failure classes and prevents silent fallbacks. It also enables SPA deep-link refreshes on client routes without collision with API prefix handlers like /admin.

**Code:** `provisa/ui_server.py`

**Tests:** `provisa-ui/e2e/infrastructure.spec.ts`, `tests/e2e/test_health_checks.py`, `tests/unit/test_ui_server_routing.py`

### REQ-1007 · Multi-Tier Deployment {#REQ-1007}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Deployment choices — federation engine (DuckDB/Trino/external-SQLAlchemy), observability mode (none/Docker-incluster/collector), and demo install — are exposed uniformly across all deployment surfaces (macOS SwiftUI, Linux AppImage, install.sh, Windows native, Helm, Terraform) with environment-variable overrides and interactive prompts where applicable.

**Use case:** Deployment parity ensures users can select the same configuration across desktop, server, container, and cloud deployments without relearning the choice model for each platform.

**Code:** `packaging/linux/first-launch.sh`, `packaging/macos/build-dmg.sh`, `packaging/windows/first-launch-native.ps1`, `packaging/windows/install-container.ps1`, `install.sh`, `helm/provisa/values.yaml`, `helm/provisa/templates/demo.yaml`, `terraform/aws/main.tf`, `terraform/azure/main.tf`, `terraform/gcp/main.tf`, `terraform/deploy.sh`

**Tests:** `tests/unit/test_desktop_install.py`

## 8. Client Access & Protocols

### REQ-1008 · MCP Server {#REQ-1008}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa exposes a Model Context Protocol (MCP) server so external AI agents (Claude Desktop, claude.ai connectors, Claude Code) can discover catalog metadata and execute SQL through the same governed pipeline as every other client. The server is a protocol adapter with zero new governance logic — every SQL path routes through the existing choke point _govern_and_route(sql, role_id) in provisa/pgwire/_pipeline.py, and catalog reads reuse build_catalog_tables(state). It runs in-process in the FastAPI app (shared state) via the official mcp Python SDK (FastMCP). Two transports remote Streamable HTTP with OAuth (token→provisa role via existing pgwire OIDC) and local stdio with a pinned role for development. A role is REQUIRED on every call; the server never defaults to admin. Tools list_schemas, list_tables(schema), describe_table(schema,table), run_sql(sql) (row-limited/paged), explain_sql(sql) (plan-only), and search_catalog(nl_text). Structured drill-down (list_schemas → list_tables → describe_table) is deterministic over the meta views (registered_tables_meta, table_columns_meta) and needs no vector store. search_catalog is semantic bottom-up detail search using small-to-big retrieval embed narrow column-leaf chunks, match on the detail, then resolve up through deterministic drill-down to return the parent table branch (all columns + FKs) plus schema breadcrumb. Three chunk tiers schema (schema+desc+table names), table (table+desc+column names), column (column+desc+table+schema); all columns embedded. A single formatter get_chunk(schema, table?, column?) is the sole chunk-text contract address depth selects the tier (schema | schema+table | schema+table+column), column requires table, and it returns embed-text as labeled plain prose (NOT markdown — markdown chrome is token noise to the embedder; the pretty structured view is describe_table's job for a different consumer). A separate iter_entities() enumerator walks the meta views yielding addresses that drive get_chunk, keeping the formatter pure (address in, text out). The address IS the provenance the args (schema, table, column) equal the {level, schema_id, table_id, column_id} stored on the vector row, so a hit resolves straight back through get_chunk/describe_table with no separate chunk-id→entity mapping. Schema questions are never answered from stale chunk text. Because one formatter serves both initial index and incremental reindex, indexed and re-indexed text cannot drift. Vector store is DuckDB VSS, full stop — no second backend. The index runs in-process with the always-running Provisa server, alongside the meta views and pg_duckdb it derives from, so there is no external vector store to provision. DuckDB's HNSW index is memory-resident (disk persistence is still experimental), so it is treated as a server-lifetime artifact rebuilt/loaded at startup by the same walk-the-meta-views code path as incremental reindex (a cold build is just the full case). Sizing the index scales with schema cardinality, not row volume, and the catalog is curated, so it stays single-node across the range (fed ~10K, enterprise ~140K, largest curated+transaction ~1.4M column chunks) — well inside DuckDB VSS's range, which is why no pgvector/multi-million fallback is needed. Phase 2 (search_catalog semantic explore) is implemented with full rebuild-on-reload; incremental re-embed (keyed off MV/catalog refresh) remains a future optimization.

**Use case:** Makes Provisa pluggable into Claude and other MCP-speaking agents for governed metadata discovery and SQL execution, without exposing a second ungoverned surface. Extends the governed-surface-for-AI-planners motivation of [REQ-884](#REQ-884) from "queryable telemetry" to a first-class agent-facing protocol. Bottom-up search answers detail-topic questions ("which dataset holds the SOFR reset rate") that structured top-down navigation cannot at enterprise catalog scale where the flat table list exceeds an agent's context window.

**Code:** `provisa/pgwire/_pipeline.py`, `provisa/api/flight/catalog.py`, `provisa/api/_meta_views.py`, `provisa/mv/refresh.py`, `provisa/api/mcp/search.py`, `provisa/api/mcp/tools.py`, `provisa/api/mcp/server.py`

**Tests:** `tests/unit/test_mcp_server.py`, `tests/unit/test_mcp_search.py`, `tests/unit/test_mcp_status.py`

## 11. Frontend UI & UX

### REQ-1009 · Component Library Standardization {#REQ-1009}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Standardize provisa-ui on the Mantine component library (MIT/OSS) as the single modern UI component system. Remove unused @mui/material and Emotion dependencies (dead deps, imported in 0 source files).

**Use case:** A single, well-maintained component library provides consistent UX, reduces bundle size, and simplifies maintenance. Mantine's hooks-first architecture aligns with React best practices and TypeScript integration.

**Code:** `provisa-ui/src/`

**Tests:** `tests/unit/test_ui_modernization_requirements.py`

### REQ-1010 · Design System {#REQ-1010}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Establish a consistent visual design theme driven by CSS-variable design tokens, mapped into Mantine's theme. Primary color is indigo (#6366f1); existing App.css token set is the source of truth.

**Use case:** CSS variables enable dynamic theming (light/dark), reduce redundancy, and provide a single point of truth for colors, spacing, and typography across the application.

**Code:** `provisa-ui/src/App.css`

**Tests:** `tests/unit/test_ui_modernization_requirements.py`

### REQ-1011 · Theming {#REQ-1011}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Support both Dark and Light color schemes via Mantine's color-scheme system, with a user-facing theme toggle. Current state is dark-only.

**Use case:** Light mode improves accessibility for users with certain vision conditions; user choice increases accessibility and user satisfaction.

**Code:** `provisa-ui/src/main.tsx`, `provisa-ui/src/theme/ColorSchemeToggle.tsx`

**Tests:** `provisa-ui/e2e/`, `tests/unit/test_ui_modernization_requirements.py`

### REQ-1012 · Internationalization {#REQ-1012}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Internationalize the entire frontend: no hardcoded user-facing strings (~536 exist today). Use react-i18next with an 'en' base catalog; include locale-aware date and number formatting. Pipeline must support adding additional locales.

**Use case:** i18n enables Provisa to serve global enterprises in their native languages, reducing adoption friction and supporting compliance requirements in non-English jurisdictions.

**Code:** `provisa-ui/src/main.tsx`, `provisa-ui/src/i18n/`

**Tests:** `provisa-ui/e2e/`, `tests/unit/test_ui_modernization_requirements.py`

### REQ-1013 · Accessibility {#REQ-1013}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Meet enterprise accessibility requirements: WCAG 2.1 Level AA across the app. Includes focus management and focus-trap in dialogs, full keyboard operability, correct ARIA roles/attributes (e.g. role="dialog"/aria-modal on the 65 modal usages that currently lack them), 4.5:1 text contrast in both color schemes, visible focus indicators, reduced-motion support, form label/error association, and live regions for async notifications.

**Use case:** WCAG 2.1 AA is required for enterprise and government adoption; accessibility is a legal and ethical requirement, not optional.

**Code:** —

**Tests:** `tests/unit/test_ui_modernization_requirements.py`

### REQ-1014 · Accessibility {#REQ-1014}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Accessibility is a tested, enforced requirement: add automated axe-core checks (@axe-core/playwright) into the Playwright e2e coverage fixture so every e2e run asserts zero accessibility violations.

**Use case:** Automated accessibility testing catches WCAG violations early, prevents regressions, and enforces compliance as a gate condition for test passage.

**Code:** `provisa-ui/e2e/coverage.ts`

**Tests:** `provisa-ui/e2e/coverage.ts`, `tests/unit/test_ui_modernization_requirements.py`

### REQ-1016 · Testing {#REQ-1016}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Migrate the provisa-ui test suite (27 Playwright e2e specs + 22 unit/component tests) in lockstep with component modernization ([REQ-1009](#REQ-1009)/[REQ-1012](#REQ-1012)/[REQ-1014](#REQ-1014)). Stop relying on hand-rolled CSS-class selectors (59 locator('.…') uses) and hardcoded literal UI text (58 text-based assertions). Adopt role-based selectors (getByRole) plus stable data-testid attributes on migrated components; text assertions must reference i18n catalog keys (t('key')) rather than literal strings. Migration done per-component in the same pass as that component's Mantine/i18n/a11y conversion, keeping the suite green incrementally.

**Use case:** Mantine migration removes original classNames and i18n moves strings to catalogs, breaking CSS-class and hardcoded-text selectors. Role-based and data-testid strategies are robust to component library changes; i18n-key assertions reflect the requirements contract. Incremental per-component migration prevents big-bang test breakage and ensures continuous test coverage.

**Code:** `provisa-ui/e2e/`, `provisa-ui/src/`

**Tests:** `provisa-ui/e2e/`, `tests/unit/test_ui_modernization_requirements.py`

## 11. Platform, Infrastructure & Delivery

### REQ-1017 · Self-Service Provisioning {#REQ-1017}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Any authenticated Firebase user with no existing org membership may create an org via a non-superadmin endpoint; the creator is inserted into user_org_memberships as role 'admin'.

**Use case:** Enables self-service SaaS adoption without platform admin involvement. Lifts the superadmin gate for the self-serve path only; superadmin endpoints in orgs_router remain for platform administration.

**Code:** `provisa/api/auth_router.py`, `provisa/api/orgs_router.py`

**Tests:** —

### REQ-1075 · SaaS Billing {#REQ-1075}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

SaaS billing is provided by Lemon Squeezy as Merchant of Record. `provisa/api/billing/` integrates over the Lemon Squeezy REST API (JSON:API) and signed webhooks. Checkout: POST /v1/checkouts and returns `checkout.data.attributes.url`; the tenant_id is carried in checkout `custom_data` and echoed back in webhook `meta.custom_data`. Webhook signature verification: HMAC-SHA256 over the RAW request body keyed by env `LEMONSQUEEZY_SIGNING_SECRET`, compared in constant time to the hex `X-Signature` header. Tenant→customer linkage stored as `tenants.ls_customer_id`. Plan lifecycle from `meta.event_name`: `subscription_created`/`subscription_updated` set plan+source_limit from the LS variant name (mapped via PLAN_LIMITS by `plan_from_variant`); `subscription_cancelled`/`subscription_expired` revert to the trial plan. Customer portal URL comes from the LS customer object `data.attributes.urls.customer_portal`. Env: `LEMONSQUEEZY_API_KEY`, `LEMONSQUEEZY_STORE_ID`, `LEMONSQUEEZY_SIGNING_SECRET` (and `LEMONSQUEEZY_BASE_URL` to override the endpoint in tests). An unrecognized variant name is a hard error, never a silent default. This replaces the prior Stripe integration.

**Use case:** Lemon Squeezy MoR consolidates global sales-tax/VAT compliance, chargebacks, and fraud handling in one system Provisa does not build; per-tenant plan gating via source_limit maps to LS subscription variants.

**Code:** —

**Tests:** `tests/unit/test_billing_lemonsqueezy.py`

### REQ-1076 · Federation Configuration {#REQ-1076}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Databricks federation TLS trust is configurable via environment variables to enable the databricks-sql connector to work behind a TLS-intercepting proxy. Honored in precedence order: DATABRICKS_TLS_NO_VERIFY=1 (disable verification, local dev only), DATABRICKS_TLS_CA_FILE (custom CA certificate), REQUESTS_CA_BUNDLE, SSL_CERT_FILE. If all are absent, connector keeps its default verified behavior. Values passed as _tls_trusted_ca_file and _tls_verify to dbsql.connect().

**Use case:** Organizations with TLS-intercepting proxies (compliance/security gateways) require the ability to supply custom certificate authorities or disable verification in development environments. Provisa must support this pattern to work in corporate proxy environments.

**Code:** `provisa/federation/databricks_tls.py`, `provisa/federation/databricks_runtime.py`, `provisa/executor/drivers/databricks.py`

**Tests:** `tests/unit/test_warehouse_config.py`

### REQ-1077 · Warehouse Configuration {#REQ-1077}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

MSSQL warehouse (Azure Synapse/Fabric) login timeout is configurable via PROVISA_MSSQL_LOGIN_TIMEOUT environment variable (seconds, default 120). Applied at both pyodbc connection sites (provisa/federation/mssql_warehouse_runtime.py and provisa/executor/drivers/mssql_warehouse.py) via the ConnectionTimeout keyword argument.

**Use case:** Serverless SQL pools in Azure Synapse auto-pause and require wake-up time before accepting connections. A configurable timeout prevents false connection failures when the pool is resuming from auto-pause.

**Code:** `provisa/federation/mssql_warehouse_runtime.py`, `provisa/executor/drivers/mssql_warehouse.py`

**Tests:** `tests/unit/test_warehouse_config.py`

### REQ-1078 · Testing Infrastructure {#REQ-1078}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

The scripts/test-all runner auto-loads .env file before executing test lanes, with caller-exported variables taking precedence. This allows credential-gated warehouse test lanes (e.g., Databricks, Snowflake, MSSQL) to run instead of skipping.

**Use case:** Enables CI/CD pipelines to load external warehouse credentials from .env (not checked in) and run full warehouse integration tests. Caller-exported variables can still override .env values for CI/CD systems that inject credentials via environment.

**Code:** `provisa/api/billing/router.py`, `provisa/api/billing/lemonsqueezy_client.py`, `provisa/api/billing/models.py`, `provisa/api/billing/tenant_db.py`, `provisa/core/schema_admin.py`

**Tests:** `tests/unit/test_billing_lemonsqueezy.py`, `tests/unit/test_org_isolation.py`

### REQ-1026 · First-Customer Deployment {#REQ-1026}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

First-customer deployment target is a single GCP Compute Engine VM (e2-standard-4: 4 vCPU / 16GB, trimmable to e2-standard-2 / 8GB) running the core docker-compose stack. GCP chosen for colocation with Firebase auth (single vendor/single bill) and one-time $300 free-tier credit.

**Use case:** Trino's ~8GB JVM footprint rules out per-service PaaS (Fly/Railway/Render). A raw VM is the cheapest usable deployment platform. GCP colocation with Firebase minimizes operational complexity and bill consolidation.

**Code:** —

**Tests:** —

### REQ-1027 · First-Customer Deployment {#REQ-1027}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Object storage externalized to Cloudflare R2 (Trino exchange/spill and any object storage), replacing bundled MinIO for first-customer profile. R2 chosen for zero egress fees at low volume.

**Use case:** MinIO adds operational overhead and state complexity. R2's zero-egress pricing aligns first-customer cost to actual usage.

**Code:** —

**Tests:** —

### REQ-1028 · First-Customer Deployment {#REQ-1028}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Trimmed first-customer runtime profile: single-node Trino (coordinator-only, no separate trino-worker; heap reduced to 3-4GB), R2 for exchange, no MinIO. Deliverable: docker-compose.first-customer.yml. The trino-worker service remains defined in compose (unused) for later multi-node scaling without re-architecture.

**Use case:** Reduces memory footprint to fit e2-standard-2 (8GB). Preserves architecture for horizontal scaling when load justifies multi-node deployment.

**Code:** —

**Tests:** —

### REQ-1029 · First-Customer Deployment {#REQ-1029}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Postgres data persists on a separate GCP persistent disk, mounted into the VM, enabling VM resize/reattach without control-plane data loss.

**Use case:** Decouples compute lifecycle from data lifecycle. Allows VM scaling or migration while preserving schema and operational state.

**Code:** —

**Tests:** —

### REQ-1030 · Email Delivery {#REQ-1030}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Email transport for first customer: AWS SES (or equivalent SMTP) at low volume, matching the SMTP-only EmailProvider (REQ-1020).

**Use case:** Removes dependency on GCP-only email services. SMTP provides portable, low-volume email delivery aligned with the pluggable EmailProvider abstraction.

**Code:** —

**Tests:** —

### REQ-1031 · First-Customer Deployment {#REQ-1031}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** infrastructure

Documented lift-and-shift portability to Vultr (flat pricing + bundled bandwidth) as the exit hatch when GCP steady-state cost is no longer preferred. Deployment is plain docker-compose on any VM; only the Firebase project is GCP-pinned, compute is host-agnostic.

**Use case:** Reduces vendor lock-in. Preserves operator freedom to migrate to lower-cost platforms as workload stabilizes and predictable costs justify optimization.

**Code:** —

**Tests:** —

### REQ-1032 · Cost Management {#REQ-1032}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

No cloud infrastructure is provisioned until a customer is signed; pre-customer sunk cost limited to a domain and a free Firebase project. Cold-start relies on [REQ-854](#REQ-854) OVA/Lima bundles for sub-hour spin-up.

**Use case:** Minimizes pre-revenue spending. Defers fixed infrastructure costs until revenue justifies them, reducing financial risk during customer acquisition.

**Code:** —

**Tests:** —

### REQ-1033 · Demo Tiers & Onboarding {#REQ-1033}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Provision two demo tiers on shared first-customer VM: (a) anonymous ephemeral sandbox requiring no signup; (b) persistent quota-limited free tier for Firebase-signed-up users. Both leverage schema-per-org tenancy on shared Trino/PG/Redis infrastructure; no per-user VMs provisioned.

**Use case:** Converts cold-start friction into zero-friction "try it now" landing page. Ephemeral sandbox captures prospects instantly; persistent free tier captures those ready to commit an email. Scales to thousands of free accounts on single shared VM because idle orgs consume only disk (no compute).

**Code:** —

**Tests:** —

### REQ-1034 · Demo Tiers & Onboarding {#REQ-1034}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

On ephemeral sandbox session start, clone a pre-seeded golden template schema (org_template, containing [REQ-414](#REQ-414) demo federated sources and sample data) into a session-scoped org schema (e.g. org_sandbox_<uid>) via the [REQ-697](#REQ-697) init_schema provisioning path.

**Use case:** Pre-seeding the template avoids re-running full data registration and source discovery on every sandbox session. Prospects can immediately explore and create sources, views, and roles without setup lag.

**Code:** —

**Tests:** —

## 2. Authentication & Identity

### REQ-1035 · Demo Tiers & Onboarding {#REQ-1035}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Ephemeral sandbox sessions are scoped by Firebase anonymous auth ([REQ-121](#REQ-121)) without email collection. Prospects get a uid-based session identity with zero friction.

**Use case:** Removes signup friction (email validation, password, ToS acceptance) for the "just looking" funnel. Firebase anonymous auth is stateless and session-scoped, aligning with ephemeral sandbox lifecycle (no user record persists).

**Code:** —

**Tests:** —

## 11. Platform, Infrastructure & Delivery

### REQ-1036 · Demo Tiers & Onboarding {#REQ-1036}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Ephemeral sandbox schemas are dropped on explicit logout OR after an idle-TTL of inactivity. Teardown is self-resetting; a reaper process periodically sweeps expired sandbox schemas.

**Use case:** Ensures sandbox cleanup without manual intervention. Eliminates abandonment debt (orphaned schemas)—each expired sandbox reclaims Postgres and Trino resources immediately. Prospect abandonment is costless.

**Code:** —

**Tests:** —

### REQ-1037 · Demo Tiers & Onboarding {#REQ-1037}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

Ephemeral sandbox sessions are subject to: (a) session-scoped TTL (e.g., 24h); (b) per-session query row limit and query cost caps; (c) data source registration restricted to bundled demo allowlist (no arbitrary outbound live sources). Platform-schema isolation ([REQ-696](#REQ-696)) already guarantees tenant isolation.

**Use case:** Prevents sandbox abuse (runaway queries, scanning arbitrary external data warehouses). Demo allowlist ensures predictable resource footprint and simplifies compliance.

**Code:** —

**Tests:** —

### REQ-1038 · Demo Tiers & Onboarding {#REQ-1038}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Free tier orgs follow a reversible archive lifecycle to reclaim disk and Postgres resources without destructive loss: idle orgs are email-nudged (REQ-1020/1022) before any action; on continued inactivity, pg_dump'd to Cloudflare R2 ([REQ-1027](#REQ-1027)) and live schema dropped; restored on re-login from R2 archive; hard tombstone (delete R2 archive) only after long tail (e.g. 12 months) with final nudge.

**Use case:** Dormant free accounts consume disk only (no idle compute on shared Trino/PG); archiving is disk-pressure-driven, not urgent. Reversible archive (vs. destructive delete) preserves user goodwill and simplifies re-engagement campaigns.

**Code:** —

**Tests:** —

### REQ-1039 · Demo Tiers & Onboarding {#REQ-1039}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Funnel routing policy: "just looking" traffic (cold-start landing page) is routed to anonymous ephemeral sandbox (self-reaping, zero abandonment debt); persistent free tier requires deliberate Firebase signup, minimizing abandoned-account volume on shared infrastructure.

**Use case:** Biases toward ephemeral first-contact (low infrastructure cost, self-cleaning) and requires a signal (email signup) before allocating a persistent schema. Maximizes conversion rate while minimizing accumulation of dormant free accounts.

**Code:** —

**Tests:** —

## 5. Query Languages, Compilation & Operations

### REQ-1040 · Trino Resource Management {#REQ-1040}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** structural

Per-tenant Trino resource groups enforce isolated CPU/memory ceilings and concurrency limits (maxRunning, maxQueued) per org_id, preventing noisy-neighbor contention on the shared cluster. Resource group configuration is generated via trino_setup.py keyed to org tier.

**Use case:** Isolates tenant workloads within a shared Trino cluster. Runaway queries from one org are queued/throttled rather than starving interactive queries from other orgs. Tier-based tuning (free tier gets tight caps; premium gets relaxed limits) enables cost control at the resource-allocation layer.

**Code:** —

**Tests:** —

## 11. Platform, Infrastructure & Delivery

### REQ-1041 · Trino Scaling & HA {#REQ-1041}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** infrastructure

Horizontal Trino worker scale-out path: staged from single-node (coordinator executes, worker replicas=0) -> dedicated workers (include-coordinator=false, trino-worker replicas>0, coordinator remains single) -> k8s HPA on workers autoscaling on CPU/queue depth. Coordinator is never scaled; workers are stateless and horizontally elastic.

**Use case:** Allows cost-effective first-customer single-node deployment (coordinator handles query execution) while providing a clear path to scale workers independently as workload grows. Worker statelesness and HPA integration enable elastic resource response to query volume spikes.

**Code:** —

**Tests:** —

## 5. Query Languages, Compilation & Operations

### REQ-1042 · Trino Resource Management {#REQ-1042}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** structural

Workload separation: interactive user queries and MV/batch-refresh workloads run under distinct Trino resource groups (and eventually distinct worker pools), preventing heavy MV rebuilds from inflating interactive query latency SLO.

**Use case:** MV fleet ([REQ-942](#REQ-942)/966) refresh cycles can spike memory and CPU; isolating batch workloads into a separate resource group with relaxed limits ensures interactive queries maintain predictable tail latency. Distinct worker pools (future) enable physical separation and independent scaling.

**Code:** —

**Tests:** —

## 11. Platform, Infrastructure & Delivery

### REQ-1043 · Multi-Tenancy & Scaling {#REQ-1043}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Tiered cluster routing for physical isolation: sandbox/free tenants route to a shared cluster with tight resource-group caps (best-effort); standard paid tenants route to a better-resourced shared cluster; whale/premium tenants route to a dedicated Trino cluster (or dedicated worker pool) pointed at that org's data sources. Routing is org-tier-aware.

**Use case:** Isolates resource demand: free tier cannot degrade premium SLA. Enables differentiated pricing (shared cluster = included in all tiers; dedicated cluster = premium SKU). Simplifies compliance (isolated VPC, isolated infra, isolated audit logs for high-security tenants).

**Code:** —

**Tests:** —

## 5. Query Languages, Compilation & Operations

### REQ-1044 · Query Cost Control & Monetization {#REQ-1044}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

Hard query-cost caps as a monetization gate: each org tier enforces ceilings on query cost (CPU time, peak memory, bytes scanned, execution time). Queries exceeding the tier's ceiling are REJECTED with an explanatory error message indicating the operation requires a higher tier/premium SKU. Enforced via Trino resource-group limits and per-query guards (query.max-memory-per-node, query.max-execution-time, query.max-scan-physical-bytes) keyed to org entitlement. Free/standard tiers carry restrictive caps; expensive queries require acquiring a premium SKU.

**Use case:** Primary monetization lever: caps are hard rejections, not silent degradation. Prevents free-tier abuse (e.g., scanning the entire data warehouse). Drives tier-upgrade conversion when users hit caps. Project rule: rejection must be explicit, never a silent cap-and-continue.

**Code:** —

**Tests:** —

## 11. Platform, Infrastructure & Delivery

### REQ-1045 · Demo Tiers & Onboarding {#REQ-1045}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

Sandbox and free-tier orgs receive the tightest resource-group and query-cost ceilings, enforced via the same tier-cap mechanism ([REQ-1044](#REQ-1044)) mapped to sandbox/free entitlements. Caps are applied at session provisioning and validated on every query submission.

**Use case:** Limits abuse surface of sandbox/free tiers without manual intervention. Sandbox sessions ([REQ-1034](#REQ-1034)/1035/1036) are especially constrained (query-row limits, data-source allowlist, idle-TTL) to minimize resource drain from abandoned demo sessions. Free-tier orgs receive soft caps that nudge users toward upgrade without forcing abandonment.

**Code:** —

**Tests:** —

### REQ-1046 · Storage Management & Quotas {#REQ-1046}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

Per-tier storage quotas: each org tier enforces a hard ceiling on platform-side storage footprint (Postgres schema bytes for landed/materialized rows, MV output size, and R2 object bytes attributable to the org). Sandbox/free tiers receive the tightest ceilings; paid tiers larger; enforced and metered per org.

**Use case:** Tiered storage model parallels query-cost caps ([REQ-1044](#REQ-1044)) as a primary monetization and resource-isolation lever. Platform storage (Postgres schema data, materialized rows, MV outputs, R2 object bytes) is the metered resource; federation keeps most source data external. Free tiers get tight bounds to prevent abuse accumulation; premium tiers permit larger workloads. Storage must be explicitly capped and enforced at ingestion/materialization time, never silently evicted.

**Code:** —

**Tests:** —

### REQ-1047 · Storage Management & Quotas {#REQ-1047}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Explicit rejection on storage-cap breach: operations that would push an org past its storage ceiling (MV materialization, landing/ingest, result persistence) are REJECTED with an explanatory error indicating a higher SKU or bring-your-own (BYO) storage is required — never silently truncated or evicted.

**Use case:** Mirrors [REQ-1044](#REQ-1044)'s monetization-gate semantics: hard rejection, not silent degradation. Prevents free-tier abuse and drives tier-upgrade conversion. Project rule (CRITICAL): enforcement must be explicit rejection, never a silent cap-and-continue. Error message guides customer toward upgrade path or BYO storage option.

**Code:** —

**Tests:** —

### REQ-1048 · Storage Management & Quotas {#REQ-1048}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** structural

Bring-your-own storage (BYO): premium/enterprise orgs may point their platform-side object storage (MV persistence, Trino exchange/spill, landing/results) at their own S3-compatible bucket (their cloud account, their bill), configured per org. Their bytes never count against platform storage and never reside on platform-owned R2. Org's BYO bucket credentials are scoped to that org only.

**Use case:** Enables premium customers with large or unpredictable storage needs to avoid platform quota limits by supplying their own cloud storage. Isolates security/compliance (customer controls bucket permissions, encryption, audit logging) and cost (customer's infrastructure bill, not platform cost). Reduces platform resource strain from whales/large enterprises.

**Code:** —

**Tests:** —

### REQ-1049 · Storage Management & Quotas {#REQ-1049}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Per-org storage metering/accounting: platform meters storage consumption per org (Postgres schema bytes + attributable R2 object bytes) to drive quota enforcement ([REQ-1046](#REQ-1046)), tier upgrade prompts, and billing.

**Use case:** Storage metering is the storage analog of per-query cost accounting behind [REQ-1044](#REQ-1044). Enables accurate billing per org (platform storage), identifies orgs approaching quota limits (triggering upgrade nudges), and feeds resource-capacity planning (platform storage forecasting). BYO orgs ([REQ-1048](#REQ-1048)) exclude their external bytes from platform metering.

**Code:** —

**Tests:** —

### REQ-1050 · Marketing & Presence {#REQ-1050}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Marketing website surface: a static marketing site (landing, features, pricing) hosted on Cloudflare Pages. Zero marginal cost at any traffic level; deployed independently of the app so it cannot affect app availability. Colocated with the Cloudflare R2 object storage vendor.

**Use case:** Public-facing marketing presence decoupled from app infrastructure. Cloudflare Pages provides global CDN distribution and cost-free hosting (unlimited traffic, automatic DDoS mitigation). Static deployment eliminates operational overhead and risk of app unavailability cascading to marketing presence. R2 colocation reduces cross-zone latency for asset serving.

**Code:** `site/`

**Tests:** —

### REQ-1051 · Documentation & Developer Experience {#REQ-1051}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** structural

Hosted documentation surface: product docs and API reference published on Cloudflare Pages (same deployment or docs subsite), sourced from doc-writer agent output. Static, free-hosted, independent of the app.

**Use case:** Product documentation and API reference must be independently accessible and performant. Cloudflare Pages provides zero-cost, globally distributed static hosting. Decoupling from the app UI ensures docs remain available even during app maintenance or outages, improving developer experience and reducing support burden.

**Code:** —

**Tests:** —

### REQ-1052 · Onboarding & Entry Points {#REQ-1052}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Website-to-SaaS entry-point wiring: the marketing site's primary CTAs (calls-to-action) map to the specified onboarding paths — "Try it now" initiates an anonymous ephemeral sandbox session ([REQ-1034](#REQ-1034)/1036, no signup required); "Sign up free" routes to Firebase self-serve org provisioning ([REQ-1017](#REQ-1017)); "Contact sales" triggers SES/SMTP email delivery to sales (REQ-1020).

**Use case:** Converts marketing interest into activation (sandbox demo, free-tier signup, sales engagement). Clear entry-point routing ensures users experience the intended onboarding flow without friction. Marketing messaging and product capability must align (e.g., "try now" genuinely allows zero-friction exploration; "sign up free" uses self-serve Firebase, not manual approval).

**Code:** —

**Tests:** —

### REQ-1053 · Pricing & Tiering {#REQ-1053}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Pricing page reflects tier entitlements: the public pricing table renders the free/standard/premium tier limits directly from the query-cost caps ([REQ-1044](#REQ-1044)) and storage caps ([REQ-1046](#REQ-1046)) — the same entitlement definitions that gate runtime enforcement drive the marketing pricing presentation, eliminating manual sync.

**Use case:** Single source of truth for tier limits: tier definitions (cost caps, storage quotas) exist once in the runtime entitlement system ([REQ-1044](#REQ-1044)/1046); the pricing page programmatically queries those limits and renders them to users. Eliminates marketing/product drift (manually updated pricing page drifting from actual runtime caps). Ensures all users see accurate tier terms before signup.

**Code:** —

**Tests:** —

### REQ-1054 · Multi-Tenancy & Routing {#REQ-1054}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Per-org wildcard subdomain routing: each org is addressable at {org}.provisa.io via a single wildcard DNS record (*.provisa.io) and wildcard TLS certificate, enabling zero-provisioning self-serve org creation ([REQ-1017](#REQ-1017)).

**Use case:** Eliminates per-org DNS and certificate provisioning overhead. Any newly created org is instantly reachable at its subdomain without manual infrastructure steps, reducing onboarding friction and operational toil.

**Code:** —

**Tests:** —

### REQ-1055 · Multi-Tenancy & Routing {#REQ-1055}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

Fixed per-protocol ports shared across all orgs: HTTP 443 (GraphQL/REST/UI), pgwire 5439, Bolt 7687, Arrow Flight 8480 are shared across all orgs, never varying per org. Org resolution comes from connection parameters (hostname, database name, username, auth principal), not from port multiplexing.

**Use case:** Simplifies port management and connection strings. Standard ports improve discoverability and reduce client configuration complexity. Org is a logical namespace, not a network multiplexing dimension.

**Code:** —

**Tests:** —

### REQ-1056 · Multi-Tenancy & Routing {#REQ-1056}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Per-surface org resolution: HTTP surfaces resolve org from Host subdomain and/or JWT tenant_id claim ([REQ-594](#REQ-594)); pgwire and Bolt (which lack Host headers) resolve org from connection handshake parameters (database name, username, or auth principal).

**Use case:** Accommodates wire protocol constraints (no Host header) while leveraging HTTP metadata where available. Ensures all surfaces can correctly identify the tenant without ambiguity or silent fallback.

**Code:** —

**Tests:** —

### REQ-1057 · Multi-Tenancy & Routing {#REQ-1057}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** infrastructure

Cloudflare wildcard routing: HTTP traffic is proxied via Cloudflare (*.provisa.io -> VM:443); raw-TCP wire protocols are routed via Cloudflare Spectrum (TCP-by-SNI) or a wildcard A record, with org derived from connection parameters.

**Use case:** Leverages Cloudflare's global edge for HTTP caching and DDoS protection. Spectrum enables low-latency TCP passthrough for pgwire and Bolt without origin-side port multiplexing. Single infrastructure path for all protocol surfaces.

**Code:** —

**Tests:** —

## 1. Access Governance & Security

### REQ-1058 · Public Data Publishing {#REQ-1058}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Public-publish anonymous access: unauthenticated requests carry no tenant_id claim; the org is resolved from Host subdomain only, and the request is bound to a built-in read-only 'public' role scoped to that org. TenantMiddleware gains a Host-based resolution path for the no-claim case instead of returning 401.

**Use case:** Enables orgs to publish datasets anonymously without requiring end-user authentication. Public role allows intentional, opt-in data sharing without building separate anonymous infrastructure. Distinct from the Firebase-anonymous sandbox uid ([REQ-1034](#REQ-1034)).

**Code:** —

**Tests:** —

### REQ-1059 · Public Data Publishing {#REQ-1059}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Publishing is a governance grant, not new machinery: each org has a built-in 'public' role starting with ZERO visibility. Publishing an asset means granting that role visibility to a specific table/view; access then flows through existing visibility/RLS/masking ([REQ-039](#REQ-039)/040) uniformly across GraphQL/SQL/pgwire/Bolt. Nothing is public by default; opt-in and explicit.

**Use case:** Reuses existing access-control infrastructure (visibility, RLS, masking) uniformly across all surfaces. Eliminates special-case code for public access. Clear opt-in model prevents accidental data leaks.

**Code:** —

**Tests:** —

### REQ-1060 · Public Data Publishing {#REQ-1060}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

Hard caps on anonymous public traffic: unauthenticated published endpoints are internet-facing with no login gate, so they inherit the tightest rate limits and query-cost caps ([REQ-1044](#REQ-1044)/1045). High-volume public publishing requires a premium public-tier SKU.

**Use case:** Protects shared infrastructure from resource exhaustion by anonymous users. Free-tier public endpoints are rate-limited; high-volume use is gated to premium. Prevents adversarial abuse or accidental DoS by published datasets.

**Code:** —

**Tests:** —

## 11. Platform, Infrastructure & Delivery

### REQ-1061 · Multi-Tenancy & Branding {#REQ-1061}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Subdomain-as-dedicated-channel value proposition: every org gets its own {org}.provisa.io endpoint, conveying dedicated tenancy and branding even though authenticated resolution does not require it.

**Use case:** Improves perceived value and tenant isolation narrative. Org-branded subdomain increases engagement and data ownership perception. Leverages the wildcard infrastructure ([REQ-1054](#REQ-1054)) to provide this for zero per-org cost.

**Code:** —

**Tests:** —

### REQ-1062 · Multi-Tenancy & Branding {#REQ-1062}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Custom domain as a premium gate: premium/enterprise tiers may bind a custom domain (e.g. data.acme.com via CNAME) with custom TLS and white-label (no Provisa branding). Free/standard tiers use {org}.provisa.io only. Tier entitlements are surfaced on the pricing page ([REQ-1052](#REQ-1052)).

**Use case:** Provides a premium branding differentiator. Custom domain + white-label appeals to enterprises embedding data apps in customer-facing portals. Clear tier gate monetizes the feature and simplifies operations (no per-org cert management for premium tenants).

**Code:** —

**Tests:** —

### REQ-1063 · Pricing & Tiering {#REQ-1063}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Usage-credit billing meter: metered consumption is billed as universal compute credits whose consumption is uniform across tiers, with the dollar price per credit rising by tier (bundling features/support) — the Starburst Galaxy model. Built on Stripe billing ([REQ-594](#REQ-594)).

**Use case:** Provides predictable, usage-based billing that aligns with customer consumption patterns. Tier-based credit pricing allows upsells (pay more per credit for premium support/features). Complements hard caps ([REQ-1044](#REQ-1044)/1046) which define ceilings; meter defines pay-as-you-go below them. Industry-proven model (Starburst, Apollo).

**Code:** —

**Tests:** —

### REQ-1064 · Pricing & Tiering {#REQ-1064}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Credit-boxed + time-boxed enterprise trial: a fixed-duration trial (e.g. 30 days) grants a fixed compute-credit balance; on expiry or non-payment, the account auto-downgrades to free tier (never silent charge).

**Use case:** Lowers barrier to enterprise evaluation. Fixed credit budget prevents runaway costs; explicit downgrade on expiry prevents surprise billing and maintains customer trust. Encourages pilot-to-production conversion.

**Code:** —

**Tests:** —

### REQ-1065 · Pricing & Tiering {#REQ-1065}

**Status:** 💡 proposed · **Priority:** MAY · **Type:** behavioral

Signup usage credit: a small usage-credit grant on signup (Apollo $50-style) provides a soft on-ramp to exploration without a time-boxed trial expiration.

**Use case:** Reduces friction for self-serve signup. Free users get compute credits to try real queries; if they exhaust credits, they can upgrade, not just expire. Increases conversion from free to paid by showcasing value through real usage.

**Code:** —

**Tests:** —

### REQ-1066 · Pricing & Tiering {#REQ-1066}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Premium feature gates beyond compute: SSO/SAML + fine-grained access control (ABAC/SCIM, leveraging [REQ-122](#REQ-122)/123/203/204), longer audit/observability retention, private networking, and SLA-backed support are gated to premium tiers.

**Use case:** Monetizes operational and compliance features that enterprises demand. Justifies premium tier beyond compute capacity alone. Clear feature gates drive upsell conversation and meet customer expectations for enterprise offerings.

**Code:** —

**Tests:** —

### REQ-1067 · Pricing & Tiering {#REQ-1067}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Bring-your-own compute / dedicated cluster as premium gate: premium tenants may run on a dedicated Trino cluster (extends tiered routing [REQ-1043](#REQ-1043)) and/or bring their own compute, complementing BYO storage ([REQ-1048](#REQ-1048)).

**Use case:** Provides high-touch enterprise offering with dedicated resources and isolation. Extends tiered routing to allocate premium tenants to reserved clusters. Enables power customers to optimize and control compute independently of Provisa infrastructure. Competitive parity with Starburst Galaxy (dedicated clusters) and Apollo GraphOS (federation customization).

**Code:** —

**Tests:** —

### REQ-1068 · Data Catalog Integration {#REQ-1068}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** structural

Pluggable metadata egress provider pattern: an abstract MetadataEgress interface (mirroring AuthProvider/EmailProvider pattern) enables organizations to publish Provisa's governance metadata OUTBOUND to external data catalogs. Configured per org in YAML with vendor-specific credentials. Outbound only — Provisa never ingests an external catalog as source of truth.

**Use case:** Allows Provisa to act as the authoritative upstream feeding customers' existing data-governance catalogs (OpenMetadata, Atlan, Collibra, DataHub, Apache Atlas), rather than competing with them. Keeps Provisa as runtime enforcement engine while projecting metadata to the customer's catalog of record.

**Code:** `provisa/api/metadata_egress/provider.py`

**Tests:** —

### REQ-1069 · Data Catalog Integration {#REQ-1069}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** structural

Standards-first metadata core: the MetadataEgress provider emits OpenLineage for lineage and maps assets to the OpenMetadata ingestion API as first-class targets. Vendor-specific adapters (Atlan, Collibra, DataHub, Apache Atlas) are implemented as concrete subclasses of MetadataEgress using the same internal metadata model.

**Use case:** Standards-based approach (OpenLineage, OpenMetadata) ensures portability and reduces vendor lock-in. First-class support for OpenMetadata reduces friction for OSS adopters; vendor adapters extend to enterprises already on Atlan or Collibra.

**Code:** —

**Tests:** —

### REQ-1070 · Data Catalog Integration {#REQ-1070}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Published metadata payload includes datasets/tables/columns, domains, stewards and ownership ([REQ-609](#REQ-609)/020), approved relationships, and descriptions/aliases. Lineage is derived from actually-compiled queries and the MV DAG (provisa/lineage/, [REQ-939](#REQ-939)/942), providing column-level + MV-DAG lineage more accurate than scanner/agent-based ingestion.

**Use case:** Provisa's query-compilation and DAG-based lineage is derived from actual execution, making it more authoritative than external scanners or agent-based metadata collection. Publishing this accuracy to external catalogs elevates DG team's source-of-truth quality.

**Code:** —

**Tests:** —

### REQ-1071 · Data Catalog Integration {#REQ-1071}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Governance-signal projection: enforcement facts Provisa already computes (which columns/tables are masked, RLS-restricted, or visibility-restricted per [REQ-039](#REQ-039)/040) are projected outward as tags, classifications, or metadata properties on the corresponding assets in the target external catalog.

**Use case:** Data-governance teams see Provisa's enforced governance policies reflected in their catalog of record, enabling data consumers to understand which data is restricted or transformed without context-switching to Provisa's UI. Strengthens audit and compliance narrative.

**Code:** —

**Tests:** —

### REQ-1072 · Data Catalog Integration {#REQ-1072}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Sync mechanism: metadata changes push to the external catalog event-driven via the [REQ-942](#REQ-942) event substrate. A scheduled full reconcile (using the existing scheduler, provisa/scheduler/jobs.py) periodically re-syncs the complete metadata projection. Per-org configuration and credentials, scoped to that org only.

**Use case:** Event-driven sync ensures near-real-time metadata propagation for operational responsiveness; scheduled full reconcile handles dropped events and corrects drift. Per-org scoping ensures multi-tenant isolation and per-customer credential management.

**Code:** —

**Tests:** —

### REQ-1073 · Data Catalog Integration {#REQ-1073}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** constraint

Premium-gated metadata egress: the metadata egress connector is an enterprise/premium tier entitlement, extending [REQ-1066](#REQ-1066). Targeting regulated-industry data-governance teams with existing external catalog investments.

**Use case:** Metadata egress to external catalogs is a high-touch enterprise feature. Restricted to premium tiers to simplify support and monetize integration value to enterprises already operating external DG systems.

**Code:** —

**Tests:** —

## 10. Admin Console & Governance

### REQ-1079 · Admin Configuration {#REQ-1079}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin UI can view and edit the security posture (security.mode: standard | high) via GET/PUT /admin/security, with changes applied on restart.

**Use case:** Enables operators to adjust security enforcement level without code deployment.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`, `provisa-ui/src/__tests__/SecurityTab.test.tsx`

### REQ-1080 · Admin Configuration {#REQ-1080}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin UI can view and edit AI model assignments (ai_models), the embedding/vector-model registry (vector_models), and the natural-language rate limit (nl.rate_limit) via GET/PUT /admin/ai-models, with changes applied on restart.

**Use case:** Allows dynamic model and rate-limit tuning without redeployment.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`, `provisa-ui/src/__tests__/AiModelsTab.test.tsx`

### REQ-1081 · Admin Configuration {#REQ-1081}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin UI cache-storage settings now also cover warm-tier promotion and engine filesystem read-cache (warm_tables), plus the default materialized-view refresh TTL (materialized_views.default_ttl).

**Use case:** Centralizes cache and materialized-view refresh configuration in the admin console.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`

### REQ-1082 · Admin Configuration {#REQ-1082}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin UI observability settings expose the full OpenTelemetry tracing-pipeline tuning surface (log_level, compact_cron, compact_batch_size, compact_file_chunk, ops_snapshot_retention_hours, span_export_delay_millis, otlp2parquet_max_age_secs, collector_batch_timeout_ms, s3_endpoint).

**Use case:** Operators can fine-tune telemetry collection and export without redeployment.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`

### REQ-1083 · Admin Configuration {#REQ-1083}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin UI platform settings now expose remote-GraphQL traversal limits (graphql_remote.max_object_depth, max_list_depth, max_list_items) and make the default query sample size editable.

**Use case:** Operators can constrain GraphQL complexity and adjust sampling without redeployment.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`

### REQ-1084 · Admin Configuration {#REQ-1084}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

The federation-engine tab now exposes S3 exchange-spool fields (exchange_spool_s3_endpoint, exchange_spool_bucket, exchange_spool_s3_region, exchange_spool_s3_access_key, exchange_spool_s3_secret_key).

**Use case:** Enables S3-backed exchange-spool configuration in the admin UI.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`

### REQ-1085 · Admin Navigation {#REQ-1085}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin navigation consolidates security posture, encryption, authentication, and local users under a single Security section with sub-tabs.

**Use case:** Improves discoverability and grouping of security-related admin settings.

**Code:** —

**Tests:** `provisa-ui/src/__tests__/SecurityTab.test.tsx`, `provisa-ui/src/__tests__/EncryptionTab.test.tsx`, `provisa-ui/src/__tests__/LocalUsersTab.test.tsx`

## 15. Proposed Features

### REQ-1086 · Encryption {#REQ-1086}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Extensible encryption provider registry (provisa/encryption/registry.py) with pluggable implementations. Each provider is a spec with UI config_fields, an availability probe, and a builder. build_encryption_service dispatches via the registry and fails closed on unknown/unavailable providers.

**Use case:** Decouples provider implementation from core service, enabling built-in (LocalKeychain, AwsKms, AzureKeyVault, Vault) and custom providers (via env var, entry-point, or API) to coexist without code changes.

**Code:** `provisa/encryption/registry.py`

**Tests:** `tests/unit/test_encryption_providers.py`

### REQ-1087 · Encryption {#REQ-1087}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

AWS KMS encryption provider (envelope wrapping via KMS Encrypt/Decrypt) with support for custom endpoint_url for KMS-compatible private endpoints. Available whenever boto3 is present.

**Use case:** Enterprises can use AWS-managed KMS keys without re-implementing envelope encryption logic. Private endpoint support enables airgapped deployments.

**Code:** `provisa/encryption/providers/aws_kms.py`

**Tests:** `tests/unit/test_encryption_providers.py`

### REQ-1088 · Encryption {#REQ-1088}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

HashiCorp Vault (Transit engine) encryption provider with auto-availability when hvac package is installed. Vault address is configurable as a custom endpoint.

**Use case:** Enterprises using Vault can leverage existing HSM or encryption backends without provisioning cloud KMS. Custom Vault endpoint supports private/airgapped deployments.

**Code:** `provisa/encryption/providers/vault.py`

**Tests:** `tests/unit/test_encryption_providers.py`

### REQ-1089 · Encryption {#REQ-1089}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

Azure Key Vault encryption provider (RSA-OAEP-256 key wrap) with auto-availability when azure-keyvault-keys package is installed.

**Use case:** Azure-native deployments can leverage customer-managed keys in Azure Key Vault without running separate KMS infrastructure.

**Code:** `provisa/encryption/providers/azure_keyvault.py`

**Tests:** `tests/unit/test_encryption_providers.py`

### REQ-1090 · Encryption {#REQ-1090}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Enterprises can register custom encryption providers (e.g., in-house KMS/HSM endpoints) without core changes via register_encryption_provider() API, PROVISA_ENCRYPTION_PROVIDER_MODULES environment variable, or provisa.encryption_providers entry-point group.

**Use case:** Organizations with proprietary KMS or HSM deployments can integrate them without maintaining a Provisa fork.

**Code:** `provisa/encryption/registry.py`

**Tests:** `tests/unit/test_encryption_providers.py`

## 10. Admin Console & Governance

### REQ-1091 · Admin Configuration {#REQ-1091}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Admin encryption tab derives its provider list live from the encryption registry. Built-in and custom providers appear automatically with their declared config fields. Unavailable providers are shown but not selectable.

**Use case:** Operators can configure encryption without code changes; custom providers are discoverable in the UI without redeploy.

**Code:** —

**Tests:** `tests/unit/test_admin_config_tabs.py`, `provisa-ui/src/__tests__/EncryptionTab.test.tsx`

## 11. Frontend UI & UX

### REQ-1092 · Catalog Explore {#REQ-1092}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** ui

Provisa provides an LLM chatbot at /explore (under the Explore nav group, branded as "Assistant") where users converse in natural language to explore and query the federated catalog. The interface is built with @chatscope/chat-ui-kit-react. A Claude model (default claude-opus-4-8, configurable via ai_models.mcp_chat) drives a manual tool-use loop with governed MCP tools—list_schemas, list_tables, describe_table, search_catalog, run_sql, explain_sql—via provisa/api/mcp/chat.py. Every tool executes under the caller's fixed role (from x-provisa-role), so the assistant respects domain access + governance boundaries; the model cannot choose or escalate the role. The backend streams assistant text and tool activity to the browser as Server-Sent Events via POST /admin/mcp/chat.

**Use case:** Users can explore and query the federated catalog through conversational AI, discovering tables/columns and running SQL through natural language without schema knowledge, constrained by their role's visibility. The assistant is bound by the same governed pipeline as any client.

**Code:** `provisa/api/mcp/chat.py`, `provisa/api/mcp/status.py`, `provisa-ui/src/pages/McpExplorePage.tsx`

**Tests:** `tests/unit/test_mcp_chat.py`

## 3. Source Registration & Data Modeling

### REQ-1093 · Unique Constraints {#REQ-1093}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A registered Table carries a table-level list of UNIQUE constraints (`unique_constraints: [{name, columns[]}]`), each an ordered column set that may be single-column or composite. On registration Provisa introspects declared UNIQUE constraints from RDB sources (PostgreSQL/MySQL information_schema.table_constraints + key_column_usage grouped by constraint_name and ordered by ordinal_position; SQLite PRAGMA index_list/index_info with origin='u') and seeds the list. Uniqueness is only recorded from a source-declared constraint, never inferred from sampled data. The admin UI exposes an expandable "Uniques" panel in the register/edit table forms: a "+ Constraint" control adds a row with a Name text field and a multi-select checkbox of the table's columns. pgwire emits each unique constraint as a pg_constraint row with contype 'u' (conkey = ordered attnums, no confrelid) and projects it into information_schema.table_constraints (constraint_type 'UNIQUE') and key_column_usage, so standard clients (DBeaver, psql \d) render the keys. The MCP describe_table tool includes the table's unique_constraints alongside foreign_keys, role-gated.

**Use case:** Foreign keys can legitimately target any unique key, not just the primary key; surfacing declared unique constraints improves relationship inference, lets the query planner reason about row identity, gives standard SQL clients and the MCP agent the natural keys users actually join and filter on, and makes the ERD/catalog show natural keys instead of only surrogate PKs.

**Code:** `provisa/core/models.py`, `provisa/discovery/fk_introspect.py`, `provisa/pgwire/catalog_constraints.py`, `provisa/api/mcp/tools.py`, `provisa/api/admin/discovery_schema.py`, `provisa/compiler/context.py`, `provisa-ui/src/components/admin/UniquesPanel.tsx`, `provisa-ui/src/pages/tables/RegisterTableForm.tsx`, `provisa-ui/src/pages/tables/TableEditForm.tsx`

**Tests:** `tests/unit/test_unique_introspect.py`, `tests/unit/pgwire/test_unique_constraints.py`, `tests/unit/test_unique_constraints_wiring.py`, `provisa-ui/src/components/admin/__tests__/UniquesPanel.test.tsx`

## 8. Client Access & Protocols

### REQ-1094 · PostgreSQL Catalog Compatibility {#REQ-1094}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

Composite foreign keys are reconstructed at pgwire catalog-build time from the FK→PK invariant. Provisa's relationship model stores each FK column as a separate join with no shared constraint ID; single-column many-to-one joins between the same source/target table pair whose target columns together form the target table's composite primary key are collapsed into one pg_constraint FK row (contype 'f') with multi-element conkey/confkey arrays, ordered by target PK column order. Joins not covering a composite PK (single-column FKs, partial coverage, independent multi-FKs to the same table) remain one single-column row each.

**Use case:** PostgreSQL clients and tools (e.g., DataGrip, psql) expect composite FKs as single pg_constraint rows with parallel conkey/confkey arrays; reconstructing them ensures correct relationship visualization and validation logic without requiring a V1-prohibited migration.

**Code:** `provisa/pgwire/catalog_constraints.py`

**Tests:** `tests/unit/pgwire/test_composite_fk_and_index.py`

### REQ-1095 · PostgreSQL Catalog Compatibility {#REQ-1095}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

pg_catalog.pg_index is populated with one row per primary-key and UNIQUE constraint (indrelid = table oid, indkey = ordered key attnums, indisprimary/indisunique flags set). This allows PostgreSQL clients (e.g., DataGrip) that resolve key columns via pg_index.indkey rather than pg_constraint to discover the correct columns via the standard pg_index → pg_attribute join.

**Use case:** Standard SQL clients resolve primary key and unique constraint columns by joining pg_index to pg_attribute on indkey; populating pg_index ensures clients that use this path (rather than pg_constraint) get correct key column visibility and metadata.

**Code:** `provisa/pgwire/catalog_constraints.py`

**Tests:** `tests/unit/pgwire/test_composite_fk_and_index.py`

## 10. UI & Admin Surfaces

### REQ-1096 · Admin & Configuration {#REQ-1096}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Live config export/diff/patch — an OPT-IN admin capability, distinct from the plain download/upload of [REQ-164](#REQ-164), that lets an admin see and export exactly what changed since startup. The on-disk config is only the boot seed; admin mutations write to the control-plane DB, so the file goes stale (an MV or relationship created in the UI, or a changed primary key, never appears in it). When enabled, Provisa rebuilds the DB-backed config sections (tables incl. views/MVs, relationships, roles, rls_rules, domains) from live state and overlays them on the file base to produce the CURRENT config. The baseline for diff/patch is a BOOT SNAPSHOT captured ONCE at the end of boot — after all runtime auto-derivation (FK tracking, graphql-remote registration) — so the diff shows only changes made SINCE startup, not derived entities that were never in the file. This is the "lock the starting config after automated changes" behavior: the automated/derived setup is frozen as the baseline, and every subsequent admin edit shows as a clean diff against it. Both sides are normalized identically (deep key-sort + stable entity-sort) so section/key/entity ORDER never differs and the diff surfaces only real changes. The admin UI (Configuration File section) exposes: View Diff (a two-pane diff of baseline vs current, editable), Download Patch (a git-apply/patch-compatible unified diff from baseline to the curated current, for CI/CD reproduction), and Apply Revised (upload the curated config and reload). When the flag is on, uploaded config is normalized on consume and persisted in normalized form so the on-disk file stays byte-faithful to the patch baseline; when off, the file is written verbatim to preserve a hand-authored file's comments/ordering. The feature is coherent only when the generated/normalized config is canonical (config built from installer/demo choices), NOT a hand-authored file whose comments/ordering a normalized patch could not stay faithful to — so it is GATED behind the live_config_export flag (config key live_config_export: true OR env PROVISA_LIVE_CONFIG_EXPORT in {1,true,yes}), exposed to the UI as features.live_config_export, which conditionally renders the View Diff / patch controls. In DEMO MODE the flag MUST be ON: the demo is the canonical-generated-config scenario, so demo builds force live_config_export regardless of an explicit config key.

**Use case:** Admins running the canonical/demo config need to see exactly what they changed in the UI since startup and export it as a patch for version control or CI/CD, instead of hand-diffing a stale on-disk file that never reflected DB-backed edits. Locking the boot snapshot after automated derivation keeps the diff free of noise. Demo users get the diff/patch surface out of the box because the demo config is generated and therefore always safe to normalize.

**Code:** `provisa/api/app.py`, `provisa/api/app_startup.py`, `provisa/api/admin/config_export.py`, `provisa/api/admin/settings_router.py`, `provisa-ui/src/pages/AdminPage.tsx`, `provisa-ui/src/components/admin/ConfigDiffView.tsx`

**Tests:** `tests/unit/test_config_export.py`, `tests/integration/test_settings_router_api.py`

## 7. Connectors & Source Adapters

### REQ-1097 · Test Coverage {#REQ-1097}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Every datasource type in SourceType enum (49 types: postgresql, mysql, mariadb, sqlserver, oracle, cockroachdb, yugabytedb, tidb, greenplum, singlestore, snowflake, bigquery, databricks, fabric, synapse, redshift, clickhouse, elasticsearch, pinot, druid, exasol, delta_lake, iceberg, hive, hive_s3, mongodb, cassandra, redis, firebird, duckdb, sqlite, openapi, graphql_remote, grpc_remote, govdata, kafka, neo4j, sparql, prometheus, rss, websocket, sharepoint, splunk, google_sheets, csv, parquet, files, airport, ingest) must have an integration or e2e test against a real (Docker or cloud-provisioned) instance. Unit tests that merely reference the type name do not satisfy this.

**Use case:** Ensures all datasource connectors work correctly in practice against real instances, not just as concept or identity-check in code. Covered with self-provisioning tests: direct-driver — mariadb, tidb, cockroachdb, yugabytedb, sqlserver, oracle, singlestore (license-gated skip); connector (federation engine → Trino catalog) — cassandra (reference pattern), exasol (arch-gated: amd64-only image), redshift + sharepoint (creds/instance-gated, no self-provision); DuckDB community extension — firebird, airport (live vs zaychik), google_sheets (opt-in gated: DuckDB gsheets extension SA-auth, PROVISA_GSHEETS_LIVE=1); Calcite/Trino — splunk (live, self-provisioned). Fixed a product bug found while covering exasol: Source.jdbc_url() omitted exasol/redshift, silently no-op'ing catalog creation. Filed issue #74 (zaychik Arrow-Flight auth/protocol gap, found via airport; the Provisa Flight→Trino transport itself is verified working — tests/integration/test_trino_flight_engine_e2e.py). The heavy multi-container analytics/lake ecosystems are now covered by self-provisioning e2e: pinot and druid (Trino catalog), hive and hive_s3 (Trino hive connector + HMS, S3 on MinIO). kudu and accumulo were removed from SourceType — neither is a SQL-variant with a downloadable driver testable in this harness (Trino 481 ships no kudu/accumulo connector).

**Code:** `provisa/core/models.py`, `provisa/federation/trino_connectors.py`

**Tests:** `tests/integration/test_mariadb_source_e2e.py`, `tests/integration/test_tidb_source_e2e.py`, `tests/integration/test_cockroachdb_source_e2e.py`, `tests/integration/test_yugabytedb_source_e2e.py`, `tests/integration/test_greenplum_source_e2e.py`, `tests/integration/test_sqlserver_source_e2e.py`, `tests/integration/test_oracle_source_e2e.py`, `tests/integration/test_singlestore_source_e2e.py`, `tests/integration/test_cassandra_source_e2e.py`, `tests/integration/test_firebird_source_e2e.py`, `tests/integration/test_exasol_source_e2e.py`, `tests/integration/test_redshift_source_e2e.py`, `tests/integration/test_sharepoint_source_e2e.py`, `tests/integration/test_google_sheets_source_e2e.py`, `tests/integration/test_airport_source_e2e.py`, `tests/integration/test_splunk_source_e2e.py`, `tests/integration/test_pinot_source_e2e.py`, `tests/integration/test_druid_source_e2e.py`, `tests/integration/test_hive_source_e2e.py`, `tests/integration/test_hive_s3_source_e2e.py`, `tests/integration/test_trino_flight_engine_e2e.py`

## 11. Platform, Infrastructure & Delivery

### REQ-1098 · Native Tier Database {#REQ-1098}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

The native (DuckDB) tier exposes the tenant control-plane as the `provisa_admin` catalog, enabling meta/ops entities (registered_tables, etc.) to be queryable with parity to the Trino tier. The catalog attaches the tenant SQLite DB READ_ONLY and exposes every table under the compiler-emitted schema `org_<id>` without a hardcoded table list.

**Use case:** Provides full parity with Trino tier where provisa_admin is a real attached catalog. Allows native users to query system metadata and operational state via standard SQL, matching Trino/Presto multi-catalog semantics.

**Code:** `provisa/federation/duckdb_runtime.py`, `provisa/federation/native_backend.py`

**Tests:** `tests/unit/test_duckdb_control_plane_attach.py`

### REQ-1099 · Native Tier Database {#REQ-1099}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The control-plane SQLite (platform + tenant) MUST run in WAL mode (journal_mode=WAL) with busy_timeout enabled. This is the concurrency guarantee that permits the native DuckDB engine to read the tenant DB READ_ONLY while aiosqlite writes config changes without transient "database is locked" errors.

**Use case:** Enables concurrent read-write access to the control-plane SQLite without lock contention or stalling queries. Required for safe multi-client scenarios where the UI (aiosqlite) updates config while the native engine queries system state.

**Code:** `provisa/core/database.py`

**Tests:** `tests/unit/test_admin_database.py`

### REQ-1100 · Desktop Installation {#REQ-1100}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

The native demo launcher MUST wait for the demo mock servers (petstore :18080, graphql :4000) to answer before starting the API. The launcher MUST gate the browser-open on the API's /health endpoint (served only after lifespan/config-load completes), not merely on the UI proxy port.

**Use case:** Ensures the demo environment is fully initialized before opening the browser to the user. Prevents premature browser navigation to a not-yet-ready environment and reduces failed first-user experience.

**Code:** `packaging/windows/provisa-native.ps1`

**Tests:** `tests/unit/test_desktop_install.py`, `packaging/windows/tests/provisa-native.Tests.ps1`

### REQ-1101 · Desktop MCP Configuration {#REQ-1101}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Native/desktop tier enables remote MCP server by default, bound to loopback (PROVISA_MCP_HOST=127.0.0.1), with the governed role pinned explicitly (native auth:none = one local admin). All settings overridable via environment variables.

**Use case:** Provides secure local MCP connectivity for Claude Desktop on native installations without additional configuration, enabling tool use against local Provisa queries.

**Code:** `provisa/api/mcp/server.py`, `packaging/windows/provisa-native.ps1`, `scripts/provisa`

**Tests:** `tests/unit/test_mcp_status.py`

### REQ-1102 · MCP Status Endpoint {#REQ-1102}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The MCP status endpoint returns a resolved, proxy-aware, editable connect URL derived from X-Forwarded-Host/Host headers plus PROVISA_MCP_PORT and /mcp path; PROVISA_MCP_EXTERNAL_URL overrides. Returns null when MCP is disabled.

**Use case:** Enables UI to display and allow editing of the MCP connection URL, accounting for reverse proxies and external domains that the server cannot reliably infer.

**Code:** `provisa/api/mcp/status.py`

**Tests:** `tests/unit/test_mcp_status.py`

### REQ-1103 · Explore MCP UI Panel {#REQ-1103}

**Status:** ✅ complete · **Priority:** MUST · **Type:** ui

The Explore/MCP surface provides a "Connect Claude Desktop" panel that emits a copy-ready claude_desktop_config.json entry with URL prefilled and editable.

**Use case:** Simplifies Claude Desktop setup by providing a complete, copy-paste configuration entry that users can add to their Claude Desktop config file.

**Code:** `provisa-ui/src/pages/McpExplorePage.tsx`

**Tests:** `tests/unit/test_mcp_status.py`

### REQ-1104 · Claude Desktop Bridge {#REQ-1104}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

The Claude Desktop connector is Node-free: mcp-proxy (Python stdio-to-Streamable-HTTP bridge) is bundled into the native standalone runtime. The emitted config's command references the bundled interpreter (PROVISA_MCP_BRIDGE_COMMAND). Feature is gated to native tier only; off-tier the panel degrades to "install mcp-proxy yourself". No npx, no Node, no user pip, fully airgapped.

**Use case:** Eliminates Node.js and npm as dependencies for Claude Desktop MCP connectivity in native installations, reducing distribution complexity and supporting offline/airgapped environments.

**Code:** `packaging/windows/build-sfx.ps1`, `packaging/macos/build-dmg.sh`, `pyproject.toml`, `provisa/api/mcp/status.py`, `packaging/windows/provisa-native.ps1`, `scripts/provisa`

**Tests:** `tests/unit/test_mcp_status.py`

### REQ-1105 · MCP Authentication {#REQ-1105}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Wire resolve_token_role into the FastMCP Streamable HTTP transport to map remote bearer tokens to Provisa roles, enabling secure per-caller MCP authentication before exposing MCP off the loopback.

**Use case:** Prerequisite for safely exposing MCP to remote callers with proper role-based access control and governance.

**Code:** `provisa/api/mcp/server.py`

**Tests:** `tests/unit/test_mcp_bearer_role.py`

### REQ-1106 · MCP Transport {#REQ-1106}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

MCP local-connector transport supports plain HTTP by default with optional TLS (PROVISA_MCP_TLS=1 for public deployment). Per-machine self-signed certificates are generated on first start with OS user-store trust and http fallback if cert creation fails. When MCP binds non-loopback (0.0.0.0), DNS-rebinding Host-header protection is disabled (stdio bridges are not subject to browser-origin attacks); loopback binds retain strict protection.

**Use case:** Enables local Provisa MCP servers to be reached by Claude Desktop via mcp-proxy stdio bridge without requiring bundled TLS. Supports optional TLS for publicly-exposed deployments while fixing 421 Misdirected Request errors when clients connect via the machine's real IP (e.g. WSL VM IP).

**Code:** `provisa/api/mcp/server.py`, `provisa/api/mcp/tls.py`, `provisa/api/mcp/status.py`

**Tests:** `tests/unit/test_mcp_status.py`

## 8. Client Access & Protocols

### REQ-1120 · Served Protocol Surface {#REQ-1120}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Provisa serves the DuckDB Airport Arrow-Flight protocol as an outbound data-as-a-service surface, allowing external DuckDB clients with the `airport` community extension to ATTACH Provisa (TYPE AIRPORT) and query its governed federated catalog with server-side governance (RLS/masking) applied.

**Use case:** DuckDB users can integrate Provisa's federated catalog directly into DuckDB workflows without separate ETL, leveraging DuckDB's in-process analytics while Provisa enforces row-level and column-level access control server-side. This peer protocol surface complements existing bolt (Neo4j), pgwire (Postgres), and Arrow Flight SQL surfaces.

**Code:** `provisa/api/airport/server.py`, `provisa/api/airport/wire.py`, `provisa/api/airport/query.py`, `provisa/api/airport/service.py`, `provisa/api/app_startup.py`

**Tests:** `tests/integration/test_airport_service_e2e.py`

### REQ-1121 · Query Pushdown & Optimization {#REQ-1121}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The DuckDB airport extension delivers filter and projection pushdown via the endpoints DoAction parameters map (json_filters JSON expression tree + column_ids indices). The server translates these to semantic WHERE + projection, folds them into the DoGet ticket, and injects them through _govern_and_route so sources apply filters/projection with RLS/masking layered on.

**Use case:** Pushdown minimizes network transfer and leverages source query optimization while maintaining governance invariants. Conformance rule: DoGet must return the FULL advertised schema; projected-out columns are null-filled client-side and DuckDB projects accordingly.

**Code:** `provisa/api/airport/pushdown.py`, `provisa/api/airport/server.py`

**Tests:** `tests/integration/test_airport_service_e2e.py`

### REQ-1122 · Access Control & Projection {#REQ-1122}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Airport catalog scan projects only registered semantic columns per role, never SELECT *. Unregistered physical columns (e.g. audit created_at) are neither leaked on read nor sent as NULL on INSERT.

**Use case:** Ensures column-level access control is enforced at the protocol boundary and physical schema details remain hidden from federated consumers, preventing accidental data leakage or schema coupling.

**Code:** `provisa/api/airport/service.py`, `provisa/api/airport/query.py`

**Tests:** `tests/integration/test_airport_service_e2e.py`

### REQ-1123 · Transaction Management {#REQ-1123}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Airport transactions use a real in-memory coordinator (provisa/api/airport/transactions.py). create_transaction mints a UUID identifier; get_transaction_status reports active/committed/aborted. Best-effort read-committed isolation: each governed mutation auto-commits.

**Use case:** Provides transaction identity and status tracking for DuckDB clients performing multi-statement workflows against a governed federated catalog without requiring persistent transaction state across connections.

**Code:** `provisa/api/airport/transactions.py`, `provisa/api/airport/service.py`

**Tests:** `tests/integration/test_airport_service_e2e.py`

### REQ-1124 · Mutation Enforcement {#REQ-1124}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Governed DML (INSERT / UPDATE / DELETE) via do_exchange decodes the Arrow stream and submits semantic SQL through _compile_govern_execute (the single governed write pipeline), gated on engine writability (is_writable_on / writable.py). Never parallel writers. UPDATE/DELETE key rows by the table's PRIMARY KEY, advertised as the airport row identity (is_rowid pseudo-column filled with the JSON-encoded PK tuple), so the WHERE is formed only from PKs RLS already let the role read and is RLS-filtered again through the pipeline — a role can never mutate a row it cannot see (the attempt is a governed no-op).

**Use case:** Ensures all mutations flow through the unified governance pipeline with consistent RLS/masking and audit enforcement, preventing bypass via direct insertion or uncontrolled parallel writes.

**Code:** `provisa/api/airport/service.py`, `provisa/api/airport/query.py`, `provisa/api/airport/server.py`

**Tests:** `tests/integration/test_airport_service_e2e.py`

### REQ-1125 · Protocol Compliance {#REQ-1125}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Operations refused with a correct Flight protocol error (never a silent no-op). UPDATE/DELETE ARE supported via PRIMARY-KEY row identity: the table's PK is advertised as the airport row identity (is_rowid pseudo-column filled with the JSON-encoded PK tuple) so DuckDB echoes it back on update/delete; the server forms a governed WHERE PK IN (...) submitted through the one write pipeline (RLS/masking/writable-ACL apply). They are refused ONLY per-table, when a table has no primary key (no safe row identity). create_table IS supported: it creates the physical table in the target domain's single writable source (schema-mutation pipeline) and registers it, so the new table joins the catalog and accepts a governed INSERT; refused only when the domain has no writable source bound (or maps to more than one — ambiguous). column_statistics + table_function_flight_info are refused (governed federation has no precomputed stats and advertises no table functions). Physical column/struct ALTERs (add_column/rename_table/add_field/ …) are refused (they need the admin schema-mutation API's source/governance context absent from the airport DDL payload). DDL create_schema/drop_schema map to Provisa domains via the control-plane domain repository.

**Use case:** Maintains protocol integrity and a consistent governance model: safe, governed operations (PK UPDATE/DELETE, create_table into a writable source) are served through the one pipeline, while operations that cannot be safely federated are rejected — preventing data inconsistency, uncontrolled mutations, or governance bypass.

**Code:** `provisa/api/airport/server.py`, `provisa/api/airport/query.py`

**Tests:** `tests/integration/test_airport_service_e2e.py`

## 9. Deployment & Distribution

### REQ-1126 · PyPI Distribution {#REQ-1126}

**Status:** ✅ complete · **Priority:** MUST · **Type:** infrastructure

Provisa ships as a pip-installable wheel via `pip install provisa[embedded]`. The wheel contains a fully functional embedded Provisa system (SQLite control plane + embedded DuckDB engine) ready to run immediately: `provisa run`. Supply-chain trust follows existing Artifactory-as-PyPI models already approved and CVE-scanned by regulated enterprises.

**Use case:** Airgapped/regulated enterprises already trust Artifactory-as-PyPI supply chains (whitelisted, CVE-scanned, hash-pinned via uv.lock) but cannot approve new registries (Docker Hub, notarization services, custom mirrors). Wheel distribution avoids new approval gates.

**Code:** `pyproject.toml`, `provisa/cli.py`, `scripts/build-wheel.sh`

**Tests:** `tests/e2e/test_pypi_packaging.py`

### REQ-1127 · UI Packaging {#REQ-1127}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

React UI is precompiled into the wheel. CI runs `vite build` (from provisa-ui/package.json), embeds the output into the package tree (e.g. provisa/_ui/), and declares it in [tool.setuptools.package-data] so `python -m build` includes it. ui_server.py serves from the packaged directory path rather than __file__.parent.parent/static, eliminating the need for npm/Node at runtime or post-install. The runtime config/ files are staged into provisa/_config/ the same way so the wheel is fully self-contained.

**Use case:** Airgapped environments have no npm/Node installed. Packaging the precompiled UI into the wheel ensures a self-contained delivery that can be deployed to systems with only Python >=3.12.

**Code:** `provisa/ui_server.py`, `provisa/core/desktop_profile.py`, `provisa/pg_extensions/catalog.py`, `scripts/build-wheel.sh`, `pyproject.toml`

**Tests:** `tests/e2e/test_pypi_packaging.py`

### REQ-1128 · Console Entry Point {#REQ-1128}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** structural

A `provisa` console entry point (provisa = provisa.cli:main) is added to [project.scripts] in pyproject.toml, allowing users to launch the system via the `provisa run` command at the shell without explicitly calling a module or uvicorn.

**Use case:** Improves user experience and follows Python packaging conventions; users expect a single command (`provisa run`) rather than python -m invocations.

**Code:** `provisa/cli.py`, `pyproject.toml`

**Tests:** `tests/e2e/test_pypi_packaging.py`

### REQ-1129 · Embedded Profile Dependencies {#REQ-1129}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The [embedded] optional dependency extra pins SQLite control plane + embedded DuckDB engine ([REQ-979](#REQ-979): native/desktop tier). The bundled Trino JVM is omitted (PyPI cannot carry a JVM server). Full multi-engine federation is preserved by pointing to a customer-provided ("bring your own") external engine via TRINO_HOST/TRINO_PORT env or federation_engine_host/port config (already supported: provisa/federation/engine.py:1201, provisa/core/desktop_profile.py:153). Embedded DuckDB is the default when no external engine is configured.

**Use case:** Defines a complete, runnable tier for airgapped enterprises while preserving federation capability via bring-your-own external engines. Omitting the JVM avoids GPL licensing complications and PyPI size/upload limits.

**Code:** `provisa/federation/engine.py`, `provisa/core/desktop_profile.py`, `pyproject.toml`

**Tests:** `tests/e2e/test_pypi_packaging.py`

### REQ-1130 · Mirror Preparation {#REQ-1130}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** constraint

Python language pin: >=3.12,<3.13. Airgap Artifactory mirrors must pre-seed cp312 platform wheels for all native-wheel dependencies: duckdb, chdb, pyodbc, pyarrow, grpcio, cryptography, bcrypt, psycopg2-binary, snowflake-connector-python, etc. A requirements.lock and "mirror these wheels" checklist is delivered so the customer's Artifactory admin can prepare in one pass.

**Use case:** Regulated enterprises run PyPI mirrors (Artifactory, Nexus) with whitelisted wheels. Pre-seeding the checklist avoids deployment surprises and allows the admin to verify all transitive deps are approved before go-live.

**Code:** `pyproject.toml`, `packaging/pypi/requirements-embedded.lock`, `packaging/pypi/MIRROR-CHECKLIST.md`

**Tests:** `tests/e2e/test_pypi_packaging.py`

## 1. Access Governance & Security

### REQ-1131 · Column Visibility Control {#REQ-1131}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A role with ADMIN capability bypasses per-column visible_to restrictions and always sees every column. The V003 column-visibility check in stage2.py implements this admin override.

**Use case:** Ensures admins can fully inspect and manage the system without being restricted by user-facing visibility policies that apply to ordinary roles.

**Code:** `provisa/compiler/stage2.py`

**Tests:** `tests/unit/test_stage2.py`, `tests/unit/test_governance.py`

### REQ-1132 · Semantic-Layer Meta Visibility {#REQ-1132}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

Metadata visibility is tiered by role. The meta domain exposes two column classes: CORE (table_name, column_name, data_type, description, alias, schema_name, domain_id, keys, relationship endpoints) and GOVERNANCE (visible_to, unmasked_to, writable_by, masking rules, view_sql, native_filter_type, scope). DEFAULT tier (every role): sees CORE meta for directly-accessible tables plus 1-hop relationship neighbors (user-defined/semantic relationships only, excluding synthetic catalog edges), bidirectional. Meta DOMAIN GRANT: sees CORE meta for all tables and relationships. Per-relationship opt-out suppresses target metadata from DEFAULT discovery.

**Use case:** Enables governed discovery over the reachable neighborhood without exposing data. Separates catalog structure knowledge (CORE) from governance configuration (GOVERNANCE), allowing each tier explicit control.

**Code:** `provisa/security/rights.py`, `provisa/compiler/stage2.py`, `provisa/core/models.py`, `provisa/core/schema_org.py`, `provisa/core/repositories/relationship.py`, `provisa/api/app.py`

**Tests:** `tests/unit/test_meta_governance_visibility.py`

### REQ-1133 · Domain Access Control {#REQ-1133}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The `ops` domain behaves like any ordinary data domain: roles must be explicitly granted rights to access it. There is no implicit access to ops tables; admins bypass this via the admin capability override.

**Use case:** Prevents unintended exposure of operational/internal tables. Each role must be explicitly granted access according to their responsibility.

**Code:** `provisa/api/startup_seed.py`

**Tests:** `tests/unit/test_ops_domain.py`

### REQ-1134 · Governance Column Visibility {#REQ-1134}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The `view_governance` capability is required to view GOVERNANCE columns in the meta domain (visible_to, unmasked_to, writable_by, mask_type, mask_pattern, mask_replace, mask_value, mask_precision, view_sql, native_filter_type, scope). A role with the meta domain grant does NOT automatically see GOVERNANCE columns; view_governance must be granted independently.

**Use case:** Separates discovery (CORE meta) from administration/audit (GOVERNANCE meta). Allows data stewards to see governance configuration without granting broad meta-domain access, and vice versa.

**Code:** `provisa/security/rights.py`, `provisa/compiler/schema_gen.py`, `provisa/compiler/stage2.py`

**Tests:** `tests/unit/test_meta_governance_visibility.py`

## 11. Platform, Infrastructure & Delivery

### REQ-1135 · Licensing {#REQ-1135}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The installation records a tamper-evident first-use timestamp that survives reinstall. On first startup with no existing anchor, the current date is recorded as first_seen. The anchor payload {first_seen, machine_id} is Ed25519-signed and written to multiple locations outside the application install directory (e.g. ~/.provisa, an OS-native store, and a secondary marker). On every startup the system reads all anchors, takes the earliest (min) first_seen, and rewrites any missing anchor with that value. No network access is required — the mechanism is fully airgapped and time-based.

**Use case:** Enables a 30-day trial clock that cannot be reset by uninstalling and reinstalling the application, without any license server or phone-home.

**Code:** `provisa/licensing/anchors.py`, `provisa/licensing/keys.py`, `provisa/licensing/machine_id.py`

**Tests:** `tests/unit/test_licensing.py`

### REQ-1136 · Licensing {#REQ-1136}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

The trial-expiry evaluation is resistant to wall-clock rollback. The system persists a monotonic wall-clock high-water mark and evaluates elapsed time as max(now, high_water) - first_seen. The trial is considered expired when elapsed >= 30 days and no valid license is present. Setting the system clock backward does not extend the trial.

**Use case:** Prevents a user from evading the nag by moving the machine clock backward.

**Code:** `provisa/licensing/monotonic.py`

**Tests:** `tests/unit/test_licensing.py`

### REQ-1137 · Licensing {#REQ-1137}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

After the trial expires and no valid license is present, a self-contained nag message is surfaced on every access surface. It is emitted in the UI as a persistent shell banner, and through each served protocol's native non-fatal notice channel only (pgwire NoticeResponse, Bolt SUCCESS-metadata notifications, Arrow Flight app_metadata, gRPC trailing metadata, REST X-Provisa-License-Notice header plus warnings[] envelope, MCP notifications/message). The nag never modifies the result body, row stream, or any schema-typed field; if a protocol has no safe out-of-band channel the nag is skipped for that protocol. Emission is rate-limited to once per session/connection. The message states that the license is FREE and is used only to gather usage information because Provisa collects no telemetry, that the trial period has elapsed, lists the fields to send (company, position/title, role, first name, last name, email, phone optional), provides both the provisa.dev registration link and the license request email, includes the machine ID, and states how to apply the received license file. The nag never gates or degrades functionality.

**Use case:** Nags the user into registering so end-user contact details are captured — the only means of gathering usage since Provisa ships with no telemetry. The license is free; nothing is charged or blocked.

**Code:** `provisa/licensing/nag.py`, `provisa/licensing/emit.py`, `provisa/licensing/state.py`, `provisa/api/app_startup.py`, `provisa/api/app.py`, `provisa/pgwire/server.py`, `provisa/bolt/session.py`, `provisa/grpc/server.py`, `provisa/api/flight/server.py`, `provisa/api/mcp/server.py`

**Tests:** `tests/unit/test_licensing.py`

### REQ-1138 · Licensing {#REQ-1138}

**Status:** ✅ complete · **Priority:** MUST · **Type:** structural

A license file is an Ed25519-signed token issued by provisa.dev (which holds the private key). Its payload contains company, position/title, role, first name, last name, email, optional phone, machine_id, and issued_at. The application embeds only the corresponding public key and verifies the license entirely offline. The license is a one-time registration capture and is the alternative to telemetry — applying it does NOT enable, start, or schedule any telemetry, usage reporting, or ongoing data collection. Provisa never phones home before, during, or after licensing.

**Use case:** Allows offline verification of an authentic, installation-bound license without any runtime call to an issuing server. Registration at install time replaces telemetry entirely; there is no continuous collection.

**Code:** `provisa/licensing/license.py`, `provisa/licensing/keys.py`

**Tests:** `tests/unit/test_licensing.py`

### REQ-1139 · Licensing {#REQ-1139}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

A license file is applied via `provisa license apply <file>`, a UI upload in Settings → License, or by placing it at ~/.provisa/license.json. On apply the system verifies the Ed25519 signature against the embedded public key and checks that the license machine_id matches the local installation's stable machine_id (derived from hardware/OS identifiers, not the app directory, so it survives reinstall). A valid, matching license permanently silences the nag on all surfaces. A signature failure or machine_id mismatch is rejected and the nag continues.

**Use case:** Lets a user remove the nag by applying the installation-bound license they received.

**Code:** `provisa/licensing/license.py`, `provisa/licensing/machine_id.py`, `provisa/cli.py`

**Tests:** `tests/unit/test_licensing.py`

## 6. Materialized Views & Caching

### REQ-1140 · Materialized View Governance {#REQ-1140}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

A materialized view may only be published if all relationships it depends on are approved. If the requesting user holds the necessary rights to create those relationships, the relationships are automatically created and approved; MV publication proceeds. If the user lacks the required rights, both the relationship creation and MV creation requests are submitted for approval, and MV publication is blocked until all relationships are approved.

**Use case:** Ensures MVs cannot be published over unapproved relationships, preventing uncontrolled data lineage and access patterns. Distinguishes between users who can auto-provision (trusted data engineers) and those who must request approval (least-privilege roles).

**Code:** `provisa/api/admin/mv_relationship_gate.py`, `provisa/api/admin/schema_mutation_ops.py`

**Tests:** `tests/unit/test_mv_relationship_gate.py`

## 6. Execution, Routing, Caching & Performance

### REQ-1141 · Caching {#REQ-1141}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A steward can mark an operational database source as load-sensitive and configure it for scheduled-refresh-only materialization, with a required constraint: at least one of three independent optional gates MUST be configured. The gates are off-peak window (a time range), refresh cadence (refresh_interval), and freshness probe (FreshnessMode.PROBE, checking for source change). The refresh fires on the AND of ONLY the configured gates — an unconfigured gate is vacuously true (ignored). Trigger clocks vary: window-only pulls once per window-open edge; cadence-only pulls every interval regardless of time; probe-only pulls on each evaluation if changed. Probe always gates further: if configured and returns unchanged, the scheduler resets its clock, zero data pull. The query path never touches the source: it serves the last snapshot via FreshnessMode.SCHEDULED (always fresh=true), decoupling query-path load from source access. REQ-1141's contract is scheduler-driven refresh: without a configured gate, there is no scheduler discipline and the source is not load-protected. Boundary principle: MATERIALIZATION REQUIRES A REFRESH POLICY (a REQ-1141 gate OR a cache_ttl). Prefer_materialized + cache_ttl (no gates) is [REQ-826](#REQ-826) lazy read-through (query-triggered on staleness). Prefer_materialized + no gates + no cache_ttl splits on reachability: if the engine can ATTACH/VIRTUAL the source live, prefer_materialized is inert and the source degrades to live serving; if unreachable (_MATERIALIZE_ONLY type — APIs/NoSQL/streams with no live option), it is a load-once-then-frozen snapshot, only meaningful for static reference data or [REQ-847](#REQ-847) mutation-driven invalidation.

**Use case:** Lets a steward protect an operational database by serving it entirely from scheduled snapshots, with fine-grained control over refresh timing (maintenance windows, periodic intervals, or change-driven). The probe signal avoids wasted re-pulls when the source is unchanged. Query traffic never touches the source — only predictable background refresh traffic, bounded by configured gates and staleness by scheduler policy, not query load.

**Code:** `provisa/core/models.py`, `provisa/federation/scheduled_refresh.py`, `provisa/federation/freshness_gate.py`, `provisa/federation/scheduler.py`, `provisa/federation/read_through.py`

**Tests:** `tests/unit/test_scheduled_refresh.py`, `tests/unit/test_scheduler_driver.py`

## 11. Platform, Infrastructure & Delivery

### REQ-1142 · Embedded Tier Deployment {#REQ-1142}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

DuckDB extensions (sqlite_scanner, postgres_scanner, iceberg, delta, community) required by the embedded engine are shipped offline via a separate PyPI package `provisa-duckdb-ext` (py3-none-any universal wheel) that embeds pinned, per-platform `.duckdb_extension` blobs. On embedded launch, the extensions are staged into a writable local directory without any call to extensions.duckdb.org. Missing extensions fail loud; silent network fallback is disabled.

**Use case:** Behind enterprise firewalls where only PyPI/Maven/npm/NuGet are proxied (e.g. Artifactory), extensions.duckdb.org is unreachable, breaking the embedded tier and `provisa run --demo`. Offline delivery via PyPI—already reachable—ensures embedded deployments work in air-gapped environments without network access to DuckDB's extension registry.

**Code:** `provisa/federation/duckdb_extensions.py`, `provisa/federation/duckdb_runtime.py`, `packaging/duckdb-ext/`

**Tests:** `tests/unit/test_duckdb_ext_staging.py`

## 6. Execution, Routing, Caching & Performance

### REQ-1143 · Caching {#REQ-1143}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The table detail view shows a derived, human-readable refresh_policy_summary that expresses the effective refresh/serving behavior per (source, engine) pair, computed server-side from the same resolution logic (federate + freshness_gate + [REQ-1141](#REQ-1141) gates) that the planner uses. The summary states the effective outcome (Live, Scheduled snapshot, or Lazy cache) and warns when prefer_materialized is inert or the table is accidentally frozen.

**Use case:** A steward cannot decode six interacting config fields (prefer_materialized, gates, cache_ttl, cadence, freshness probe, off-peak window) to understand whether a table is served live, from a scheduled snapshot, or from a lazy cache. The server-derived summary, shown in one sentence, eliminates manual decoding and surface misconfiguration warnings at author time.

**Code:** `provisa/federation/policy_summary.py`, `provisa/api/admin/_refresh_summary.py`, `provisa-ui/src/pages/tables/TableEditForm.tsx`

**Tests:** `tests/unit/test_policy_summary.py`, `tests/unit/test_refresh_summary_view.py`

## 11. Platform, Infrastructure & Delivery

### REQ-1144 · Windows Native Launcher {#REQ-1144}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The Windows native launcher displays a visible WinForms splash screen (startup monitor) during first launch instead of running fully hidden. The monitor launches first-launch-native.ps1 hidden, tails a progress breadcrumb file (%USERPROFILE%\.provisa\.startup-status), polls the /health endpoint for readiness, warms first-paint UI endpoints, then opens the browser and closes.

**Use case:** The prior -WindowStyle Hidden chain showed nothing for tens of seconds during API startup and swallowed fatal staging errors to a hidden console, making failures appear as "nothing happened." A visible monitor surfaces startup progress and fatal errors immediately.

**Code:** `packaging/windows/startup-monitor.ps1`

**Tests:** —

### REQ-1145 · Windows Native Launcher {#REQ-1145}

**Status:** ✅ complete · **Priority:** MUST · **Type:** behavioral

The launcher surfaces startup progress via breadcrumbs. first-launch-native.ps1 and provisa-native.ps1 write STATE|message lines (STAGING, CONFIG, DEMO, START, WAIT, ERROR) to %USERPROFILE%\.provisa\.startup-status for the monitor to display. Failed breadcrumb writes must never abort startup (best-effort).

**Use case:** Breadcrumbs let the monitor and user track each stage of startup (staging runtime, configuring services, starting demo data, waiting for API, etc.), giving visibility into slow stages and enabling the monitor to display progress in real-time.

**Code:** `packaging/windows/first-launch-native.ps1`, `packaging/windows/provisa-native.ps1`

**Tests:** —

### REQ-1146 · Windows Native Launcher {#REQ-1146}

**Status:** ✅ complete · **Priority:** MUST · **Type:** constraint

Runtime upgrade must kill ALL processes holding the runtime before replacing it, including the demo mock servers tracked in .demo.pid, not just .native.pid. A leftover demo python.exe locked the runtime DLLs and made re-staging silently abort, pinning the install to its old runtime.

**Use case:** Without killing all demo processes, a stale Python process holding the runtime directory prevents the upgrade from replacing DLLs, causing the installer to silently fail and continue using the old runtime. This constraint ensures clean, uninterrupted runtime upgrades.

**Code:** `packaging/windows/provisa-native.ps1`

**Tests:** —

### REQ-1147 · Windows Native Launcher {#REQ-1147}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

When the startup monitor drives the launch (env PROVISA_STARTUP_UI=1), provisa-native.ps1 must NOT open the browser itself — the monitor owns the readiness wait and browser open. This avoids a double-open race where both the monitor and the native script attempt to open the same URL simultaneously.

**Use case:** A double-open race (monitor + native script both calling Start-Process browser) causes unpredictable behavior (duplicate tabs, missed readiness wait, flipped window focus). Centralizing browser open to the monitor ensures a single, coordinated open after confirmed readiness and UI warmup.

**Code:** `packaging/windows/provisa-native.ps1`, `packaging/windows/startup-monitor.ps1`

**Tests:** —

## 6. Execution, Routing, Caching & Performance

### REQ-1148 · Freshness / Change Detection {#REQ-1148}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A new freshness PROBE transport whose opaque token comes from a zero-byte SENTINEL marker rather than from scanning the data. Reuses the [REQ-855](#REQ-855) gate unchanged: freshness_token() -> str | None (provisa/events/probes.py:126) compared by equality; a new sentinel drop changes the token -> changed -> re-pull; unchanged -> keep. The sentinel lives at a configured path reachable over a file mount, HTTP, FTP, or SFTP; the token is exists+mtime+size for file/FTP/SFTP, or ETag/Last-Modified for HTTP -- the same HASH token shape files already use (provisa/events/probes.py:46-49). Adds a `sentinel` probe transport + a `sentinel_path` (with transport/credentials) config field; capability classes and the gate machinery are untouched. Selected via change_signal in {probe, ttl_probe} with probe_type=hash pointed at the sentinel.

**Use case:** Lets a steward detect upstream readiness from a published sentinel marker (a near-free poll) instead of scanning or querying the source -- and, combined with [REQ-1141](#REQ-1141) load protection, the cheap sentinel poll replaces hammering the operational source while the heavy re-pull still defers to the off-peak window.

**Code:** `provisa/events/probes.py`, `provisa/events/sentinel_probe.py`, `provisa/events/boot.py`, `provisa/core/models.py`

**Tests:** `tests/unit/test_sentinel_probe.py`

### REQ-1149 · Freshness / Change Detection {#REQ-1149}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A new change_signal PUSH variant, `signal`, for a DATA-LESS external trigger (a Kafka control message on a topic, or an HTTP webhook) that says "the source changed / refresh now" WITHOUT carrying the rows. Distinct from the existing CDC push signals (kafka/debezium/native, provisa/core/change_signal.py:29) where the message CARRIES per-row events and lands as upserts (select_landing_shape -> CDC). A `signal` message instead INVALIDATES the materialized snapshot (marks it stale); the rows are then re-pulled from the SOURCE OF TRUTH on the next refresh -- it does not land the message. It sits between the axes: push-triggered timing, poll-style full/delta re-pull. Adds `signal` to the push-signal vocabulary + a receiver (kafka control topic / webhook endpoint) mapping a trigger to a staleness mark. Under [REQ-1141](#REQ-1141) the signal is the change-detector and the load-protected scheduler still owns WHEN the heavy pull runs (deferred to the off-peak window) -- decoupling the cheap control signal from the data pull.

**Use case:** Lets an upstream system announce a change over Kafka/webhook without Provisa polling or the message carrying data; Provisa invalidates and re-pulls from the source on its own schedule -- ideal for load-protected operational sources ([REQ-1141](#REQ-1141)) where a control-plane signal must not translate into immediate source load.

**Code:** `provisa/core/change_signal.py`, `provisa/federation/change_trigger.py`, `provisa/federation/scheduled_refresh.py`, `provisa/federation/scheduler.py`

**Tests:** `tests/unit/test_change_trigger_signal.py`

## 8. Client Access & Protocols

### REQ-1156 · Protocol Support {#REQ-1156}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

Registered commands (tracked functions and webhooks -- [REQ-205](#REQ-205)..208, [REQ-242](#REQ-242), [REQ-885](#REQ-885)) MUST be invocable and discoverable across ALL client-facing protocol surfaces, not only the three that expose them today (GraphQL, pgwire SQL, REST). A command carries two orthogonal dimensions that every surface must honor: `kind` (query vs mutation) and set-vs-scalar return shape (`return_schema` / `returns = "schema.table"` -> set-returning / table-valued). Current state (verified): GraphQL exposes commands as query/mutation fields (provisa/compiler/actions_schema.py:183-185, wired at provisa/compiler/schema_gen.py:846-848); pgwire SQL invokes `SELECT fn(...)` / `SELECT * FROM fn(...)` and projects them into the pg_proc/routines catalog (provisa/pgwire/function_call.py:83-87, provisa/pgwire/function_catalog.py:67-90), but SQL invocation is gated by a SINGLE hook `maybe_invoke_registered_function` that lives ONLY in `execute_pgwire_sql` (provisa/pgwire/_pipeline.py:528); REST invokes via provisa/api/rest/registered_call.py:162-164. The following surfaces are DARK and MUST be lit: (1) MCP -- run_sql and any tool that runs SQL through `_govern_and_route`/`_execute_plan` (provisa/api/mcp/tools.py:270-272) bypasses the function hook, so even `SELECT fn(...)` does not invoke a command; expose commands as MCP tools and/or route MCP SQL through the shared function hook. (2) Arrow Flight -- ListFlights/catalog lists tables only (provisa/api/flight/catalog.py:87) and do_get runs SQL via `_govern_and_route` (provisa/api/flight/server.py:536-543), bypassing the hook; commands must be listed as Flights and invocable. (3) gRPC (Provisa's OWN server) -- proto_gen emits only per-table Query{T}/Insert{T} RPCs (provisa/grpc/proto_gen.py:231-239) and never iterates si.functions; generate an RPC per command (query-kind -> unary/server-streaming for set returns, mutation-kind -> unary MutationResponse). (4) Cypher/Bolt -- no tracked-function path exists; APOC-style CALL is rejected (provisa/cypher/parser.py:864) and Bolt RUN routes only to _execute_cypher (provisa/bolt/session.py:284); expose commands via a CALL procedure surface (or equivalent) that reaches invoke_tracked_function. The single shared executor `invoke_tracked_function` (provisa/api/data/action_exec.py) MUST remain the one invocation path so per-mutation writable_by / governance ([REQ-869](#REQ-869)) is enforced uniformly regardless of surface; the fix is to route each protocol's function-call path through that executor rather than reimplementing dispatch.

**Use case:** A steward registers a command once and every client -- a GraphQL app, a psql / DBeaver session, a REST caller, an MCP-connected assistant, an Arrow Flight consumer, a gRPC service, and a Neo4j/Bolt/Cypher client -- can invoke and discover it with identical governance, instead of the command silently working on three surfaces and being invisible or a no-op on the rest.

**Code:** `provisa/api/data/action_exec.py`, `provisa/api/mcp/tools.py`, `provisa/api/mcp/server.py`, `provisa/api/flight/server.py`, `provisa/api/flight/catalog.py`, `provisa/grpc/proto_gen.py`, `provisa/grpc/server.py`, `provisa/cypher/parser.py`, `provisa/bolt/session.py`, `provisa/api/rest/registered_call.py`

**Tests:** `tests/unit/test_command_surface_discovery.py`, `tests/unit/test_grpc_command_endpoint.py`, `tests/unit/test_bolt_command_call.py`, `tests/unit/test_proto_gen.py`, `tests/unit/test_tracked_function_invocation.py`

## 5. Query Languages, Compilation & Operations

### REQ-1157 · Materialized Views & Caching {#REQ-1157}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

A registered relation backed by `view_sql` (a Provisa-managed view or, with materialize=true, a materialized view -- [REQ-885](#REQ-885)/[REQ-543](#REQ-543), provisa/core/models.py:496-498) is DERIVED, not a base table, and MUST be exposed query-only: the schema generator MUST NOT emit insert/upsert/update/delete GraphQL mutations for it, and raw-SQL surfaces (pgwire / Flight SQL / Bolt) MUST reject writes to it. Rationale: a write to a view either fails at the source (non-updatable view) or lands in the mv_cache snapshot that the next [REQ-879](#REQ-879) refresh silently overwrites -- data loss with no error, violating the project's no-silent-failure rule. Implementation: fetch_tables carries `view_sql` (provisa/api/admin/db_queries.py:114), _build_visible_tables sets _TableInfo.read_only = bool(view_sql) (provisa/compiler/schema_types.py, provisa/compiler/schema_gen.py:_build_visible_tables), and the mutation-field loop skips read_only relations (provisa/compiler/schema_gen.py). The GraphQL read surface (query fields, aggregates, relationships) is UNCHANGED -- only the write surface is withdrawn. A base table is never affected (view_sql is null).

**Use case:** A steward composes an MV over a Provisa-hosted function ([REQ-885](#REQ-885)) and it appears in GraphQL for querying, but insert/update/delete mutations are no longer generated for it -- so a client cannot issue a write that would be lost on the next refresh. The relation reads like any table; it just cannot be written.

**Code:** `provisa/api/admin/db_queries.py`, `provisa/compiler/schema_types.py`, `provisa/compiler/schema_gen.py`

**Tests:** —

## 9. Installation & Deployment

### REQ-1158 · Package Distribution {#REQ-1158}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** infrastructure

A universal py3-none-any wheel package `provisa-pg-ext` carrying the pinned per-platform PostgreSQL extension and FDW binaries (sqlite_fdw, parquet_fdw, parquet_s3_fdw, pg_lake, pg_duckdb, pg_analytics — the [REQ-898](#REQ-898) set), published to PyPI so the embedded tier acquires them via PyPI/Artifactory instead of github.com/kenstott/provisa/releases (unreachable behind PyPI/Maven/npm-only enterprise proxies). Built by new `wheel` job in .github/workflows/build-pg-extensions.yml collecting per-platform build tarballs. Package skeleton at packaging/pg-ext/ with provisa_pg_ext.ext_root() locator for runtime discovery.

**Use case:** Embedded tier (and enterprise deployments behind restricted proxies) can install PostgreSQL extensions offline via standard PyPI/Artifactory channels instead of fetching from unreachable GitHub releases.

**Code:** `packaging/pg-ext/`, `.github/workflows/build-pg-extensions.yml`

**Tests:** —

## 11. Platform, Infrastructure & Delivery

### REQ-1150 · Embedded CLI Startup {#REQ-1150}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

The `provisa run` console command signals startup completion by polling the API /ready endpoint (the warm gate, not /health), prints a "✓ Provisa is ready — <url>" completion line, and opens the browser automatically with `?tour=1` for a `--demo` run (so the guided tour auto-starts). A new `--no-browser` flag opts out of the open while still printing the URL. Failures and 300s timeouts are best-effort and non-fatal: the servers keep running, and the URL is printed manually as a fallback.

**Use case:** Previously `provisa run` printed URLs then served forever with no completion signal and no browser open, so `--demo` appeared stuck even though the app was up. Readiness polling, completion line, and browser auto-open provide immediate visual feedback that the system is ready and reachable.

**Code:** `provisa/cli.py`

**Tests:** —

## 5. Query Languages, Compilation & Operations

### REQ-1159 · Extensible Functions {#REQ-1159}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

Registered commands (tracked functions) MUST be composable INLINE within a larger SQL statement — e.g. `SELECT o.id, e.embedding FROM orders o JOIN enrich_cmd('main.public.orders') e ON o.id = e.id` — not only as a standalone top-level `SELECT * FROM fn(args)` / `SELECT fn(args)`, and this MUST work across ALL client-facing surfaces (GraphQL, pgwire SQL, REST, MCP run_sql, Arrow Flight, Provisa gRPC, Bolt/Cypher), routed through the one shared invoke_tracked_function executor and the shared _govern_and_route / _govern_and_route_compiled pipeline. Current gap: provisa/pgwire/function_call.py detect_sql_function_call returns the FIRST Anonymous node match and maybe_invoke_registered_function runs only that command, returning its rows as the whole result, so a command embedded in a join/CTE within a larger statement misfires. Target: a command-localization rewrite pass inside the shared pipeline that (a) detects every command call in the parsed tree, (b) executes each to a local relation typed by the command's declared output dataset contract, substituting the call site via a size-adaptive mechanism (VALUES CTE below a row threshold, registered Arrow table above), (c) forces local federation execution (Route.ENGINE — a command cannot be pushed to a remote source), (d) applies governance at the correct DEFINER/INVOKER identity for the command's input plus the caller's governance for the outer SQL, and (e) validates rows in/out against the command's declared input/output dataset schema (fail-loud). Prerequisite: a canonical per-dataset contract — one or more scalar (column_value) args plus one or more TYPED input datasets (result_set/table_ref args, each carrying an ordered relational field-list) and exactly one TYPED output dataset; canonical field-list is the source of truth (single GraphQL-scalar vocabulary), with JSON Schema and GraphQL SDL as edge projections (seed-in / render-out), not stored truth.

**Use case:** A steward composes a governed external transform directly in ordinary SQL joined against federated tables, on any surface, and — because the command declares and validates its input/output dataset columns — column-level lineage can close across the opaque RPC boundary via taint-closure splicing at the function node.

**Code:** `provisa/pgwire/_pipeline.py`, `provisa/pgwire/function_call.py`, `provisa/executor/function_dispatch.py`, `provisa/api/data/action_exec.py`, `provisa/processors/contract.py`, `provisa/core/models.py`, `provisa-ui/src/pages/commands/CommandFormFields.tsx`, `provisa-ui/src/api/actions.ts`

**Tests:** —

## 6. Data Lineage & Provenance

### REQ-1160 · Column-Level Lineage Graph {#REQ-1160}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

From a SINGLE SQL statement — including one or more inline-composed commands ([REQ-1159](#REQ-1159)) — the platform MUST be able to produce a complete column-level lineage DAG for the entire view, from source inputs to final output columns, with command nodes as first-class, contract-validated (non-opaque) nodes in the graph. Computable STATICALLY from the view definition plus each command's declared input/output dataset contract (no execution required); optionally reconciled with runtime refresh-version records ([REQ-862](#REQ-862) spans / input versions) to stamp which version flowed. Requires extending the lineage resolver from the current leaf-collapsed output-column -> source-column list (provisa/lineage/columns.py resolve_column_lineage returns flat ColumnDerivation leaves, discarding intermediate nodes) to a full node+edge graph that preserves intermediate relations (CTEs, subqueries, joins). Each inline command, once localized into a typed CTE/Arrow relation, contributes real edges: taint-closure by default (each declared output column derives from ALL declared input columns) or precise per-column edges when declared; command nodes are visually distinct opaque-transform boundaries tagged with the command name and its contract. Two grains from one graph: relation-level zoom-out and column-level drill-in. New surfaces (none exist today — column lineage is currently only OTel span attributes emitted at MV refresh): an API that takes a view/statement and returns graph JSON (nodes + edges), and a UI DAG visualization. Each transform (edge) MUST be NAMED and/or DEFINED, not left as an opaque SQL-expression string: the resolver decomposes each projection expression and matches every operation against the union vocabulary of SQL operators/standard functions (SQLGlot expression nodes), engine built-in scalar functions, and registered commands (projected into pg_proc/routines, each carrying its declared contract) — so e.g. lat,lng <- city, state is labeled as the chain "concat/+ (SQL op) -> geocode (command, contract as its def)". This is a sub-DAG inside the edge: the column-level graph states the derivation, and zooming into an edge reveals the named operation chain (expression-level command nodes reuse the same contract / taint-closure machinery). A transform matching no known op is itself a signal worth flagging.

**Use case:** A steward inspects the end-to-end derivation of any view's output columns — across SQL and opaque command boundaries alike — as a rendered DAG; the visualization doubles as a discipline mirror, since an over-declared "frankenstein" command renders as a wide fan-in node, making sloppy contracts visible.

**Code:** `provisa/lineage/columns.py`, `provisa/lineage/versions.py`, `provisa/pgwire/_pipeline.py`, `provisa/api/`, `provisa-ui/src/`

**Tests:** —

### REQ-1161 · Federation-Wide Provenance Graph {#REQ-1161}

**Status:** 💡 proposed · **Priority:** MUST · **Type:** behavioral

The platform MUST be able to merge every per-view / per-statement column-level lineage DAG ([REQ-1160](#REQ-1160)) into ONE federation-wide provenance graph — spanning all registered views, MVs, and inline-composed commands across all sources — by unioning nodes on canonical resolved dataset identity + column name. Base source columns -> every derived column everywhere, with command boundaries transparent via their declared contracts. Node key = resolved dataset identity (physical/semantic, via the centralized dataset identity service, never ad-hoc string comparison — docs/dataset-handling-standards.md "Dataset Identity"): if two references to the same dataset do not resolve to one node, the merged graph fragments silently, which MUST NOT happen. Computed whole but RENDERED progressively: relation-level is the default view, column-level on drill-in, with focus/scoping tools required (ego-graph around a node, upstream-only / downstream-only slice, domain filter) — a literal full-graph render is unusable at federation scale. Built INCREMENTALLY: recompute a view's sub-DAG on its definition change and union incrementally; never rebuild the entire graph. CYCLE handling is detect + CHARACTERIZE + surface, NOT auto-reject: a cycle that crosses a materialization boundary (an MV reading a prior snapshot of a downstream MV) is a LEGAL time-lagged feedback loop — the snapshot is the version boundary that makes it well-defined; a cycle with NO materialization boundary on it is the likely design error (no evaluation order, no stable value). The platform describes the distinction and surfaces it (in the graph and as a diagnostic) for operator judgment; it does not unilaterally reject.

**Use case:** An operator runs impact analysis (change a source column, see everything affected across every domain), traces any derived column back to its sources across the whole federation, and inspects circular definitions with enough context to judge whether a cycle is an intended time-lagged feedback loop or a design error.

**Code:** `provisa/lineage/`, `provisa/api/`, `provisa-ui/src/`

**Tests:** —

## 4. Source Connectors

### REQ-1162 · Materialization Store {#REQ-1162}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A materialized view's time-travel capability is a property of its DEFINITION, guaranteed on any materializing substrate (Iceberg, PostgreSQL, DuckDB, ...). Maintenance is APPEND-ONLY — a version, once written, is never updated or deleted (no valid_to write-back), so it works on engines that cannot cheaply UPDATE a federated store. Each refresh appends one of two ways: `snapshot` appends the whole fresh dataset stamped with system time; `delta` appends only the rows the ENGINE computes as changed (anti-joins in INSERT ... SELECT) plus tombstones for removed keys — never folded row-by-row in Provisa. "As of" history is DERIVED at read time from the immutable append log (latest batch for snapshot; a ROW_NUMBER window per business key for delta), so consumer-facing `as_of` semantics are identical across every substrate. On Iceberg each append is exactly one snapshot ([REQ-844](#REQ-844)). Declared end to end via mv_bitemporal_mode / mv_bitemporal_key (TableInput -> core Table -> MVDefinition.bitemporal -> refresh_mv) with a UI control in TableEditForm. NOTE: exposing `as_of` as a query-language argument that routes bitemporal-MV reads through reconstruction is tracked as [REQ-1163](#REQ-1163) (the consumer query surface, analogous to [REQ-372](#REQ-372) for sources).

**Use case:** Time-travel previously existed only on Iceberg; when Iceberg was unreachable the substrate degraded to replace/append/upsert, losing history. Append-only bitemporal materialization makes time-travel available on any substrate — the same definition retargets across engines without remodeling.

**Code:** `provisa/mv/bitemporal.py`, `provisa/mv/refresh.py`, `provisa/mv/models.py`, `provisa/api/admin/schema_common.py`, `provisa/core/schema_org.py`, `provisa/core/schema.sql`, `provisa/core/repositories/table.py`, `provisa/api/admin/schema_helpers.py`, `provisa/api/app_rebuild.py`

**Tests:** `tests/unit/test_bitemporal.py`, `tests/unit/test_bitemporal_exec.py`, `tests/unit/test_bitemporal_pg.py`, `tests/unit/test_bitemporal_iceberg.py`, `tests/unit/test_bitemporal_refresh_e2e.py`

### REQ-1163 · Materialization Store {#REQ-1163}

**Status:** ✅ complete · **Priority:** SHOULD · **Type:** behavioral

A bitemporal materialized view ([REQ-1162](#REQ-1162)) is READABLE through the query languages: consumers get current state by default and can travel in time. (1) Current-by-default — a bitemporal materialized view's inline-expansion entry (state.view_sql_map, app_rebuild.py) is pointed at a reconstruction over the physical append log (bitemporal.py: view_read_sql), so a plain read returns the CURRENT reconstructed state (latest batch for snapshot; latest-version-per-business-key window dropping tombstones for delta), never the raw log. (2) Time travel — a request-level `X-Provisa-As-Of` header (validated to a safe SQL timestamp literal by parse_as_of; malformed → HTTP 400) applies to the whole query: _prepare_compiled overlays each bitemporal view's expansion entry with an as-of reconstruction over its append log (as_of_view_map + view_read_sql(ts)), giving identical consumer-facing as-of semantics across every substrate. A whole-query as-of context (like a bitemporal system time) rather than a per-field arg keeps it out of the GraphQL schema. Analogous to [REQ-372](#REQ-372) (`as_of` for Iceberg/Delta SOURCES).

**Use case:** Without routing reads through reconstruction a bitemporal MV read would surface all historical versions instead of current state. Current-by-default makes bitemporal MVs usable; the X-Provisa-As-Of header adds opt-in, whole-query time travel.

**Code:** `provisa/api/app_rebuild.py`, `provisa/api/app.py`, `provisa/api/data/endpoint.py`, `provisa/api/data/endpoint_dev.py`, `provisa/mv/bitemporal.py`

**Tests:** `tests/unit/test_bitemporal_exec.py`, `tests/unit/test_bitemporal_read_integration.py`, `tests/integration/test_bitemporal_http_e2e.py`

### REQ-1164 · Materialization Store {#REQ-1164}

**Status:** 💡 proposed · **Priority:** SHOULD · **Type:** behavioral

Declarative `entity` and `fact` shortcuts that LOWER to the existing MV / bitemporal / relationship primitives — the two constructs both a star schema and a Data Vault are built from, kept methodology-neutral. (1) `entity` — a keyed, deduplicated, optionally-historized projection of a source; lowers to a materialized view (view_sql projecting the key + attributes) with materialize=true, and when history is requested a bitemporal MV ([REQ-1162](#REQ-1162)): history=scd2 → mode=delta, history=snapshot → mode=snapshot, keyed on the business key. Serves a star DIMENSION (SCD1/SCD2) and a Data Vault HUB+SATELLITE. (2) `fact` — join to entity keys, reduce to a declared grain, aggregate measures; lowers to a materialized view (SELECT grain + dimension FK columns + aggregated measures, GROUP BY grain+FKs) plus registered relationships to the entities. Serves a star FACT and a DV LINK (a degenerate fact of keys). The lowering is pure (a spec → the same registration a user would hand-write), so the whole warehouse retargets across engines as IR — the model GENERATES the star/vault, it does not decide the methodology (grain, conformance, SCD choice stay the modeler's).

**Use case:** Star schemas and Data Vaults are compositions of a tiny primitive set (dimension/fact, hub/ satellite/link). First-class entity/fact sugar lets a modeler declare them once and have Provisa generate the underlying portable MV definitions, instead of hand-writing each view_sql + flags.

**Code:** `provisa/mv/modeling.py`

**Tests:** `tests/unit/test_modeling.py`
