Skip to content

# Dataset Handling Standards

## Principles

1. Schema = physical grouping from the source. The schema name shown to users must be the name the data source itself uses to group datasets. For a relational database, that is the schema (or database in MySQL). For a flat/API source with no native grouping concept, use a fixed constant that names the source type.

2. Trino is preferred when a connector exists. When a Trino connector is configured for a source type (i.e., the type is in `SOURCE_TO_CONNECTOR`), use Trino for schema and table introspection. Trino is the canonical query path; introspection via the same path ensures consistency. Native driver introspection is only used for source types with no Trino connector.

3. No intermediate layers exposed. PG cache schemas, Trino catalog names, and other implementation-layer namespaces must never appear as schema or table names presented to the user.

4. Single endpoint per operation. The UI calls one `availableSchemas` endpoint and one `availableTables` endpoint. Backend routing selects the correct introspection strategy per source type internally.

5. Dataset stewardship — one domain owner, one responsible party. Registering a dataset is a governance act: the registering domain becomes the sole steward of that dataset and is accountable for its governance, access policy, quality, and lifecycle. A datasource may serve multiple domains, but each physical dataset may be stewarded by exactly one domain. Once claimed, no other domain may register it. Uniqueness of virtual names (alias or table_name) within each domain is enforced by Provisa. Source-level uniqueness is inherent to the datasource. Together these guarantees eliminate the need for domain-prefixed aliases — table names are registered under their native source names.

6. All source types must be handled. A source type not listed below must return `[]` for both schemas and tables — never `None`, which triggers a broken Trino fallback for non-Trino sources.

## Source Type Reference

### Direct-route RDBMS — Trino connector + SQLGlot dialect

These types are in both `SOURCE_TO_CONNECTOR` and `SOURCE_TO_DIALECT`. They support direct-route execution (single-source queries bypass Trino) and federated execution.

Source type | Trino connector | SQLGlot dialect | Physical schema concept | Trino catalog schema query
--- | --- | --- | --- | ---
`postgresql` | `postgresql` | `postgres` | Schema | `information_schema.schemata` (filter system schemas)
`mysql` | `mysql` | `mysql` | Database | `information_schema.schemata` (filter system dbs)
`mariadb` | `mariadb` | `mysql` | Database | `information_schema.schemata` (filter system dbs)
`singlestore` | `singlestore` | `singlestore` | Database | `information_schema.schemata`
`sqlserver` | `sqlserver` | `tsql` | Schema | `information_schema.schemata` (filter system schemas)
`oracle` | `oracle` | `oracle` | Schema | `information_schema.schemata`
`duckdb` | `memory` | `duckdb` | Schema | `information_schema.schemata`
`snowflake` | `snowflake` | `snowflake` | Schema | `information_schema.schemata`
`bigquery` | `bigquery` | `bigquery` | Dataset | `information_schema.schemata`
`clickhouse` | `clickhouse` | `clickhouse` | Database | `information_schema.schemata`
`redshift` | `redshift` | `redshift` | Schema | `information_schema.schemata`
`databricks` | `delta_lake` | `databricks` | Schema | `information_schema.schemata`
`hive` | `hive` | `hive` | Database | `information_schema.schemata`
`druid` | `druid` | `druid` | Schema | `information_schema.schemata`
`exasol` | `exasol` | `exasol` | Schema | `information_schema.schemata`

Tables: `information_schema.tables WHERE table_schema = ?`.

### TRINO_ONLY lake sources — Trino connector, no direct driver

These types are in `SOURCE_TO_CONNECTOR` and `TRINO_ONLY_SOURCES`. All queries route through Trino. No SQLGlot dialect; no direct driver. Time-travel (`as_of`) is supported on `iceberg` and `delta_lake` (REQ-372).

Source type | Trino connector | Physical schema concept | Trino catalog schema query
--- | --- | --- | ---
`iceberg` | `iceberg` | Namespace | `information_schema.schemata`
`hive_s3` | `hive` | Database | `information_schema.schemata`
`delta_lake` | `delta_lake` | Schema | `information_schema.schemata`

Tables: `information_schema.tables WHERE table_schema = ?`.

### NoSQL / non-relational — Trino connector, no direct driver

These types are in `SOURCE_TO_CONNECTOR` but have no SQLGlot dialect and no direct driver. All queries route through Trino using the mapping DSL (REQ-251). Introspection falls back to Trino.

Source type | Trino connector | Physical schema concept | Trino catalog schema query
--- | --- | --- | ---
`mongodb` | `mongodb` | Database | `information_schema.schemata`
`cassandra` | `cassandra` | Keyspace | `information_schema.schemata`
`redis` | `redis` | Key-pattern namespace | `information_schema.schemata`
`elasticsearch` | `elasticsearch` | Index | `information_schema.schemata`
`prometheus` | `prometheus` | Metric namespace | `information_schema.schemata`

Tables: `information_schema.tables WHERE table_schema = ?`.

### Calcite-based connectors — Trino connector via Apache Calcite

These types are in `SOURCE_TO_CONNECTOR` using the `kenstott/calcite` Trino plugin. All queries route through Trino.

Source type | Trino connector | Physical schema concept | Trino catalog schema query
--- | --- | --- | ---
`sharepoint` | `sharepoint` | Site/list hierarchy | `information_schema.schemata`
`splunk` | `splunk` | Splunk index | `information_schema.schemata`
`files` | `file` | Directory/path | `information_schema.schemata`

Tables: `information_schema.tables WHERE table_schema = ?`.

### Flat sources — fixed schema constant, native introspection

These types have no Trino connector. `native_schemas` returns a fixed constant; `native_tables` queries the source directly.

Source type | Fixed schema constant | Table discovery method | Native endpoint
--- | --- | --- | ---
`kafka` | `"kafka"` | Topics from `kafka_topics` config table | Provisa config DB
`openapi` | `"openapi"` | GET operations with non-null response schema | OpenAPI spec (stored in state)
`graphql_remote` | `"graphql"` | Query-type fields returning LIST | GraphQL HTTP introspection endpoint
`grpc_remote` | `"grpc"` | Server-streaming RPCs + non-streaming RPCs with repeated response | Remote gRPC server via proto reflection or proto file
`sqlite` | `"main"` | `sqlite_master WHERE type='table'` | Local file via `sqlite3`
`neo4j` | `"neo4j"` | No list introspection — user supplies a Cypher query | Bolt connection
`sparql` | `"sparql"` | No list introspection — user supplies a SPARQL query | SPARQL HTTP endpoint
`govdata` | Category names from `sources.database` | `fetch_tables` per category | GovData JDBC adapter

Note: `kafka` is in `SOURCE_TO_CONNECTOR` (Trino `kafka` connector used for execution) but uses native config-DB introspection for schema/table discovery — it does not use Trino for introspection.

For `graphql_remote`: introspection connects to the remote GraphQL HTTP endpoint directly. The PG cache holds previously-executed query results only — it is not the source and must not be used for introspection.

For `grpc_remote`: introspection connects to the remote gRPC server (via proto reflection or a proto file path/URL in `sources.path`). PG is not the source.

For `neo4j`: datasets are user-supplied Cypher queries that must return a JSON array. Neo4j node labels could theoretically be introspected but combining labels is common, so a fixed query model is used instead.

For `sparql`: datasets are user-supplied SPARQL queries that must return a JSON array. No standard introspection path exists. The schema constant exists only to allow the UI form to complete.

### Event / embedded / file sources — no introspection (return `[]`)

These types have no Trino connector and no native introspection path. Both `available_schemas` and `available_tables` return `[]`. Table definitions are supplied by config, user query, or file path.

Source type | Description
--- | ---
`websocket` | External WebSocket feed — schema is event-driven; no introspection
`rss` | RSS 2.0 / Atom feed — schema fixed to feed fields; no introspection
`csv` | Local or remote CSV file — schema from file headers at registration
`parquet` | Local or remote Parquet file — schema from file metadata at registration
`google_sheets` | Google Sheets API — schema from spreadsheet columns at registration
`ingest` | HTTP push receiver — schema defined in config at registration (REQ-333)
`pinot` | Apache Pinot — Trino `pinot` connector (`pinot.controller-urls`); introspects live via Trino (REQ-1097)

## Introspection Routing

    native_schemas(source_id, source_type, pool, config_conn)
      ├─ "graphql_remote"  → ["graphql"]
      ├─ "grpc_remote"     → ["grpc"]
      ├─ "kafka"           → ["kafka"]
      ├─ "neo4j"           → ["neo4j"]
      ├─ "sparql"          → ["sparql"]
      ├─ "openapi"         → ["openapi"]
      ├─ "sqlite"          → ["main"]
      ├─ "govdata"         → [categories from sources.database]
      ├─ source_type in SOURCE_TO_CONNECTOR → None  (available_schemas falls through to Trino)
      └─ unknown           → []   (never None — avoids broken Trino fallback)

    native_tables(source_id, source_type, schema_name, pool, config_conn, state)
      ├─ "openapi"         → parse spec, filter by schema_name == "openapi"
      ├─ "graphql_remote"  → HTTP introspect remote endpoint, filter list fields
      ├─ "grpc_remote"     → parse proto from remote endpoint, filter streaming/repeated methods
      ├─ "kafka"           → query kafka_topics where source_id = ?
      ├─ "neo4j"           → []  (no introspection)
      ├─ "sparql"          → []  (no introspection)
      ├─ "sqlite"          → sqlite_master WHERE type='table', filter by schema_name == "main"
      ├─ "govdata"         → fetch_tables per category
      ├─ source_type in SOURCE_TO_CONNECTOR → None  (available_tables falls through to Trino)
      └─ unknown           → []

## Name Normalization

Dataset names may be normalized to a semantic standard (e.g., `snake_case`) and may also carry an alias. Source-level naming conventions differ: OpenAPI uses camelCase operation IDs, SQL sources use whatever the DBA chose, GraphQL fields are camelCase, gRPC methods are PascalCase.

Never compare two dataset names with `===`. Always normalize both sides through the centralized normalization function before comparison.

- In the UI: use `toSnakeCase(a) === toSnakeCase(b)` via the shared utility exported from `provisa-ui/src/naming.ts`.
- In the backend: use the canonical function in `provisa.compiler.naming` (REQ-640).

This applies everywhere a source-introspected name is compared against a stored registered name: `isRegistered` checks, alias lookups, duplicate detection, and any cross-source name matching.

Inline assumptions about source casing (e.g., `name.toLowerCase()`, hardcoded `.replace(/-/g, "_")`) are forbidden — they are invisible and diverge from the canonical logic.

## Dataset Identity

Every registered dataset has two forms of identity:

- Physical identity: the native name as returned by the source (`findPetsByStatus`, `orders`, `pets` field in GraphQL, `StreamPets` gRPC method). This is what the query engine uses.
- Semantic identity: `domainId + tableName` (the alias, normalized). This is what the API, GraphQL schema, and end users see.

A reference to a dataset may use either form. Any code that needs to determine whether two references point to the same dataset must resolve both through a centralized identity service — never by ad-hoc string comparison.

### Resolution rules

1. Physical → semantic: look up `registered_tables` by `(sourceId, normalizedPhysicalName)` → returns `(domainId, tableName)`.
2. Semantic → physical: look up `registered_tables` by `(domainId, tableName)` → returns `(sourceId, physicalName)`.
3. Same-form comparison: normalize both names with `toSnakeCase` before comparing. Two normalized names that are equal refer to the same dataset within the same source and schema context.
4. Cross-form comparison: resolve both sides to their physical `(sourceId, normalizedName)` tuple first, then compare tuples.

### Query-language representations

A dataset may also be referenced by name inside a query payload. Each query language has a canonical naming convention for dataset names:

Language | Canonical form | Example
--- | --- | ---
SQL | snake_case, qualified as `schema.table` | `pet_store.pet_by_status`
Cypher | PascalCase node label or relationship type | `PetByStatus`
GraphQL | camelCase field name | `petByStatus`

The primary control over query-language naming is the alias set at registration time (`registered_tables.alias`). When an alias is present it is the authoritative name — it is not a hint, it is the name. The alias is consumed differently per language: SQL uses the alias verbatim (no convention applied); GQL applies its configured convention to the alias; Cypher derives from the GQL name (per REQ-351 the Cypher node label is the PascalCase GraphQL `type_name`), not directly from the alias. When no alias is set, each language derives its name from `tableName` (normalized) via its configured convention.

Resolution priority for the query-language name of a dataset:

1. `alias` (if set) → SQL verbatim; GQL applies convention; Cypher derived from the GQL name
2. `tableName` (normalized) → each language applies its configured convention

Any name extracted from a SQL, Cypher, or GraphQL query must be resolvable to a physical dataset through the same centralized identity service. Resolution de-normalizes from query-language form → checks alias match first → falls back to tableName match → returns `(sourceId, physicalName)`.

### Implementation contract

- Backend: `provisa.core.dataset_identity` module must expose:
- `resolve(ref, lang=None) -> DatasetIdentity` — accepts physical name, semantic `domain.table`, or query-language name (with `lang` in `{"sql", "cypher", "graphql"}`).
- `same_dataset(ref_a, ref_b) -> bool` — resolves both sides before comparing.
- `to_query_name(identity, lang) -> str` — applies the active naming convention to produce the correct form for a given query language.
- Frontend: shared utility in `provisa-ui/src/utils/datasetIdentity.ts` — no per-component name-matching logic.
- All inline normalization (`_normalize_op_id`, ad-hoc `toSnakeCase` comparisons, etc.) must be replaced by calls to these utilities.
- The service reads the active naming convention from `registered_tables.naming_convention` → `sources.naming_convention` → global default, in that priority order (three levels: table, source, global).