Dataset Handling Standards¶

Principles¶

Schema = physical grouping from the source. The schema name shown to users must be the name the data source itself uses to group datasets. For a relational database, that is the schema (or database in MySQL). For a flat/API source with no native grouping concept, use a fixed constant that names the source type.
Trino is preferred when a connector exists. When a Trino connector is configured for a source type (i.e., the type is in SOURCE_TO_CONNECTOR), use Trino for schema and table introspection. Trino is the canonical query path; introspection via the same path ensures consistency. Native driver introspection is only used for source types with no Trino connector.
No intermediate layers exposed. PG cache schemas, Trino catalog names, and other implementation-layer namespaces must never appear as schema or table names presented to the user.
Single endpoint per operation. The UI calls one availableSchemas endpoint and one availableTables endpoint. Backend routing selects the correct introspection strategy per source type internally.
Dataset stewardship — one domain owner, one responsible party. Registering a dataset is a governance act: the registering domain becomes the sole steward of that dataset and is accountable for its governance, access policy, quality, and lifecycle. A datasource may serve multiple domains, but each physical dataset may be stewarded by exactly one domain. Once claimed, no other domain may register it. Uniqueness of virtual names (alias or table_name) within each domain is enforced by Provisa. Source-level uniqueness is inherent to the datasource. Together these guarantees eliminate the need for domain-prefixed aliases — table names are registered under their native source names.
All source types must be handled. A source type not listed below must return [] for both schemas and tables — never None, which triggers a broken Trino fallback for non-Trino sources.

Source Type Reference¶

Direct-route RDBMS — Trino connector + SQLGlot dialect¶

These types are in both SOURCE_TO_CONNECTOR and SOURCE_TO_DIALECT. They support direct-route execution (single-source queries bypass Trino) and federated execution.

Source type	Trino connector	SQLGlot dialect	Physical schema concept	Trino catalog schema query
`postgresql`	`postgresql`	`postgres`	Schema	`information_schema.schemata` (filter system schemas)
`mysql`	`mysql`	`mysql`	Database	`information_schema.schemata` (filter system dbs)
`mariadb`	`mariadb`	`mysql`	Database	`information_schema.schemata` (filter system dbs)
`singlestore`	`singlestore`	`singlestore`	Database	`information_schema.schemata`
`sqlserver`	`sqlserver`	`tsql`	Schema	`information_schema.schemata` (filter system schemas)
`oracle`	`oracle`	`oracle`	Schema	`information_schema.schemata`
`duckdb`	`memory`	`duckdb`	Schema	`information_schema.schemata`
`snowflake`	`snowflake`	`snowflake`	Schema	`information_schema.schemata`
`bigquery`	`bigquery`	`bigquery`	Dataset	`information_schema.schemata`
`clickhouse`	`clickhouse`	`clickhouse`	Database	`information_schema.schemata`
`redshift`	`redshift`	`redshift`	Schema	`information_schema.schemata`
`databricks`	`delta_lake`	`databricks`	Schema	`information_schema.schemata`
`hive`	`hive`	`hive`	Database	`information_schema.schemata`
`druid`	`druid`	`druid`	Schema	`information_schema.schemata`
`exasol`	`exasol`	`exasol`	Schema	`information_schema.schemata`

Tables: information_schema.tables WHERE table_schema = ?.

TRINO_ONLY lake sources — Trino connector, no direct driver¶

These types are in SOURCE_TO_CONNECTOR and TRINO_ONLY_SOURCES. All queries route through Trino. No SQLGlot dialect; no direct driver. Time-travel (as_of) is supported on iceberg and delta_lake (REQ-372).

Source type	Trino connector	Physical schema concept	Trino catalog schema query
`iceberg`	`iceberg`	Namespace	`information_schema.schemata`
`hive_s3`	`hive`	Database	`information_schema.schemata`
`delta_lake`	`delta_lake`	Schema	`information_schema.schemata`

Tables: information_schema.tables WHERE table_schema = ?.

NoSQL / non-relational — Trino connector, no direct driver¶

These types are in SOURCE_TO_CONNECTOR but have no SQLGlot dialect and no direct driver. All queries route through Trino using the mapping DSL (REQ-251). Introspection falls back to Trino.

Source type	Trino connector	Physical schema concept	Trino catalog schema query
`mongodb`	`mongodb`	Database	`information_schema.schemata`
`cassandra`	`cassandra`	Keyspace	`information_schema.schemata`
`redis`	`redis`	Key-pattern namespace	`information_schema.schemata`
`elasticsearch`	`elasticsearch`	Index	`information_schema.schemata`
`prometheus`	`prometheus`	Metric namespace	`information_schema.schemata`

Tables: information_schema.tables WHERE table_schema = ?.

Calcite-based connectors — Trino connector via Apache Calcite¶

These types are in SOURCE_TO_CONNECTOR using the kenstott/calcite Trino plugin. All queries route through Trino.

Source type	Trino connector	Physical schema concept	Trino catalog schema query
`sharepoint`	`sharepoint`	Site/list hierarchy	`information_schema.schemata`
`splunk`	`splunk`	Splunk index	`information_schema.schemata`
`files`	`file`	Directory/path	`information_schema.schemata`

Tables: information_schema.tables WHERE table_schema = ?.

Flat sources — fixed schema constant, native introspection¶

These types have no Trino connector. native_schemas returns a fixed constant; native_tables queries the source directly.

Source type	Fixed schema constant	Table discovery method	Native endpoint
`kafka`	`"kafka"`	Topics from `kafka_topics` config table	Provisa config DB
`openapi`	`"openapi"`	GET operations with non-null response schema	OpenAPI spec (stored in state)
`graphql_remote`	`"graphql"`	Query-type fields returning LIST	GraphQL HTTP introspection endpoint
`grpc_remote`	`"grpc"`	Server-streaming RPCs + non-streaming RPCs with repeated response	Remote gRPC server via proto reflection or proto file
`sqlite`	`"main"`	`sqlite_master WHERE type='table'`	Local file via `sqlite3`
`neo4j`	`"neo4j"`	No list introspection — user supplies a Cypher query	Bolt connection
`sparql`	`"sparql"`	No list introspection — user supplies a SPARQL query	SPARQL HTTP endpoint
`govdata`	Category names from `sources.database`	`fetch_tables` per category	GovData JDBC adapter

Note: kafka is in SOURCE_TO_CONNECTOR (Trino kafka connector used for execution) but uses native config-DB introspection for schema/table discovery — it does not use Trino for introspection.

For graphql_remote: introspection connects to the remote GraphQL HTTP endpoint directly. The PG cache holds previously-executed query results only — it is not the source and must not be used for introspection.

For grpc_remote: introspection connects to the remote gRPC server (via proto reflection or a proto file path/URL in sources.path). PG is not the source.

For neo4j: datasets are user-supplied Cypher queries that must return a JSON array. Neo4j node labels could theoretically be introspected but combining labels is common, so a fixed query model is used instead.

For sparql: datasets are user-supplied SPARQL queries that must return a JSON array. No standard introspection path exists. The schema constant exists only to allow the UI form to complete.

Event / embedded / file sources — no introspection (return `[]`)¶

These types have no Trino connector and no native introspection path. Both available_schemas and available_tables return []. Table definitions are supplied by config, user query, or file path.

Source type	Description
`websocket`	External WebSocket feed — schema is event-driven; no introspection
`rss`	RSS 2.0 / Atom feed — schema fixed to feed fields; no introspection
`csv`	Local or remote CSV file — schema from file headers at registration
`parquet`	Local or remote Parquet file — schema from file metadata at registration
`google_sheets`	Google Sheets API — schema from spreadsheet columns at registration
`ingest`	HTTP push receiver — schema defined in config at registration (REQ-333)
`pinot`	Apache Pinot — Trino `pinot` connector (`pinot.controller-urls`); introspects live via Trino (REQ-1097)

Introspection Routing¶

native_schemas(source_id, source_type, pool, config_conn)
  ├─ "graphql_remote"  → ["graphql"]
  ├─ "grpc_remote"     → ["grpc"]
  ├─ "kafka"           → ["kafka"]
  ├─ "neo4j"           → ["neo4j"]
  ├─ "sparql"          → ["sparql"]
  ├─ "openapi"         → ["openapi"]
  ├─ "sqlite"          → ["main"]
  ├─ "govdata"         → [categories from sources.database]
  ├─ source_type in SOURCE_TO_CONNECTOR → None  (available_schemas falls through to Trino)
  └─ unknown           → []   (never None — avoids broken Trino fallback)

native_tables(source_id, source_type, schema_name, pool, config_conn, state)
  ├─ "openapi"         → parse spec, filter by schema_name == "openapi"
  ├─ "graphql_remote"  → HTTP introspect remote endpoint, filter list fields
  ├─ "grpc_remote"     → parse proto from remote endpoint, filter streaming/repeated methods
  ├─ "kafka"           → query kafka_topics where source_id = ?
  ├─ "neo4j"           → []  (no introspection)
  ├─ "sparql"          → []  (no introspection)
  ├─ "sqlite"          → sqlite_master WHERE type='table', filter by schema_name == "main"
  ├─ "govdata"         → fetch_tables per category
  ├─ source_type in SOURCE_TO_CONNECTOR → None  (available_tables falls through to Trino)
  └─ unknown           → []

Name Normalization¶

Dataset names may be normalized to a semantic standard (e.g., snake_case) and may also carry an alias. Source-level naming conventions differ: OpenAPI uses camelCase operation IDs, SQL sources use whatever the DBA chose, GraphQL fields are camelCase, gRPC methods are PascalCase.

Never compare two dataset names with ===. Always normalize both sides through the centralized normalization function before comparison.

In the UI: use toSnakeCase(a) === toSnakeCase(b) via the shared utility exported from provisa-ui/src/naming.ts.
In the backend: use the canonical function in provisa.compiler.naming (REQ-640).

This applies everywhere a source-introspected name is compared against a stored registered name: isRegistered checks, alias lookups, duplicate detection, and any cross-source name matching.

Inline assumptions about source casing (e.g., name.toLowerCase(), hardcoded .replace(/-/g, "_")) are forbidden — they are invisible and diverge from the canonical logic.

Dataset Identity¶

Every registered dataset has two forms of identity:

Physical identity: the native name as returned by the source (findPetsByStatus, orders, pets field in GraphQL, StreamPets gRPC method). This is what the query engine uses.
Semantic identity: domainId + tableName (the alias, normalized). This is what the API, GraphQL schema, and end users see.

A reference to a dataset may use either form. Any code that needs to determine whether two references point to the same dataset must resolve both through a centralized identity service — never by ad-hoc string comparison.

Resolution rules¶

Physical → semantic: look up registered_tables by (sourceId, normalizedPhysicalName) → returns (domainId, tableName).
Semantic → physical: look up registered_tables by (domainId, tableName) → returns (sourceId, physicalName).
Same-form comparison: normalize both names with toSnakeCase before comparing. Two normalized names that are equal refer to the same dataset within the same source and schema context.
Cross-form comparison: resolve both sides to their physical (sourceId, normalizedName) tuple first, then compare tuples.

Query-language representations¶

A dataset may also be referenced by name inside a query payload. Each query language has a canonical naming convention for dataset names:

Language	Canonical form	Example
SQL	snake_case, qualified as `schema.table`	`pet_store.pet_by_status`
Cypher	PascalCase node label or relationship type	`PetByStatus`
GraphQL	camelCase field name	`petByStatus`

The primary control over query-language naming is the alias set at registration time (registered_tables.alias). When an alias is present it is the authoritative name — it is not a hint, it is the name. The alias is consumed differently per language: SQL uses the alias verbatim (no convention applied); GQL applies its configured convention to the alias; Cypher derives from the GQL name (per REQ-351 the Cypher node label is the PascalCase GraphQL type_name), not directly from the alias. When no alias is set, each language derives its name from tableName (normalized) via its configured convention.

Resolution priority for the query-language name of a dataset:

alias (if set) → SQL verbatim; GQL applies convention; Cypher derived from the GQL name
tableName (normalized) → each language applies its configured convention

Any name extracted from a SQL, Cypher, or GraphQL query must be resolvable to a physical dataset through the same centralized identity service. Resolution de-normalizes from query-language form → checks alias match first → falls back to tableName match → returns (sourceId, physicalName).

Implementation contract¶

Backend: provisa.core.dataset_identity module must expose:
resolve(ref, lang=None) -> DatasetIdentity — accepts physical name, semantic domain.table, or query-language name (with lang in {"sql", "cypher", "graphql"}).
same_dataset(ref_a, ref_b) -> bool — resolves both sides before comparing.
to_query_name(identity, lang) -> str — applies the active naming convention to produce the correct form for a given query language.
Frontend: shared utility in provisa-ui/src/utils/datasetIdentity.ts — no per-component name-matching logic.
All inline normalization (_normalize_op_id, ad-hoc toSnakeCase comparisons, etc.) must be replaced by calls to these utilities.
The service reads the active naming convention from registered_tables.naming_convention → sources.naming_convention → global default, in that priority order (three levels: table, source, global).

Dataset Handling Standards¶

Principles¶

Source Type Reference¶

Direct-route RDBMS — Trino connector + SQLGlot dialect¶

TRINO_ONLY lake sources — Trino connector, no direct driver¶

NoSQL / non-relational — Trino connector, no direct driver¶

Calcite-based connectors — Trino connector via Apache Calcite¶

Flat sources — fixed schema constant, native introspection¶

Event / embedded / file sources — no introspection (return [])¶

Introspection Routing¶

Name Normalization¶

Dataset Identity¶

Resolution rules¶

Query-language representations¶

Implementation contract¶

Event / embedded / file sources — no introspection (return `[]`)¶