← Training overview

10. Catalogs and datasets

Every artefact you registered in the previous modules — features, processes, datasets — lives in one of three catalogs. The catalogs are what makes tdfs4ds a feature store rather than a pile of views: they record what was registered, when, by whom, and with which lineage.

The three catalogs

tdfs4ds.feature_catalog()    # registered features (one row per feature × entity × domain)
tdfs4ds.process_catalog()    # registered processes (one row per process / view)
tdfs4ds.dataset_catalog()    # registered datasets (one row per dataset view)

Catalog	Stores	Use it to
Feature	FEATURE_NAME, ENTITY_NAME, FEATURE_TYPE, DATA_DOMAIN, source table/view	Discover which features exist, where the values land
Process	PROCESS_ID, VIEW_NAME, original SQL, registered metadata	Re-run a feature engineering step, inspect its DDL
Dataset	DATASET_ID, DATASET_NAME, DATASET_DATABASE	Find a downstream dataset, look up its features

All three are temporal: FEATURE_STORE_TIME = '2024-01-01 00:00:00' returns the catalog as it was on that date.

What a dataset is — and why it matters

A tdfs4ds dataset is a registered view that joins multiple features across one or more entities into a single wide table. It is never materialised; the view is built on top of the per-feature storage tables and re-evaluated every time it is queried. That design is deliberate, and it gives you four advantages:

1. Composition. Pick any subset of features and produce a dataset in one build_dataset(...) call — no ad-hoc joins littered across notebooks.
2. Versioning. Each feature reference in the dataset is pinned to a process_id (the version of the engineering logic that produced it). Re-running the upstream process under a new process_id doesn't change the dataset's definition until you rebuild it.
3. Time travel. Because the storage tables are temporal, every dataset inherits time travel for free — set FEATURE_STORE_TIME and the dataset returns the values that were valid at that timestamp.
4. Governance. The dataset_catalog gives the consumer agent (and any downstream BI tool) a single source of truth for "which datasets exist and what features do they contain".

from tdfs4ds.feature_store.feature_query_retrieval import get_feature_versions

versions = get_feature_versions('customer', ['lifetime_amount', 'churn_score'])
dataset_id, view_name = tdfs4ds.build_dataset(
    entity_id=['customer_id'],
    selected_features=versions,
    view_name='V_CUSTOMER_360',
)

Inspecting catalogs at runtime

from tdfs4ds.dataset.dataset_catalog import DatasetCatalog

DatasetCatalog(
    schema_name=tdfs4ds.SCHEMA,
    name=tdfs4ds.DATASET_CATALOG_NAME,
).get_dataset_features(dataset_id).to_pandas()

The inspect & monitor user-guide page covers the rest of the read-only introspection surface.

Learn more

• Data model — catalog schemas, hidden-table pattern, how the public views are built on top.
• Datasets — build_dataset, grouped strategies, time-travel snapshots.
• Inspect & monitor — read-only catalog helpers and follow-up reports.
• First dataset — the end-to-end walkthrough.