Skip to content

Architecture

tdfs4ds is a layered Python package on top of teradataml. All state lives inside the Teradata database — there is no out-of-database metadata store, no hidden cache, and no sidecar service to deploy.

Module map

tdfs4ds/
├── __init__.py            — Public API + global config
├── config.py              — JSON / .env / OS env var loader
├── lifecycle.py           — setup(), connect()
├── execution.py           — run(), upload_features(), roll_out()
├── catalog.py             — feature_catalog(), process_catalog(), dataset_catalog()
├── data_domain.py         — get/create/select_data_domain()
├── feature_store/         — Entity, feature, ingestion
├── process_store/         — Process registration + follow-up
├── dataset/               — build_dataset(), Dataset class
├── genai/                 — LLM auto-documentation (OpenAI / Bedrock / vLLM / Azure)
├── agent/                 — Consumer agent + Query optimizer + Chroma index
├── lineage/               — Dependency graph + Sankey + migration manifest
├── utils/                 — FilterManager, TimeManager, crystallize_view, …
└── skills/                — Self-contained SKILL.md files for code agents

Layered view

Layer Purpose Lives in
Storage Temporal feature tables, hidden _HIDDEN mutable tables fronted by public views Teradata database
Catalogs Feature, process, dataset, data-distribution, follow-up, business-dictionary catalogs (all VALIDTIME) Teradata database
Engine Registration + ingestion + dataset building feature_store, process_store, dataset, execution
Lineage SQL → dependency graph → Sankey / migration manifest lineage
AI LLM documentation, query optimizer, consumer agent genai, agent
Skills SKILL.md files that teach external agents how to use the package skills/

Hidden tables pattern

Every catalog uses a _HIDDEN suffix for the mutable table, with a public view of the same name (without the suffix) granting immutable read access:

FS_FEATURE_CATALOG_HIDDEN     ←  writes
FS_V_FEATURE_CATALOG          →  CURRENT VALIDTIME reads

Two consequences:

  • Every read is automatically scoped to a FEATURE_STORE_TIME — point-in-time queries are a one-line change.
  • The mutable table is never queried directly, so the optimizer cannot inline it into a bad plan.

Temporal by default

All catalogs and feature tables are VALIDTIME temporal:

Table Temporal level
Feature catalog VALIDTIME
Process catalog VALIDTIME
Dataset catalog VALIDTIME
Feature data VALIDTIME (per FEATURE_STORE_TIME)
Business dictionary (3 tables) VALIDTIME
Process documentation (4 tables) VALIDTIME

tdfs4ds.FEATURE_STORE_TIME is the universal handle:

tdfs4ds.FEATURE_STORE_TIME = '2024-06-01 00:00:00'   # historical state
tdfs4ds.FEATURE_STORE_TIME = None                     # current state

See Temporal model for the gritty details.

Lazy-loaded heavy modules

genai and agent import LangChain / LangGraph / Chroma / Gradio — a few seconds of import time. tdfs4ds/__init__.py defers both via a module __getattr__ hook, so plain feature-store use (no AI) keeps import tdfs4ds under 500 ms.

import tdfs4ds                       # fast — no AI dependency loaded
tdfs4ds.genai.document_process(...)  # langchain* imported here for the first time

Next