Skip to content

fs-federation

Read other feature stores read-only from the store you work in: federated catalogs, and datasets built on another store's features.

name: "Feature Store: Federation" description: "Generate code to read other feature stores read-only from the store you work in: connect(databases_readonly=[...]), the federated catalogs with their FEATURE_STORE column, and a dataset built from another store's features. Invoke when a lab must use production features (or several teams' stores must be browsed together) without copying anything. SKIP when the work must be copied into another store — use fs-migrate." argument-hint: "[primary-store][read-only-stores]" allowed-tools: "Read, Edit, Write, Bash, Glob, Grep"

Generate Python code that keeps working in one feature store (read-write) while reading one or more other stores read-only.

User context: $ARGUMENTS

Step Skill Why
Alternative fs-migrate Copy (promote) work into another store instead of reading it.
After fs-dataset Build a dataset from the other store's features.
Inspect fs-inspect The catalogs now cover every attached store.

Demo: demos/notebooks/P3 - Use production features from the lab.ipynb.

The model

  • The primary store (database=) is read-write: every write, every run(), every registration goes there.
  • The read-only stores (databases_readonly=) add their rows to every catalog read: features, processes, datasets, business dictionary, lineage. Each row says which store it comes from in a FEATURE_STORE column.
  • build_dataset can select a read-only store's features: the dataset is created in the primary store and reads the other store's storage tables. A model trained in the lab then reads exactly what production computed.
  • Nothing is copied, and nothing is written in a read-only store.

Ask for any missing values before generating code: - Primary store (read-write) and the read-only store(s). - Data domain of the features to use (a promoted production store usually has its own).

Emit the following pattern, substituting real values for every <PLACEHOLDER>:

import tdfs4ds                                  # reads the TDFS4DS_* settings of .env
tdfs4ds.reconnect_teradata(raise_error=True)
tdfs4ds.connect(database='<PRIMARY_DATABASE>', databases_readonly=['<READONLY_DATABASE>'])
tdfs4ds.select_data_domain('<DATA_DOMAIN_OF_THE_FEATURES>')
print(tdfs4ds.list_read_stores())               # what is attached (a store that failed validation is skipped and logged)

# 1. One catalog, several stores
cat = tdfs4ds.feature_catalog(all_data_domains=True).to_pandas()
print(cat.groupby(['FEATURE_STORE', 'DATA_DOMAIN'])['FEATURE_NAME'].count())

# 2. A dataset in the primary store, on the read-only store's features
procs = tdfs4ds.process_catalog().to_pandas()
pid = procs[(procs['FEATURE_STORE'].str.upper() == '<READONLY_DATABASE>'.upper())
            & (procs['VIEW_NAME'].str.split('.').str[-1].str.strip('"').str.upper() == '<PROCESS_VIEW>'.upper())
           ]['PROCESS_ID'].iloc[0]
ds = tdfs4ds.build_dataset(entity_id=['<ENTITY_COLUMN>'],
                           selected_features={name: pid for name in ['<FEATURE_1>', '<FEATURE_2>']},
                           view_name='<DATASET_VIEW_NAME>')

# 3. Back to the primary store alone
tdfs4ds.detach_read_stores()

Open the session on a working store and read-only stores (0.3.6.8)

import tdfs4ds
tdfs4ds.open_session(database='<PRIMARY_DATABASE>', databases_readonly=['<READONLY_DATABASE>'])
# or: TDFS4DS_SCHEMA_READONLY=<READONLY_DATABASE> in the .env, then open_session(database='<PRIMARY_DATABASE>')
# or, once connected: tdfs4ds.extend_read_stores('<READONLY_DATABASE>')
print(tdfs4ds.list_read_stores())

databases_readonly=None passes nothing (the .env list applies), a list replaces it, [] detaches all. The report returned by open_session has the attached stores under read_only.

Rules

  • Writes and run() stay in the primary store. run() on a read-only store's process is refused (LookupError with on_failure='raise'): to compute it, connect to that store or promote it (fs-migrate).
  • A feature is (data domain, entity, name). Give each store its own data domains (a promoted production store gets its own with target_data_domain=): the agent and the counting helpers do not tell apart two stores that reuse the same triplet.
  • The user needs SELECT on each read-only store's catalog and business-dictionary views, its FS_T_* storage tables (for datasets), and its vector collections (for the agent's search).
  • connect(databases_readonly=[...]) resets the read set; tdfs4ds.extend_read_stores([...]) adds stores during a session, detach_read_stores() removes them. TDFS4DS_SCHEMA_READONLY in .env sets them for every session.
  • A store is a string ('DB_PROD') or a dict {'schema': ..., 'schema_view': ..., 'vector_prefix': ...} when its views live in another database or its Vector Store collections use another prefix.
  • Semantic search covers every attached store with both backends: the Teradata Vector Store searches each store's own collections; Chroma (0.3.3.15) keeps one set of collections per store in the local index, built from the store's public business-dictionary views. tdfs4ds.build_vector_index() indexes every attached store, build_vector_index(stores=['<READONLY_DATABASE>']) one of them.
  • In the admin app, Parameters picks the primary and the read-only stores; the status bar shows them and the background index builds. Run history and documentation of a read-only store are read through its public views (FS_V_FOLLOW_UP, FS_V_PROCESS_DOCUMENTATION_*, FS_V_DOCUMENTATION_PROVENANCE); a store without them (an older version) shows its processes without runs or documentation.