Connect and configure¶

Every session starts the same way: open a Teradata context with teradataml, then import and connect tdfs4ds.

Minimal setup¶

import warnings
warnings.filterwarnings("ignore")

import teradataml as tdml
import os
from dotenv import load_dotenv

load_dotenv()
tdml.create_context(
    host              = os.getenv('TDML_HOST'),
    user              = os.getenv('TDML_USERNAME'),
    password          = os.getenv('TDML_PASSWORD'),
    database          = os.getenv('TDML_DATABASE'),
    temp_database_name= os.getenv('TDML_TEMP_DATABASE'),
)

# Import AFTER create_context — tdfs4ds auto-detects SCHEMA from the session
import tdfs4ds
tdfs4ds.connect(database=os.getenv('TDML_DATABASE'), create_if_missing=True)
tdfs4ds.create_data_domain('MY_PROJECT')   # idempotent; warns if it already exists

# Optional in-session overrides
# tdfs4ds.FEATURE_STORE_TIME = '2024-01-01 00:00:00'   # time-travel point
# tdfs4ds.DISPLAY_LOGS       = True

Why import order matters

import tdfs4ds reads the active teradataml context to auto-detect SCHEMA. If you import the package before create_context, SCHEMA is None and you must set it manually.

`connect()` vs `setup()`¶

Function	When to call	What it does
`tdfs4ds.setup()`	First time per database	Provisions every catalog table and view (feature catalog, process catalog, data distribution, follow-up, business dictionary). Idempotent.
`tdfs4ds.connect(create_if_missing=True)`	Every session	Hydrates Python globals from existing catalogs; creates any missing tables when `create_if_missing=True`.

If you are unsure, call connect(create_if_missing=True) — it covers both cases.

Which databases hold a feature store?¶

tdfs4ds.find_feature_stores()
#    DATABASE_NAME  VIEW_DATABASE  COMPLETE  MISSING  READABLE  N_PROCESSES  N_DATA_DOMAINS  ERROR
# 0         DB_LAB         DB_LAB      True               True           13               6
# 1        DB_PROD        DB_PROD      True               True            2               1

It reads DBC.TablesV once, so it lists only what your session may see, then counts each store's processes and data domains through its public process-catalog view.

Column	Meaning
`COMPLETE`	every catalog table `connect()` needs is there; when False, `MISSING` names the others and `connect(create_if_missing=False)` returns False
`VIEW_DATABASE`	where the catalog views are (another database for a store set up with `database_view`)
`READABLE`	the count worked; when False, `ERROR` holds the whole refusal (usually a missing SELECT right)

complete_only=True keeps the stores connect() accepts as they are, details=False skips the counts, and databases=[...] restricts the search. The admin app's Parameters panel lists the same stores (searched at startup, with a Refresh list button).

connect() also works on a store you may only read. Refreshing its catalog views needs the DROP right; when that is refused, the existing views are kept and a warning says so.

Separate database for views (`database_view`)¶

Some environments keep views in one database and tables in another (governance / access-control policy). tdfs4ds supports this with an opt-in database_view argument on both setup() and connect():

# Tables (FS_T_*, catalogs, business dictionary, lineage) go in DATA_DB;
# every view (process views, dataset views, all FS_V_* framework views) goes in VIEW_DB.
tdfs4ds.setup(database='DATA_DB', database_view='VIEW_DB')
# or, every session:
tdfs4ds.connect(database='DATA_DB', database_view='VIEW_DB', create_if_missing=True)

database_view is optional. When omitted (None), views are co-located with the tables in database — identical to the historical behaviour, so existing code is unaffected.
You can also set it via the TDFS4DS_SCHEMA_VIEW environment variable / .env entry, or the schema_view JSON config key (same tier as SCHEMA).
Once set, registering a process or dataset with an unqualified name (e.g. target_view_name='my_source_view') places the view in the view database automatically — no need to prefix it every time. An explicit db.view name is still honoured as-is.

Grant required

A view in the view database that reads a table in the table database needs SELECT on the table database (and WITH GRANT OPTION to expose the view to downstream consumers). Grant this before using database_view.

Migrating an existing single-database store

Set database_view and call connect(create_if_missing=True) — the framework views are (re)created in the new view database. The now-stale views left in the old database are not dropped automatically.

Read from other feature stores (`databases_readonly`)¶

You can connect to your primary store read-write as usual and extend the session to one or more other feature stores in read-only mode. Your uploads still write only to the primary store, but catalog / documentation / lineage reads — and build_dataset — span every attached store.

# Primary store DATA_DB (read-write) + two other stores read-only:
tdfs4ds.connect(database='DATA_DB', databases_readonly=['SHARED_FS', 'TEAM_B_FS'],
                create_if_missing=True)

# ...or attach/detach at any time after connecting:
tdfs4ds.extend_read_stores('SHARED_FS')          # a bare name (schema == view schema)
tdfs4ds.extend_read_stores({'schema': 'T_DB',    # or a descriptor for a table/view split
                            'schema_view': 'V_DB'})
tdfs4ds.list_read_stores()                       # inspect what's attached
tdfs4ds.detach_read_stores('SHARED_FS')          # remove one, or detach_read_stores() for all

databases_readonly is optional. When omitted, nothing changes — reads are single-store, byte-identical to before.
Also settable via the TDFS4DS_SCHEMA_READONLY environment variable / .env (comma-separated), or the schema_readonly JSON config key.
Once attached, feature_catalog(), dataset_catalog(), the consumer agent, and lineage all show rows from every store, tagged with a FEATURE_STORE column naming the source. build_dataset can select a read-only store's features into a dataset built in your primary store.
Writes never change. upload_features, documentation, and lineage writes always target your primary SCHEMA; read-only stores are never modified.

Grants required

The connecting user needs SELECT on each read-only store's FS_V_* catalog / business-dictionary / lineage views, its FS_T_* feature tables (for build_dataset), and — with VECTOR_STORE_BACKEND='teradata' — its vector collections. A store that isn't readable (or is a different tdfs4ds version) is logged and skipped; the others still attach.

Same-name features across stores

Resolution is exact when the stores hold different data domains / entities / feature names (the usual case). If two stores independently register the same (data domain, entity, feature name), the agent resolves it to the primary store. The admin app's Parameters panel counts such features when you attach a store.

Semantic search across stores

Both vector backends search every attached store. Each store keeps its own collections: in its own database with VECTOR_STORE_BACKEND='teradata', and in your local index with Chroma (since 0.3.3.15). A read-only store's Chroma collections are built from its public business-dictionary views (read rights are enough, nothing is written to it): tdfs4ds.build_vector_index() indexes every attached store, build_vector_index(stores=['DB_PROD']) one of them. Until a store is indexed, the agent still finds its objects through the catalog and business-dictionary reads.

Vector Store authentication¶

If you use tdfs4ds.VECTOR_STORE_BACKEND = 'teradata' (the Enterprise Vector Store, instead of the default local Chroma index), the consumer agent's vector index needs its own session auth against the Vantage Vector Store REST API — a separate channel from the tdml.create_context() connection above, via teradataml.set_auth_token(...).

Both setup() and connect() (and the tdfs4ds.serve server on startup) attempt this automatically, non-fatally, from environment variables. Set TDFS4DS_VECTOR_STORE_AUTH_BASE_URL plus one of the following groups — the first fully-configured one wins, in this order:

Mechanism	Additional env vars
JWT (tried first)	`TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN` — a token you already hold
PAT	`TDFS4DS_VECTOR_STORE_AUTH_PAT_TOKEN`, `TDFS4DS_VECTOR_STORE_AUTH_PEM_FILE` (+ optional `TDFS4DS_VECTOR_STORE_AUTH_USERNAME`)
Basic	`TDFS4DS_VECTOR_STORE_AUTH_USERNAME`, `TDFS4DS_VECTOR_STORE_AUTH_PASSWORD`
OAuth	`TDFS4DS_VECTOR_STORE_AUTH_CLIENT_ID`

PAT, Basic and OAuth make tdfs4ds generate a token. JWT instead uses one you already have, so it is tried first — an explicitly supplied token is never silently ignored in favour of a generated one.

In-cluster: map the platform's token onto the variable

In a Kubernetes / VantageCloud Lake deployment the platform injects an access token into the pod under its own name (OAUTH_ACCESS_TOKEN or similar). Point the JWT mechanism at it explicitly:

TDFS4DS_VECTOR_STORE_AUTH_BASE_URL=http://vectorstore.vectorstore.svc.cluster.local:8000
TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN=$OAUTH_ACCESS_TOKEN

$VAR is not expanded by tdfs4ds itself, so that line works only when something else interpolates it first (load_dotenv(), a shell, or a Docker --env-file). Relying on tdfs4ds auto-discovering the file means writing the resolved value instead. On AI Studio, generate_ai_studio_env() does that for you.

That is the equivalent of calling tdml.set_auth_token(base_url=…, auth_token=os.getenv('OAUTH_ACCESS_TOKEN')) yourself.

Only TDFS4DS_-prefixed variables are read

tdfs4ds never picks up an unprefixed environment variable for authentication, however conventional the name. A generic name may be owned by an unrelated component in the same environment, and reading it would send that component's credential to the Vector Store endpoint with nothing in your configuration naming it. The mapping above is one line and makes the intent explicit.

The mechanism used (or "not configured") is logged at connect()/setup() time; the token value never is — only the base_url and the name of the variable it came from. If none of these are set, VECTOR_STORE_BACKEND='teradata' will not work until you configure one — the resulting error message repeats this list.

To confirm it actually works — along with the Vector Store service, the instruct model and the embedding model — run check_configuration().

Checking your configuration¶

tdfs4ds.check_configuration() validates the AI-facing configuration and, for anything that does not pass, tells you which setting to change:

tdfs4ds.check_configuration()             # real handshake, one prompt, one embedding
tdfs4ds.check_configuration(live=False)   # resolve config + build clients, call nothing

tdfs4ds configuration check — v0.3.1.154

✅ Vector Store auth       JWT (base_url=http://vectorstore…:8000 token from TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN)
✅ Vector Store service    healthy
✅ Instruct model          openai / gpt-4o — replied 'hello world' in 0.61s
❌ Embedding model         openai / text-embedding-3-small — embedded 'hello world' into 1024 dimensions
                           → EMBEDDING_MODEL_DIM is 1536 but the model returns 1024
                           fix: Set tdfs4ds.EMBEDDING_MODEL_DIM = 1024 …

When a database session is open, two more checks run first (since 0.3.1.222):

Temporal (VALIDTIME) — a session-only VALIDTIME table is created, written, read and dropped. Every tdfs4ds table is temporal, so a failure here means the Temporal feature is not available to this user.
Query band & DBQL — the session query band can be set (your own band is restored), and DBQL is readable with logging active for this user. Process executions are tagged with a query band so that tdfs4ds.followup_resource_usage() can join them to DBQL. Missing DBQL access or logging does not fail the check; it is reported with the GRANT or BEGIN QUERY LOGGING statement to ask the DBA for.

Without a session, both checks are skipped with a hint.

It runs eight checks: Config load (below), Vector Store auth (the mechanism above), Vector Store service (health() then list(), only with the Teradata backend), LLM transport (TLS), instruct model (a hello-world prompt), JSON constraints, embedding model, and MCP server (only when MCP_SERVER_URL is set). A ninth, the write-path Vector Store self-test, runs only with selftest=True.

LLM transport runs before the model and sends no credential: it completes a TLS handshake with INSTRUCT_MODEL_URL, reports the certificate's issuer and days to expiry, and separates an untrusted issuer from a hostname mismatch from an expired certificate — three failures that look alike and have three different fixes. JSON constraints probes typed output along both paths tdfs4ds uses, and only if one of them reports trouble does it run a nine-request sweep identifying the serving stack. Both are covered in LLM & embedding models. The embedding one depends on the backend — with the Teradata Vector Store and TERADATA_AI_API_TYPE set it checks that TeradataAI(...) instantiates, since in-database embedding needs nothing more from the client; otherwise it embeds a string and verifies the vector length against EMBEDDING_MODEL_DIM (a mismatch there never raises, it just builds an index that cannot be searched).

The MCP check runs in two stages — list the tools, then have a model call one — so which stage fails tells you whether to look at the connection or at the model. A connection failure is never caused by a missing instruct model: without one, the second stage is skipped rather than failed.

The Vector Store service check is two-stage for the same reason: health() says the service is running, list() says it will accept your credential. Only the second opens a database session, so only the second can catch an expired token.

All green, but rebuild_vector_indexing() fails

If indexing reports "service is unreachable or not enabled … check that the Enterprise Vector Store feature is licensed" while this check passes, read the detail at the end of the message. If it contains SQLState 28000 (The UserId, Password or Account is invalid), the service is reachable and licensed — it was refused a database session on your behalf. Do not go looking at licensing.

The usual cause is an expired token: teradataml stores a ready-made JWT without validating it, so set_auth_token() and a health probe both succeed with a stale one. Re-read the current token into TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN and call connect() again. If the credential is fresh, check the Basic username and password, then that the database user may log on the way the service logs on for it — set_auth_token requires the privilege to log in with a NULL password (see GRANT LOGON in the Teradata documentation).

Versions from 0.3.1.164 report this as an auth failure with those causes listed, and the service check above catches it before you reach indexing.

The function never raises and never prints a secret — only variable names, URLs and model names. It returns a dict ({"ok", "version", "live", "checks"}) so it can gate a startup script. The same check is available in the admin UI under Parameters → Configuration check.

The Teradata database connection is not checked: it is external to the platform and opened by you (in-cluster, typically tdml.create_context(host=…, logmech='JWT', logdata=…)).

\"not configured\" for something you did configure

tdfs4ds reads TDFS4DS_* into tdfs4ds.<ATTR> once, during import tdfs4ds. A variable exported after that import — or a .env outside the working directory — never reaches the module config. Vector Store auth is the exception (read live from os.environ), which is why it alone can pass while everything else reports nothing set.

The Config load check names the affected variables instead of letting you hunt for a setting you already made. Fix it with:

tdfs4ds.load_config()                              # re-read env + discovered files
tdfs4ds.load_config(dotenv_path='/path/to/.env')   # or point at the file

It also reports values that differ from the environment — not a failure, since an in-session override is legitimate, but it catches an attribute left at its default (e.g. VECTOR_STORE_BACKEND='chroma') while the environment asks for something else.

Exporting a configuration¶

tdfs4ds.export_config() captures a configuration as a file — to reproduce a working session, or to turn scattered environment variables into one:

tdfs4ds.export_config()                                  # returns the text; writes nothing
tdfs4ds.export_config('tdfs4ds.json')                    # reusable JSON config
tdfs4ds.export_config('.env', fmt='env', source='env')   # env vars -> a file

source='module' (default) snapshots the running configuration; source='env' converts the TDFS4DS_* environment variables even when they never reached the module config. fmt='env' is the only format that can carry connection and Vector Store auth variables — they are pass-through, never module attributes.

Secrets are redacted to a commented placeholder by default (include_secrets=True writes them in clear); JSON never carries credentials at all, since load_config() rejects them from JSON by design.

Data domains¶

Features are isolated by DATA_DOMAIN. Use the dedicated helpers — never set the global directly:

tdfs4ds.get_data_domains()                # list every domain (* marks active)
tdfs4ds.create_data_domain('PROJECT_A')   # create + activate (idempotent)
tdfs4ds.select_data_domain('PROJECT_B')   # activate an existing domain

select_data_domain raises ValueError if the domain is unknown — guarding against silent typos that would otherwise cross-pollute features.

Cleaning up¶

tdml.remove_context()

Always call this at the bottom of every script or notebook. Leaving open contexts behind blocks ports and slowly leaks connections.

Server deployment¶

When you're ready to expose the package to other users, tdfs4ds ships an OpenAI-compatible API server with an optional admin chatbot UI, documentation server, and interactive lineage graph. See Deployment for setup instructions (locally with python -m tdfs4ds.serve or in Docker with docker compose up).

Next step¶

Register your first feature in First feature.