Skip to content

tdfs4ds.config

tdfs4ds.config

Configuration loading for tdfs4ds.

Three external configuration sources are supported, applied in this order so that higher-priority sources always win:

  1. JSON config file — project defaults, searched in order:
  2. ./tdfs4ds.json (project-local)

  3. ~/.tdfs4ds/config.json (user-global)

  4. .env file — local environment overrides, searched in order:

  5. ./.env (project-local)

  6. ~/.tdfs4ds/.env (user-global) Only TDFS4DS_* variables are read. Module-config vars are set on the tdfs4ds module; pass-through vars (connection TDFS4DS_TD_*, serve toggles TDFS4DS_SERVE_*, TDFS4DS_SKILLS_FOLDER) are exported to os.environ because the serve layer reads them from there directly. Unlike the JSON config, credentials are accepted here (add .env to .gitignore to keep secrets out of source control).

  7. OS environment variables — prefix TDFS4DS_, e.g. TDFS4DS_SCHEMA, TDFS4DS_SCHEMA_VIEW, TDFS4DS_DATA_DOMAIN.

Credentials (INSTRUCT_MODEL_API_KEY) are rejected from the JSON config file to prevent accidental commits; use a .env file or an OS env var.

Full priority chain (highest first): programmatic > OS env var > .env file > JSON config > built-in defaults

logger module-attribute 

logger = getLogger(__name__)

_BOOL_VARS module-attribute 

_BOOL_VARS = frozenset({'DISPLAY_LOGS', 'DEBUG_MODE', 'USE_VOLATILE_TABLE', 'BUILD_DATASET_AT_UPLOAD', 'INSTRUCT_MODEL_VERIFY_SSL', 'MCP_TOOL_DISCOVERY', 'QUERY_BAND', 'AGENTIC_FEATURES', 'UPLOAD_DOCUMENT', 'UPLOAD_EXPLAIN', 'UPLOAD_OPTIMIZE_QUERY', 'UPLOAD_VECTOR_INDEXING', 'UPLOAD_CASCADE_REDOCUMENTATION', 'OPIK_VENDOR_TELEMETRY', 'PROCESS_SQL_VERSIONING', 'TIME_MANAGER_DETECT', 'EXPOSED_VIEW_AUTO_ATTACH', 'EXECUTION_RESOURCES', 'LOG_COLOR', 'LOG_PROPAGATE'})

_NULLABLE_BOOL_VARS module-attribute 

_NULLABLE_BOOL_VARS = frozenset({'LOG_COLOR'})

_INT_VARS module-attribute 

_INT_VARS = frozenset({'VARCHAR_SIZE', 'FEATURE_PARTITION_N', 'FEATURE_PARTITION_EACH', 'EMBEDDING_MODEL_DIM', 'EMBEDDING_MODEL_MAX_TOKENS', 'CHROMA_PORT', 'SCHEDULING_MAX_WORKERS', 'MCP_MAX_TOOLS', 'MCP_FALLBACK_MAX_TOOLS', 'EMBEDDING_BATCH_SIZE', 'CREDENTIAL_RENEWAL_ATTEMPTS'})

_NULLABLE_INT_VARS module-attribute 

_NULLABLE_INT_VARS = frozenset({'SCHEDULING_MAX_WORKERS'})

_FLOAT_VARS module-attribute 

_FLOAT_VARS = frozenset({'EMBEDDING_BATCH_DELAY'})

_LIST_VARS module-attribute 

_LIST_VARS = frozenset({'SCHEMA_READONLY'})

_DICT_VARS module-attribute 

_DICT_VARS = frozenset({'MCP_SERVER_HEADERS'})

_FILE_VARS module-attribute 

_FILE_VARS = frozenset({'SCHEMA', 'SCHEMA_VIEW', 'SCHEMA_READONLY', 'DATA_DOMAIN', 'FEATURE_STORE_TIME', 'DISPLAY_LOGS', 'LOG_COLOR', 'LOG_PROPAGATE', 'DEBUG_MODE', 'STORE_FEATURE', 'REGISTER_FEATURE', 'REGISTER_PROCESS', 'USE_VOLATILE_TABLE', 'BUILD_DATASET_AT_UPLOAD', 'QUERY_BAND', 'AGENTIC_FEATURES', 'UPLOAD_DOCUMENT', 'UPLOAD_EXPLAIN', 'UPLOAD_OPTIMIZE_QUERY', 'UPLOAD_VECTOR_INDEXING', 'UPLOAD_CASCADE_REDOCUMENTATION', 'VARCHAR_SIZE', 'FEATURE_PARTITION_N', 'FEATURE_PARTITION_EACH', 'SCHEDULING_MAX_WORKERS', 'TIME_MANAGER_CHECK', 'PROCESS_SQL_VERSIONING', 'TIME_MANAGER_DETECT', 'EXPOSED_VIEW_AUTO_ATTACH', 'EXECUTION_RESOURCES', 'INSTRUCT_MODEL_PROVIDER', 'INSTRUCT_MODEL_URL', 'INSTRUCT_MODEL_MODEL', 'INSTRUCT_MODEL_REGION', 'INSTRUCT_MODEL_AZURE_DEPLOYMENT', 'INSTRUCT_MODEL_AZURE_API_VERSION', 'INSTRUCT_MODEL_GCP_PROJECT', 'INSTRUCT_MODEL_GCP_LOCATION', 'EMBEDDING_MODEL_REGION', 'EMBEDDING_MODEL_AZURE_DEPLOYMENT', 'EMBEDDING_MODEL_AZURE_API_VERSION', 'EMBEDDING_MODEL_GCP_PROJECT', 'EMBEDDING_MODEL_GCP_LOCATION', 'INSTRUCT_MODEL_CLIENT_MODE', 'EMBEDDING_MODEL_CLIENT_MODE', 'INSTRUCT_MODEL_PROXY', 'INSTRUCT_MODEL_VERIFY_SSL', 'INSTRUCT_MODEL_CA_BUNDLE', 'STRUCTURED_OUTPUT_METHOD', 'EMBEDDING_MODEL_PROVIDER', 'EMBEDDING_MODEL_URL', 'EMBEDDING_MODEL_MODEL', 'EMBEDDING_MODEL_PROXY', 'EMBEDDING_MODEL_DIM', 'EMBEDDING_MODEL_MAX_TOKENS', 'EMBEDDING_BATCH_DELAY', 'EMBEDDING_BATCH_SIZE', 'CREDENTIAL_RENEWAL_ATTEMPTS', 'CHROMA_MODE', 'CHROMA_PATH', 'CHROMA_HOST', 'CHROMA_PORT', 'VECTOR_STORE_BACKEND', 'TERADATA_VECTOR_STORE_PREFIX', 'TERADATA_AI_API_TYPE', 'TERADATA_AI_MODEL_NAME', 'TERADATA_AI_REGION', 'TERADATA_AI_API_BASE', 'MCP_SERVER_URL', 'MCP_SERVER_TRANSPORT', 'MCP_SERVER_EDITION', 'MCP_AUTH', 'MCP_USERNAME', 'MCP_SERVER_CA_BUNDLE', 'MCP_MAX_TOOLS', 'MCP_TOOL_DISCOVERY', 'MCP_FALLBACK_MAX_TOOLS', 'TRACING', 'OPIK_PROJECT_NAME', 'OPIK_VENDOR_TELEMETRY'})

_CREDENTIAL_VARS module-attribute 

_CREDENTIAL_VARS = frozenset({'INSTRUCT_MODEL_API_KEY', 'EMBEDDING_MODEL_API_KEY', 'TERADATA_AI_ACCESS_KEY', 'MCP_PASSWORD', 'MCP_SERVER_HEADERS'})

_PASSTHROUGH_ENV_VARS module-attribute 

_PASSTHROUGH_ENV_VARS = frozenset({'TD_HOST', 'TD_USERNAME', 'TD_PASSWORD', 'TD_LOGMECH', 'TD_DATABASE', 'TD_ENCRYPT', 'TD_LOGDATA', 'SERVE_PORT', 'SERVE_BUILD_INDEX', 'SERVE_CHATBOT', 'SERVE_CHATBOT_PORT', 'SERVE_DOCS', 'SERVE_DOCS_PORT', 'SERVE_ROOT_PATH', 'SERVE_MCP_URL', 'SERVE_DEFAULT_VIEW', 'SKILLS_FOLDER', 'VECTOR_STORE_AUTH_BASE_URL', 'VECTOR_STORE_AUTH_AUTH_TOKEN', 'VECTOR_STORE_AUTH_PAT_TOKEN', 'VECTOR_STORE_AUTH_PEM_FILE', 'VECTOR_STORE_AUTH_USERNAME', 'VECTOR_STORE_AUTH_PASSWORD', 'VECTOR_STORE_AUTH_CLIENT_ID'})

_ALL_ENV_VARS module-attribute 

_ALL_ENV_VARS = _FILE_VARS | _CREDENTIAL_VARS

_SECRET_SUFFIXES module-attribute 

_SECRET_SUFFIXES = ('PASSWORD', 'TOKEN', 'API_KEY', 'ACCESS_KEY', 'SECRET', 'LOGDATA')

PLACEHOLDER module-attribute 

PLACEHOLDER = '<set-this>'

_JSON_CANDIDATES module-attribute 

_JSON_CANDIDATES = [Path('tdfs4ds.json'), home() / '.tdfs4ds' / 'config.json']

_DOTENV_CANDIDATES module-attribute 

_DOTENV_CANDIDATES = [Path('.env'), home() / '.tdfs4ds' / '.env']

_coerce 

_coerce(name, value)

Cast value to the correct Python type for name.

_serialize 

_serialize(value)

Render value the way _coerce will read it back.

_is_placeholder 

_is_placeholder(value)

True when value is a generator placeholder rather than a real value.

_is_secret 

_is_secret(attr)

True when attr names a credential that must never be written in clear.

_apply_file 

_apply_file(module, path)

Read path (JSON) and apply matching keys to module.

_strip_trailing_comment 

_strip_trailing_comment(value)

Drop a trailing # comment from an UNQUOTED .env value.

Only whitespace followed by # starts a comment, so a # inside a value (common in generated passwords) survives. Quote the value to keep a literal " #".

_needs_quoting 

_needs_quoting(value)

True when leaving value bare would change what the parser reads back.

Lives next to :func:_parse_dotenv on purpose: the parser defines the rule this has to satisfy, and a renderer that drifts from its own parser produces a file that looks right and authenticates with something else.

_render_dotenv_value 

_render_dotenv_value(value, *, force_quote=False)

Render one value for a .env line, quoting when it matters.

There is no escape syntax in :func:_parse_dotenv, so a value containing both quote characters cannot be represented; that case is written bare with a warning rather than silently corrupted.

force_quote is for values whose shape is not known in advance (a credential, typically): quoting one that did not need it is harmless, while failing to quote one that did truncates it at the first " #".

_parse_dotenv 

_parse_dotenv(path)

Parse path as a .env file and return a {ATTR: raw_value} dict for every TDFS4DS_* variable found.

Handles: - KEY=VALUE - KEY="VALUE" / KEY='VALUE' - export KEY=VALUE - # comments (whole-line and trailing) and blank lines

Trailing-comment rule, matching python-dotenv so a file behaves the same whichever loader reads it: in an unquoted value, whitespace followed by # starts a comment and is dropped. A # with no whitespace before it is part of the value (PASSWORD=abc#def is abc#def), and quoting keeps everything verbatim (PASSWORD="abc #def") — which is how to write a value that genuinely contains " #".

This used to keep the comment: PASSWORD=secret # prod became the literal secret # prod, which then failed authentication somewhere far away from the file that caused it.

_apply_dotenv 

_apply_dotenv(module, path)

Parse path as a .env file and apply TDFS4DS_* entries.

Module-config vars are set on module; pass-through vars (connection / serve / skills — read straight from os.environ by the serve layer) are exported to os.environ so a .env works regardless of how it is loaded.

_apply_env 

_apply_env(module)

Apply TDFS4DS_* OS environment variables to module.

_load_config 

_load_config(module)

Called automatically during import tdfs4ds.

Load order (lowest → highest priority): JSON config file → .env file → OS env vars

export_config 

export_config(path=None, *, fmt='json', source='module', include_secrets=False)

Capture the current configuration as a JSON config file or a .env.

Two uses, selected by source:

  • source='module' (default) — snapshot the running configuration (tdfs4ds.<ATTR>), including anything set in-session. Use it to reproduce a session that works.
  • source='env' — convert the TDFS4DS_* environment variables into a file, whether or not they ever reached the module config.
Parameters

path : str or pathlib.Path, optional Where to write. When None nothing is written and the text is only returned — inspect it before committing it to disk. fmt : {'json', 'env'} 'json' writes a tdfs4ds.json that load_config() reads; 'env' writes TDFS4DS_* lines for a .env / container env-file. Only 'env' can carry connection and Vector Store auth variables — those are pass-through, never module attributes, so fmt='json' cannot express them. include_secrets : bool, default False Secrets (anything whose name contains PASSWORD / TOKEN / API_KEY / ACCESS_KEY / SECRET) are written commented out with a placeholder, so the result is a shareable template. Pass True to write the real values — then treat the file as a secret. Ignored for fmt='json': _apply_file rejects credentials from JSON by design, so they are never written there.

Returns

str The file content, whether or not path was given.

Examples

tdfs4ds.export_config() # inspect # doctest: +SKIP tdfs4ds.export_config('tdfs4ds.json') # reusable config tdfs4ds.export_config('.env', fmt='env', source='env') # env -> file

patch_dotenv 

patch_dotenv(path, updates, *, comment=None, allow_secrets=False)

Set individual TDFS4DS_* variables in an existing .env, in place.

:func:export_config and :func:tdfs4ds.generate_ai_studio_env both rewrite the whole file, which loses its comments, its ordering and the commented-out alternatives they deliberately put there. This changes one line at a time, so a probed value can be recorded into a file a human maintains.

It is line-based rather than parse-and-rewrite because :func:_parse_dotenv is lossy for this purpose — it drops comments, ordering and quoting, so it cannot serve as the read side of a read-modify-write.

Which line is changed, and why:

  • The last uncommented occurrence. _parse_dotenv assigns as it reads, so on a duplicate the last one is what takes effect; patching any other would leave the file saying one thing and meaning another.
  • Otherwise the first commented-out slot (#TDFS4DS_X=) is uncommented in place. A generated file offers such slots precisely so that turning one on is an edit.
  • Otherwise appended, preceded by comment if given.

A commented slot is never uncommented while a live line exists — a generated file carries commented alternatives next to the live choice (two MCP_SERVER_URL lines, for instance), and enabling one of those would silently switch the setting to the alternative.

Parameters

path : str or Path An existing .env. Creating one is :func:export_config's or :func:tdfs4ds.generate_ai_studio_env's job, not this function's, so a missing file raises rather than being conjured half-populated. updates : dict {'STRUCTURED_OUTPUT_METHOD': 'function_calling'} — keys with or without the TDFS4DS_ prefix. Values are rendered the way :func:_coerce reads them back, so None is written as the literal None (which loads as None), not as an empty line. comment : str, optional Provenance note, written as a trailing comment on the line itself. Say where the value came from and when: a pinned setting that was right in one session is wrong in another, and a stale pin should be visible rather than silent.

Trailing rather than above, because the whole line is rewritten on each
call — so re-pinning replaces the note instead of stacking a new one on
every run. It is dropped for a value that has to be quoted: the parser
takes a quoted value verbatim, so a trailing comment would become part
of it. Newlines are folded to "; " for the same reason.

allow_secrets : bool, default False Required to write a credential. A general-purpose file patcher must not be the thing that puts a password into a file someone later commits, so by default a name :func:_is_secret recognises is refused.

Returns

dict {'updated': [...], 'added': [...], 'unchanged': [...]} of full TDFS4DS_* names. Nothing is written when everything is unchanged.

Raises

ValueError path does not exist, a key is not a registered variable (it would only earn an "unknown .env variable" warning on load), or it is a credential without allow_secrets.

describe_config 

describe_config(*, source='env', required=None, display=True)

Show the configuration with secret VALUES hidden but their presence confirmed.

The counterpart to :func:export_config for the question you actually ask when a deployment misbehaves: is this variable set? export_config cannot answer it — it redacts a secret to a commented placeholder, which reads identically whether the variable holds a password or nothing at all.

Never returns or prints a secret value. A secret reports its length instead, which is enough to spot a truncated paste or a stray quote and not enough to reconstruct. "Is this a credential?" is decided by :func:_is_secret — the one authority for it — never re-derived here.

Parameters

source : {'env', 'module'} 'env' (default) reads the TDFS4DS_* environment variables, so it reports what is set even when it never reached the module config (the load-order trap: config.py populates tdfs4ds.<ATTR> once, at import). 'module' snapshots the running configuration. Pass-through variables have no module attribute and are always read from the environment, whatever source says. required : iterable of str, optional Names to flag when missing, with or without the TDFS4DS_ prefix. An unset variable is normally omitted to keep the report readable; a required one is always shown, marked NOT SET (required). display : bool, default True Print the report. Set False to only receive the rows.

Returns

list of dict One row per reported variable: name (with prefix), attr, status ('set' / 'hidden' / 'unset'), value (the real value, or None when hidden), secret, required and length.

Examples

tdfs4ds.describe_config() # doctest: +SKIP tdfs4ds.describe_config(required=['TDFS4DS_TD_HOST']) # doctest: +SKIP rows = tdfs4ds.describe_config(display=False) # doctest: +SKIP

_print_config_rows 

_print_config_rows(rows, source)

Render :func:describe_config's rows. Plain ASCII, by convention.

A diagnostic must never be the thing that breaks: a legacy Windows console raises UnicodeEncodeError on non-ASCII, so nothing here uses it.

load_config 

load_config(path=None, dotenv_path=None)

(Re-)load configuration from external sources.

Useful for pointing at non-default config files or refreshing values mid-session. Applies sources in priority order (lowest first):

  1. JSON config file (at path or auto-discovered)
  2. .env file (at dotenv_path or auto-discovered)
  3. OS TDFS4DS_* environment variables (always applied)
Parameters

path : str or pathlib.Path, optional Explicit path to a JSON config file. Skips auto-discovery when given. dotenv_path : str or pathlib.Path, optional Explicit path to a .env file. Skips auto-discovery when given.

Examples

Auto-discover all sources::

import tdfs4ds
tdfs4ds.load_config()

Explicit paths::

tdfs4ds.load_config(
    path='/project/configs/feature_store.json',
    dotenv_path='/project/.env.local',
)

JSON config format (tdfs4ds.json)::

{
    "schema": "MY_DATABASE",
    "data_domain": "MY_PROJECT",
    "display_logs": true,
    "store_feature": "MERGE",
    "varchar_size": 1024,
    "instruct_model_provider": "openai",
    "instruct_model_model": "gpt-4o"
}

Keys are case-insensitive. instruct_model_api_key is rejected from JSON config — use a .env file or OS env var instead.

.env file format::

TDFS4DS_SCHEMA=MY_DATABASE
TDFS4DS_SCHEMA_VIEW=MY_VIEWS_DATABASE
TDFS4DS_DATA_DOMAIN=MY_PROJECT
TDFS4DS_INSTRUCT_MODEL_API_KEY=sk-...

Only TDFS4DS_* variables are read from the .env file. Add .env to .gitignore to keep secrets out of source control.