tdfs4ds.agent.vector_store_teradata¶
Teradata Enterprise Vector Store backend for the tdfs4ds business dictionary.
Provides the same public surface as the Chroma backend
(:func:tdfs4ds.agent.vector_index.build_vector_index and
:func:tdfs4ds.agent.vector_index.search_vector_index) but persists the three
collections as Teradata vector stores via langchain_teradata:
{prefix}_objects->FS_BUSINESS_DICTIONARY_OBJECTS{prefix}_sections->FS_BUSINESS_DICTIONARY_SECTIONS{prefix}_columns->FS_BUSINESS_DICTIONARY_COLUMNS
Embeddings are produced by
:func:tdfs4ds.agent.embedding.get_vector_store_embedding which selects either
teradatagenai.TeradataAI (when tdfs4ds.TERADATA_AI_API_TYPE is set) or
the standard LangChain Embeddings — TeradataVectorStore accepts both at the
embedding= argument.
Indexing source (v0.3.1.131): documents are supplied to
TeradataVectorStore.from_datasets/add_datasets reading from a
permanent CURRENT VALIDTIME view (FS_V_BUSINESS_DICTIONARY_*,
created by genai.documentation.documentation_tables_creation()) — NOT
via from_documents/add_documents (LangChain Document list
input). The Document-list path has a confirmed bug: it internally
stages the input through teradataml.DataFrame.from_dict(..., persist=True)
/ DataFrame.materialize(), which takes no database argument and may
materialize as a volatile table (session-scoped) or a view — either way
teradataml itself documents this choice as internal/opaque
("decided and taken care by teradataml"). A volatile table is invisible to
the separate Vector Store REST session, causing
TDML_2412: Object ... does not exist regardless of target_database
(which only scopes where the FINAL collection metadata is registered, not
where the input data is read from).
Deeper pitfall found on live VCL2 testing (v0.3.1.14x): reading from the
permanent view is necessary but not sufficient on its own — the way the
view is handed to from_datasets/add_datasets matters too.
_from_datasets_kwargs() used to wrap the view in
tdml.DataFrame.from_query("SELECT ... FROM <view>") before v0.3.1.14x.
Confirmed live: DataFrame.from_query(...) sets _table_name to a
freshly materialized, ephemeral ml___frmqry_v_<id> view at
construction time (before teradatagenai is even called), and
teradatagenai bakes whatever _table_name it receives permanently
into the collection's _inputdata view DDL (confirmed via SHOW VIEW
on a live collection: ... FROM "data_scientist"."ml___frmqry_v_<id>").
Once Vantage garbage-collects that ephemeral view (observed to happen
almost immediately — confirmed absent from DBC.TablesV within the same
session that created it), _inputdata — and collectionV_<name>,
which joins through it — is permanently broken: every query touching
it, from any client (similarity_search, a plain teradataml query, an
unrelated MCP session), fails with Error 3807 / TDML_2412: Object ...
does not exist, forever, not transiently (confirmed: 3/3 retries fail
identically). The fix: data= is now passed as a plain fully-qualified
"schema.view" string naming the permanent view directly (see
_from_datasets_kwargs) — never a from_query() result, which
guarantees no ephemeral materialization ever happens.
No row-level filtering of data= — ever. Neither from_datasets
nor add_datasets exposes any filter/predicate/where-clause parameter
(checked directly against the teradatagenai signatures/docstrings).
Pointing data= at a filtered view instead of the full source view was
considered and rejected: since _inputdata is a live view (re-evaluated
on every query, not a frozen snapshot), a filtered data= would
permanently narrow what the whole collection can ever return — hiding
previously-indexed rows outside the filter from search forever. Separately,
_index (the real embeddings table) is keyed by TD_ID, which
_inputdata assigns via ROW_NUMBER() OVER (ORDER BY <text column>) —
recomputed fresh on every evaluation of whatever data= points to, so any
change in row count/ordering between the _index write and a later
_inputdata evaluation risks silently misaligning embeddings with the
wrong row's text/metadata. Always pass the entire, unfiltered source view.
module-attribute
¶
_SERVICE_DOWN_MSG = "Teradata Vector Store service is unreachable or not enabled on this system (VECTOR_STORE_BACKEND='teradata'). Check that the Enterprise Vector Store feature is licensed/activated on this Vantage instance. Detail: {detail}"
module-attribute
¶
_NOT_CONFIGURED_MSG = "Teradata Vector Store client has no base URL configured (VECTOR_STORE_BACKEND='teradata'). tdfs4ds automatically calls teradataml.set_auth_token(...) during connect()/setup() when one of the following TDFS4DS_VECTOR_STORE_AUTH_* environment variable groups is set: BASE_URL+AUTH_TOKEN (JWT, tried first — a token you already hold; in-cluster, map the platform's injected token onto it), BASE_URL+PAT_TOKEN+PEM_FILE (PAT), BASE_URL+USERNAME+PASSWORD (Basic), or BASE_URL+CLIENT_ID (OAuth). Reaching this error means either none of those groups were configured, or set_auth_token itself failed — check the connect()/setup() log for a 'Vector Store auth ... failed' warning line for the underlying reason. Detail: {detail}"
module-attribute
¶
_AUTH_REJECTED_MSG = "The Teradata Vector Store service REJECTED THE CREDENTIAL (SQLState 28000 — invalid authorization). The service itself is reachable and enabled: it answered, then failed to open a database session on your behalf. This is NOT a licensing/activation problem. Most likely causes, in order: (1) the token behind TDFS4DS_VECTOR_STORE_AUTH_AUTH_TOKEN has EXPIRED — a platform-injected OAuth/JWT access token is short-lived, and teradataml stores a ready-made token WITHOUT validating it, so set_auth_token() and a health probe both still succeed with a stale one; re-read the current token and call tdfs4ds.setup()/connect() again (or tdfs4ds.utils.vector_store_auth.configure_vector_store_auth()). (2) For the Basic group, TDFS4DS_VECTOR_STORE_AUTH_USERNAME/_PASSWORD are wrong for this system. (3) The database user cannot log on the way the service logs on for it — teradataml's own set_auth_token documentation requires the privilege to log in with a NULL password (see GRANT LOGON in the Teradata documentation). Detail: {detail}"
module-attribute
¶
_BROKEN_COLLECTION_MSG = "Teradata Vector Store collection '%s' is broken: its backing object is missing (%s). It cannot be searched or refreshed, and recreating it is the remedy, but this session could not destroy it (destroy() needs admin rights). Ask an admin to run tdfs4ds.agent.vector_store_teradata.reset_teradata_store(['%s']), or use tdfs4ds.VECTOR_STORE_BACKEND = 'chroma' meanwhile."
module-attribute
¶
_COLLECTION_FAILED_MSG = "Teradata Vector Store collection '{name}' is in a failed state ('{status}') and will not recover on its own — retrying writes or searches against it will not help. Call tdfs4ds.agent.vector_store_teradata.reset_teradata_store(sources=['{source}']) (requires admin privileges) to destroy and recreate it from the current business dictionary, then rebuild the index."
module-attribute
¶
_SOURCE_ALREADY_ATTACHED_MSG = "build_teradata_index: '%s' already has its business-dictionary source attached, so add_datasets() added nothing — the collection is unchanged and its existing rows stay searchable. Business-dictionary rows documented since it was created are NOT in the index. The Enterprise Vector Store offers no admin-free way to resync an attached source (see build_teradata_index's KNOWN LIMITATION), so to pick them up call build_vector_index(force_rebuild=True) — or tdfs4ds.agent.vector_store_teradata.reset_teradata_store(sources=['%s']) directly. Both destroy and recreate the collection and require admin privileges."
module-attribute
¶
_RESYNC_FAILED_MSG = "build_teradata_index: could not refresh '%s'. The collection is intact and its existing rows stay searchable, but business-dictionary rows documented since it was last built are NOT in the index. Both mechanisms declined: %s. The usual cause of the second one is missing ADMIN rights — Collection.destroy() is admin-only, so a non-admin session cannot perform the full refresh that the incremental path fell back to. Ask an admin to run tdfs4ds.agent.vector_store_teradata.reset_teradata_store(sources=['%s']), or run build_vector_index(force_rebuild=True) from an admin session."
module-attribute
¶
_SOURCE_VIEW_SPEC = {'objects': ('DOCUMENTATION_BUSINESS_DICTIONARY_OBJECTS_VIEW', 'BUSINESS_DESCRIPTION', ['DATABASE_NAME', 'OBJECT_NAME', 'OBJECT_TYPE']), 'sections': ('DOCUMENTATION_BUSINESS_DICTIONARY_SECTIONS_VIEW', 'SECTION_CONTENT', ['DATABASE_NAME', 'OBJECT_NAME', 'SECTION_NAME']), 'columns': ('DOCUMENTATION_BUSINESS_DICTIONARY_COLUMNS_VIEW', 'BUSINESS_DESCRIPTION', ['DATABASE_NAME', 'TABLE_NAME', 'COLUMN_NAME'])}
module-attribute
¶
_SOURCE_DESCRIPTIONS = {'objects': 'tdfs4ds business dictionary — object-level (table/view/dataset) descriptions.', 'sections': 'tdfs4ds business dictionary — dataset documentation sections (overview, entity, feature themes, business questions, intended audience).', 'columns': 'tdfs4ds business dictionary — column-level descriptions.'}
module-attribute
¶
_TEXT_KEYS = ('content', 'page_content', 'text', 'document', 'business_description', 'section_content')
Return the embedding object for the active vector-store backend.
Selection rule
tdfs4ds.TERADATA_AI_API_TYPEset ->teradatagenai.TeradataAI(...)(in-database embedding; used by the'teradata'backend).- otherwise ->
get_embeddings()(LangChain Embeddings; works with vLLM / TEI / OpenAI-compatible setups and is also accepted asembedding=byTeradataVectorStore).
The ChromaDB backend always uses get_embeddings() regardless of this
setting — TeradataAI is a Vantage-side construct and is only meaningful
when the vector store also lives in Vantage.
True when exc is the Vector Store service reporting that the DATABASE
rejected the credential it presented on the caller's behalf — e.g.
Failed to execute collections_list. Response Code: 500, Message:
[SQLState 28000] The UserId, Password or Account is invalid.
SQLState 28000 is the SQL standard's "invalid authorization specification":
a credential rejection, nothing to do with reachability or licensing. It
must be checked BEFORE _reraise_if_service_down's generic
"response code:" branch, which would otherwise report it as
"service unreachable or not enabled ... check licensing/activation" and
send the user hunting for a feature flag while the real fix is a fresh
token (see _AUTH_REJECTED_MSG).
Deliberately NOT gated on TeradataMlException: the same rejection can
surface through other wrappers, and both markers matched here are specific
enough that a false positive would have to be quoting a DB auth error
verbatim.
True when exc is the transient "Collection ... cannot be updated as it
is not in a ready state" 409 — the collection is still processing a prior
async operation (its own creation, or a previous update), NOT a real
failure. Distinguished from a generic 409/HTML service-down error (which
also contains "Response Code:") by requiring the more specific
"not in a ready state" phrase; must be checked BEFORE
_reraise_if_service_down's generic "response code:" branch, which
would otherwise misclassify this transient, retryable condition as
"service unreachable/unlicensed."
True when exc is a 400 "Object ... does not exist" from
similarity_search/ask — confirmed live: the Enterprise Vector
Store REST backend materializes a short-lived per-query staging view
(ml___frmqry_v_<id>) to run the search, and that object can be gone
(dropped/never-committed/session mismatch) by the time the search reads
it back — a platform-side race, not a licensing/activation problem. Must
be checked BEFORE _reraise_if_service_down's generic
"response code:" branch, which otherwise misclassifies this as
"service unreachable/unlicensed" (every TeradataMlException from this
API embeds "Response Code:", so that check alone can't distinguish a
genuine gateway failure from this per-query staging race).
True when exc is a bare HTTP 5xx gateway error (502/503/504) —
confirmed live on a VCL2 test instance: the Vector Store REST gateway
occasionally returns a raw <html>...502 Bad Gateway...</html> body
(or 503/504) with no relation to any code path or configuration issue —
pure transient infrastructure flakiness that resolves on retry within
seconds. Distinguished from a genuine "service unreachable/unlicensed"
condition only by being retryable: unlike a persistent misconfiguration
(missing auth, wrong URL, licensing), a gateway 502/503/504 is by
definition a transient proxy/gateway-layer failure, so retrying a few
times before giving up is safe and appropriate. Checked BEFORE
_reraise_if_service_down's generic "<html"/"response code:"
branch so a transient gateway blip doesn't immediately surface as the
terminal _SERVICE_DOWN_MSG on the very first retry-able occurrence.
True when exc is the 400 the Vector Store returns from
add_datasets() because the source object is ALREADY attached to the
collection — Failed to execute collection. Response Code: 400, Message:
You can only add object names that do not exist in the collection.
This is the server saying "nothing to add", not a failure of the service.
_from_datasets_kwargs() always names the same
FS_V_BUSINESS_DICTIONARY_* view, so EVERY second-and-later
build_teradata_index() call reaches it once the collection exists (see
that function's KNOWN LIMITATION) — this is an expected steady-state
condition, not an edge case.
Must be checked BEFORE _reraise_if_service_down's generic
"response code:" branch: matching on that alone reported this benign,
expected condition as "service unreachable or not enabled ... check
licensing/activation", sending the user hunting for a feature flag on a
system whose Vector Store had just answered the request. Same
misclassification shape as _is_auth_rejected_error (SQLState 28000) and
_is_not_ready_error (the 409) — every TeradataMlException from this
API embeds "Response Code:", so that substring can never distinguish a
genuine gateway failure from an in-service one.
Deliberately NOT gated on TeradataMlException (same reasoning as
_is_auth_rejected_error): the marker matched here is a full sentence
specific to this API, so a false positive would have to be quoting it
verbatim.
True when the collection's OWN backing object is gone (seen live, 0.3.2.1):
Response Code: 500, Message: Object 'collection_<name>_index' does not exist.
The collection is listed and answers, but its _index / _inputdata
table or its collectionV_ view no longer exists (dropped with the
database objects, or a create that never finished), so it can neither be
searched nor refreshed. Waiting or retrying changes nothing; recreating it
loses nothing. It used to fall into the generic "Response Code:" branch and
be reported as "service unreachable or not enabled ... check licensing" -
the fourth misclassification of that shape (after the 409, 28000 and the
attached-source 400). The match is on the collection's own object names, so
the per-query staging race (ml___frmqry_v_*, a 400) is not caught here.
_retry_if_transient_gateway_error(fn, label: str, timeout_seconds: float = 60.0, poll_interval: float = 5.0)
Call fn(), retrying while it raises a transient gateway 5xx (see
_is_transient_gateway_error). Re-raises the original exception once
timeout_seconds elapses, or immediately for any other exception shape.
A short default timeout (60s, vs. the 120s/1800s used for genuine async
"not ready" waits) is deliberate — a gateway 502/503/504 either clears in
a few seconds or reflects a real outage, in which case failing fast and
surfacing the clear _SERVICE_DOWN_MSG (via the caller's own
_reraise_if_service_down) is more useful than a long silent wait.
Inspect exc and raise a clear RuntimeError when it looks like either:
- the Vector Store client itself is unconfigured (e.g.
requests'MissingSchema/InvalidURLfrom a request built with aNonebase URL — "Invalid URL 'None/api/v2/...': No scheme supplied" — meaningset_auth_token/session auth was never set up). This is distinguished from "service down" because the fix is entirely client-side (callset_auth_token), not a licensing/activation problem on the Vantage instance. - the service answered but the DATABASE rejected the credential it
presented (SQLState 28000, see
_is_auth_rejected_error) — reported as an auth problem naming the token/credential variables, never as "unreachable or not enabled", because the fix is a fresh credential and not a licensing check. - the Vector Store REST gateway itself is unreachable/not enabled (any other transport-level failure, or a reachable-but-wrong endpoint returning a raw HTML/status body — e.g. a gateway 502 — instead of a JSON error).
Returns normally (does nothing) for any other exception, leaving it to
the caller — in particular for the transient "not in a ready state" 409
(see _is_not_ready_error), the transient per-query staging-object
race (see _is_transient_staging_object_error), and a transient
gateway 502/503/504 (see _is_transient_gateway_error), which callers
should retry rather than treat as fatal, so none of the three is ever
raised as service-down here; and for the benign "source already attached"
400 (see _is_source_already_attached_error), which is terminal but
means "nothing to add" and is handled by the caller.
Shared by every call site that talks to teradatagenai/langchain_teradata
directly (_lookup_store_database, build_teradata_index's
from_datasets/add_datasets/status calls) so neither failure mode
is ever mistaken for "store doesn't exist yet, fall back to rebuild" — which
would just fail again later with a much less clear message.
Import and instantiate teradatagenai's CollectionManager.
Raises a clear RuntimeError (mirroring _import_vector_store) if the
package is missing — the Teradata backend depends on it transitively via
langchain_teradata, so a missing import here means a broken install.
Return {"target_database": ..., "collection_status": ...} for a
named vector store, or None if no store with that name exists.
target_database (cross-store federation, v0.3.1.148): when given, a row
matches only if its own target_database also matches — because two feature
stores using the same TERADATA_VECTOR_STORE_PREFIX have identically-named
collections (e.g. tdfs4ds_objects) in different databases, and a name-only
match would return the wrong store's status. Default None ⇒ name-only match
(unchanged for existing callers).
Uses CollectionManager.list() — the same list-and-match pattern
teradatagenai uses internally (Collection.exists,
VectorStore.__init__) — so a caller can discover where a store lives
before attaching, instead of assuming it matches tdfs4ds.SCHEMA. This
catches the case where a store was created before a target_database fix
(or a schema change) and now sits in a different database than expected.
collection_status is the authoritative signal for whether a write
actually succeeded — confirmed live to take values like 'ready',
'update_failed', 'create_failed' (see _is_failed_status). A
write call (from_datasets/add_datasets) returning without an
exception only means "the request was accepted" (these operations are
asynchronous) — it does NOT mean the operation succeeded; only
collection_status reflects the true server-side outcome.
Raises RuntimeError (via _raise_service_unavailable) when the
Vector Store service itself is unreachable/not enabled — distinguished from
"store not found", which returns None normally. A transient gateway
502/503/504 (confirmed live on a VCL2 test instance — pure infrastructure
flakiness, unrelated to code/config) is retried a few times first via
_retry_if_transient_gateway_error before falling through to that
service-down classification.
Backward-compatible wrapper over :func:_lookup_store_info returning
just the target_database (existing callers only need this).
True when status (a collection_status value from
:func:_lookup_store_info) is a terminal failure — e.g. update_failed,
create_failed. Matches any ..._failed suffix rather than
hardcoding the two observed strings, in case other failure variants exist.
A failed collection will NOT recover by waiting or retrying — it requires
an explicit :func:reset_teradata_store (destroy + recreate).
Build the data/data_columns/key_columns/metadata_columns
kwargs for TeradataVectorStore.from_datasets/add_datasets.
data is passed as a plain fully-qualified "schema.view" string
pointing directly at the permanent FS_V_BUSINESS_DICTIONARY_* view —
NOT a tdml.DataFrame.from_query(...) result. Confirmed live:
DataFrame.from_query(...) sets _table_name to a freshly
materialized, ephemeral ml___frmqry_v_<id> view at construction
time — before teradatagenai ever sees it — and teradatagenai
bakes whatever _table_name it receives permanently into the
collection's _inputdata view DDL. Once Vantage garbage-collects that
ephemeral view (observed to happen almost immediately), _inputdata
(and anything joining through it, including similarity_search) is
permanently broken (see module docstring). The view already exposes
exactly the columns needed, so no projection/SELECT is needed here —
data_columns/key_columns/metadata_columns tell
from_datasets/add_datasets which columns of the named view to
use.
Cheap existence check — avoids calling from_datasets/add_datasets
on an empty source (mirrors the old if not docs: continue guard).
target_database= kwarg for from_datasets/add_datasets, scoped
to tdfs4ds.SCHEMA — where the Vantage REST API registers the final
collection's metadata.
Destroy a named vector store if it already exists.
Looks up the store's ACTUAL database via CollectionManager.list()
first, rather than assuming it lives at tdfs4ds.SCHEMA — a store
created before a target_database/schema change can be sitting in a
different database, and attaching only at the expected database would
silently miss it, leaving an orphaned duplicate once a fresh store is
created elsewhere on force_rebuild.
Calls destroy() unconditionally once existence is confirmed — do NOT
gate on status() first. Collection.status() reports the async
create/destroy/update OPERATION history, not current existence; a store
that exists but has no reported operation history returns an empty
DataFrame, which previously made this function conclude "nothing to
destroy" and skip the call — leaving the real collection in place. The
subsequent from_datasets() call then collided with a
TeradataGenAIException: Collection ... already exists (Collection.create()
does its own authoritative self.exists check), silently caused by a
swallowed destroy() failure below (e.g. non-admin session — destroy()
requires admin privileges per its docstring) going unnoticed because this
function used to only log a warning and return.
A destroy() failure is therefore re-raised (not swallowed): silently
continuing after a failed destroy just defers the same collision to the
from_datasets()/add_datasets() call with a far less clear error.
Best-effort destroy of collection name for a force_rebuild=True
index build. Returns True only when the collection is CONFIRMED gone (the
caller may then treat the build as a fresh create); False, after logging a
warning, whenever it was not destroyed for any reason.
:func:build_teradata_index deliberately never destroys on its routine
path — destroy() is admin-only ("Only admin users can use this method"
per Collection.destroy()'s docstring), so keeping it out is what lets
a non-admin session index at all. But force_rebuild=True is an
EXPLICIT opt-in, and from 0.3.1.133 to 0.3.1.166 it did nothing
whatsoever: both branches ran identical code. A user who set it — which
they do precisely because the index looks stale — got the same no-op, and
once the source was attached, the same failure on the retry. Honouring it
here restores the flag's meaning.
A refused destroy is a warning, never fatal: the caller falls through to the ordinary reuse path, so a non-admin session ends up exactly where it was before this function existed rather than newly broken.
_wait_until_collection_ready(collection, name: str, timeout_seconds: float = 1800.0, poll_interval: float = 5.0) -> None
Poll collection.get_details() — a read-only method — every
poll_interval seconds until it succeeds, treating any exception as "the
collection is still processing a prior async operation" (Collection
create/update/destroy are all asynchronous: the server reports a request
as "accepted and in progress" well before the operation actually
completes, so a write issued right after can hit the collection while
it's still busy and fail with a 409 "not in a ready state").
Deliberately read-only: unlike retrying the write itself, this probe can't create additional server-side contention while checking. Raises the last exception observed if the collection is not ready within timeout_seconds (a genuinely stuck/failed operation should surface clearly rather than hang the caller forever).
_verify_write_succeeded(name: str, source: str, timeout_seconds: float = 300.0, poll_interval: float = 5.0, before: Optional[dict] = None, stale_polls: int = 3) -> None
After a write call (from_datasets/add_datasets) returns without
raising, confirm the operation actually completed by polling
collection_status (via :func:_lookup_store_info) until it reports
'ready'.
A write call returning cleanly only means "the request was accepted"
(Collection writes are asynchronous) — NOT that the operation
succeeded. Confirmed live: a store can log "indexed successfully" here
while collection_status is actually update_failed/
create_failed and the collection holds no searchable data.
If collection_status ever reports a *_failed status while
polling, raises immediately via :func:_raise_collection_failed (no
point waiting further — it's terminal). Raises a plain RuntimeError
on timeout if it never resolves to 'ready' or a failed status.
before is the collection's info (:func:_lookup_store_info) read just
before a write to an EXISTING collection. Its 'ready' status is still
reported for a moment after the write is accepted, so a 'ready' only counts
once last_updated has changed or a non-ready status has been seen. A
customer's log showed the race: a finalize pass logged "complete" seconds
before the collection turned update_failed. When the service reports
no last_updated, 'ready' is accepted after stale_polls polls.
Not needed after a create: the collection did not exist before.
Create the collection for source and confirm it really became ready.
Make an EXISTING collection reflect the current rows of its source view.
Returns (mechanism, collection) — the collection to use afterwards, which
is a NEW object when the fallback recreated it.
Incremental first, full refresh as the fallback.
update(data_refresh=True) is the documented way to "refresh/regenerate
vectors ... using existing backing/source tables", and that is exactly
tdfs4ds's situation: the attached object never changes (always the same
FS_V_BUSINESS_DICTIONARY_* view), only its rows do. It is cheap, needs
no admin rights, and preserves the collection.
Until 0.3.1.169 the only call tried here was add_datasets, which means
something else entirely: attach a NEW source object. It correctly refuses
an object that is already attached (400 "You can only add object names that
do not exist in the collection"), so on tdfs4ds's always-same-view source it
was guaranteed to fail on every second and later build — the reported
behaviour. It is not tried here at all any more: it cannot succeed in this
branch by construction, and it is not free (~50s per attempt observed live
before the 400 comes back).
data_refresh was investigated in 0.3.1.140 and rejected as unreliable —
it worked once, then repeatedly failed with "At least one of
collection_index, collection_parameters, ... must be provided for update".
That finding stands for the service version it was made on; it was never
evidence about newer ones, and a 2.2.17 service is reported to update
incrementally without trouble. Where it does fail, the fallback below still
gets the rows in, so an old service degrades instead of breaking.
The fallback is a full refresh: destroy and recreate the collection from
the same source. It needs admin rights (destroy() is admin-only) and
costs a full re-embed, which is why it is second — but it is the only
mechanism that is guaranteed to pick up new rows.
Whether unchanged rows get re-embedded is the service's business, not
ours. data_refresh's contract is only "refresh/regenerate vectors
... using existing backing/source tables"; it does not say whether it
skips rows whose text is unchanged, and the client offers no way to ask for
a delta. Verified against the installed teradatagenai: get_ids()
raises NotImplementedError ("Currently this is not being supported by
service"), and delete_by_ids keys on TD_ID, which _inputdata
recomputes as ROW_NUMBER() OVER (ORDER BY <text>) — a position, not
an identity, so a surgical delete can leave _index's frozen embeddings
aligned to different rows (the 0.3.1.140 corruption). So a hand-rolled
per-row incremental is expressible but unsafe, and is deliberately NOT
attempted: data_refresh is assumed to handle it, as the best mechanism
the service exposes.
Build or refresh the three Teradata vector stores.
Creates each collection on first use, and thereafter refreshes it in place
via update(data_refresh=True) — falling back to destroy + recreate only
when that does not apply (see :func:_resync_attached_collection).
destroy()
requires admin privileges ("Only admin users can use this method" per
Collection.destroy()'s own docstring) and previously ran as a routine
step of every force_rebuild=True call — a non-admin session, or any
other destroy failure, was silently swallowed, leaving the stale
collection in place so the following from_datasets() call then failed
with TeradataGenAIException: Collection ... already exists (raised by
Collection.create()'s own authoritative existence check). Reusing the
collection sidesteps the admin requirement entirely and matches how the
Enterprise Vector Store is meant to be refreshed incrementally.
Refreshing an existing collection (0.3.1.169): incremental first, full
refresh as the fallback — see :func:_resync_attached_collection.
update(data_refresh=True) is the documented "regenerate vectors from
the existing backing tables" call and matches tdfs4ds's situation exactly
(same attached view every time, changing rows); if it does not apply, the
collection is destroyed and recreated, which always picks the rows up but
needs admin rights.
Until 0.3.1.169 this branch called only add_datasets(), which means
"attach a NEW source object" and therefore could not succeed against an
always-identical source: every second and later build failed with 400 "You
can only add object names that do not exist in the collection", so newly
documented rows never reached the index. That 400 is no longer produced
here at all; the classifier
(:func:_is_source_already_attached_error) is kept for collections whose
source was attached outside tdfs4ds, and because until 0.3.1.167 the 400
surfaced — via _reraise_if_service_down's "Response Code:" branch — as
"service unreachable or not enabled ... check licensing/activation", which
was doubly wrong: the service had just answered, and the two later stores
were skipped.
force_rebuild=True really rebuilds (0.3.1.167): it destroys and
recreates each collection (see :func:_destroy_for_force_rebuild), which
is the only working way to pick up business-dictionary rows added since
the collection was created, and also the way out of a *_failed
collection. It requires admin privileges; when the destroy is refused
the call falls back to the ordinary reuse path with a warning, so a
non-admin session is never worse off. Between 0.3.1.133 and 0.3.1.166 the
flag was purely informational — both branches ran identical code, so
setting it (which users do precisely when the index looks stale) changed
nothing at all.
force_rebuild=False still never destroys, so rows whose source
business-dictionary entry was deleted are NOT purged from the index by
that path (no admin-free per-id deletion API is available yet — see
Collection.get_ids, currently an unimplemented stub upstream).
Call fn(), retrying while it raises the transient "not in a ready
state" 409 (see _is_not_ready_error) — the collection is still
processing a prior async operation (its own creation, or a previous
update/add_datasets call). Every Collection write operation is
asynchronous ("... request is accepted and in progress"); calling another
write immediately after can hit the collection while it's still busy.
Re-raises the original exception once timeout_seconds elapses, or immediately for any other exception shape (not a not-ready 409).
Poll _lookup_store_database(name) until it reports the collection
gone, or timeout_seconds elapses.
Collection.destroy() is asynchronous — it returns as soon as the
request is accepted ("Collection destroy request is accepted and in
progress"), not once the collection is actually gone. Calling
from_datasets() (create) immediately afterward is a race: the
collection can still exist server-side, so create collides with
TeradataGenAIException: Collection ... already exists even though
destroy() itself reported success.
Returns True if the collection was confirmed gone, False on timeout (caller decides whether that is fatal).
Destroy and recreate the named Teradata vector store collection(s) from
scratch — the true "clean rebuild" that :func:build_teradata_index no
longer performs (it always reuses/upserts, see its docstring), so stale
rows (business-dictionary entries deleted since the last index) are never
purged there.
Requires the connected user to have admin privileges — destroy()
(via :func:_safe_destroy) raises otherwise (see Collection.destroy()'s
docstring: "Only admin users can use this method"). Call this explicitly
when you know you have admin rights and want stale rows gone; the routine
build_vector_index()/rebuild_vector_indexing() path deliberately
does not call this, so it keeps working for non-admin sessions.
Parameters¶
sources:
Subset of "objects", "sections", "columns" to reset.
None (default) resets all three.
Normalise a single similarity-search hit into the standard row schema.
Confirmed live: a Teradata Vector Store hit's dict shape is flattened —
the text lives directly under the source's real business-dictionary
column name (business_description for objects/columns,
section_content for sections), NOT a generic content/text key,
and metadata (object_name, section_name, column_name,
database_name, table_name) is flattened into the same dict, not
nested under a metadata key. Before business_description/
section_content were added to _TEXT_KEYS, every hit's text
stayed empty and was silently dropped by the if not text.strip()
guard below — search_teradata_index would return 0 rows even when
the underlying similarity_search call itself returned real,
relevant hits.
Yield individual hits from whatever TeradataVectorStore.similarity_search returns (json string, list of Documents, list of dicts, object with .similar_objects attribute, etc.).
search_teradata_index(query: str, sources: list[str] = ('objects', 'sections', 'columns'), section_names: Optional[List[str]] = None, top_k: int = 10) -> pd.DataFrame
Search the Teradata vector stores using similarity search.
Cross-store federation (v0.3.1.148): searches the primary store's collections
AND, when read-only stores are attached (tdfs4ds.read_stores()), each
read-only store's OWN materialized collections — each store keeps its own
collections in its own database, so this needs no re-index, only additional
queries scoped by (vector_prefix, target_database) per store. Hits are
merged, tagged with FEATURE_STORE provenance (only when >1 store), deduped
on (OBJECT_NAME, FEATURE_STORE) and run through the shared distance gate.
Fail-open: a store whose collections are absent/failed is skipped with a
warning (the agent still answers it via the catalog / business-dictionary
paths). A single store (the default) is byte-identical to the historical read.
Chroma is a single local index and does NOT federate (see search_vector_index).