tdfs4ds.agent.vector_index¶
Chroma-backed vector index for the tdfs4ds business dictionary.
Three collections per feature store mirror the three business dictionary tables:
- <prefix>__<store>__objects -> FS_BUSINESS_DICTIONARY_OBJECTS
- <prefix>__<store>__sections -> FS_BUSINESS_DICTIONARY_SECTIONS
- <prefix>__<store>__columns -> FS_BUSINESS_DICTIONARY_COLUMNS
One set per store (0.3.3.15, see chroma_collection_name): the primary and
each read-only store (tdfs4ds.read_stores()) keep their own collections in
the same local index, so switching or attaching a store never wipes another
one's embeddings, and a search covers every attached store. The collections
named tdfs4ds_objects / _sections / _columns (before 0.3.3.15, one
set for whichever store was primary) are no longer used.
module-attribute
¶
_SOURCE_TO_COLLECTION = {'objects': _COLLECTION_OBJECTS, 'sections': _COLLECTION_SECTIONS, 'columns': _COLLECTION_COLUMNS}
module-attribute
¶
_KEYS = {'objects': (DOCUMENTATION_BUSINESS_DICTIONARY_OBJECTS, ('DATABASE_NAME', 'OBJECT_NAME'), 'BUSINESS_DESCRIPTION'), 'sections': (DOCUMENTATION_BUSINESS_DICTIONARY_SECTIONS, ('DATABASE_NAME', 'OBJECT_NAME', 'SECTION_NAME'), 'SECTION_CONTENT'), 'columns': (DOCUMENTATION_BUSINESS_DICTIONARY_COLUMNS, ('DATABASE_NAME', 'TABLE_NAME', 'COLUMN_NAME'), 'BUSINESS_DESCRIPTION')}
module-attribute
¶
_SOURCE_BUILDERS = {'objects': _objects_to_docs, 'sections': _sections_to_docs, 'columns': _columns_to_docs}
Return a LangChain Embeddings instance based on the active provider.
Resolution order for each setting:
- EMBEDDING_MODEL_PROVIDER -> falls back to INSTRUCT_MODEL_PROVIDER
- EMBEDDING_MODEL_API_KEY -> falls back to INSTRUCT_MODEL_API_KEY
- EMBEDDING_MODEL_URL -> falls back to INSTRUCT_MODEL_URL
- EMBEDDING_MODEL_MODEL -> used as-is (default: text-embedding-3-small)
Injects tdfs4ds config variables into environment so that boto3, Azure, and Google Cloud clients pick up credentials and endpoints automatically.
Returns:
| Name | Type | Description |
|---|---|---|
Embeddings |
Embeddings
|
A configured LangChain embedding model. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the resolved provider is not supported. |
Truncate text to stay within the embedding model's token limit.
Respects tdfs4ds.EMBEDDING_MODEL_MAX_TOKENS (default 450) to prevent "Input validation error: inputs must have less than 512 tokens" from TEI servers.
A store descriptor {schema, schema_view, vector_prefix, is_primary}.
store is None (the primary), a database name, or a descriptor dict. A name matching an attached store (table or view database, case-insensitive) gets that store's descriptor; any other name is read as a store that is not attached (Maintenance can index one before it is attached).
The Chroma collection of source (objects / sections / columns) for store.
<vector_prefix>__<database>__<source>, the database being the store's
table database, lowercased. A name Chroma cannot take as it is (a character
outside [a-z0-9_-], or too long) is shortened and followed by a hash of
the database name, so two stores never share a collection. Teradata names
are case-insensitive, so DB_LAB and db_lab are the same store.
Return a LangChain Chroma vector store for the given collection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
collection_name
|
str
|
Name of the Chroma collection. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Chroma |
A configured LangChain Chroma vector store. |
Rows of query: through this thread's own connection when it has one, else through teradataml (the historical path).
True when a raw connection can be opened from the TDFS4DS_TD_* settings.
Give this thread its own raw teradatasql connection for the index reads.
Opened from the TDFS4DS_TD_* settings (utils.reconnect); closed on
exit. When none can be opened, reads go through teradataml as before and
the reason is logged. Yields the connection or None.
Read a store that is not the primary through its public business-dictionary view: read rights are enough, and the view already applies CURRENT VALIDTIME and drops blank descriptions.
WHERE|AND ValidStart >= TIMESTAMP '<since>+00:00' ('' when since is None).
since is a UTC YYYY-MM-DD HH:MM:SS read from the database
(_db_now_utc), never the client's clock.
Fetch all rows from the business dictionary objects table.
Uses execute_sql because CURRENT VALIDTIME must be at the outermost level. A store other than the primary is read through its public view. since: only rows documented at or after that UTC time (primary only).
Fetch all rows from the business dictionary sections table.
Fetch all rows from the business dictionary columns table.
Ids of the primary's documented rows of source now, without their text
(the same ids and the same blank-text rule as the _<source>_to_docs).
The database's current time in UTC, YYYY-MM-DD HH:MM:SS (None if unreadable).
The indexing date compares with the documentation's ValidStart, written by the database: a client clock would drift from it.
{source: {'indexed_at', 'embedding_model', 'documents'}} of store (empty
when never recorded). Read-only: never creates anything.
Is store's Chroma index behind its documentation? No embedding, no Chroma scan.
Per source: indexed_at (UTC, the database's time when the last build
started), changed = documented rows written since (ValidStart at or
after it), and state:
'up_to_date'/'behind'- from the dates;'never'- no build recorded;'model_changed'- built with another embedding model (everything re-embeds);'unknown'- a read-only store: its public views carry no ValidStart, so only a full comparison (the next build) can tell.
A row removed from the documentation is not counted (it ends a period, it does not open one); the next build drops it from the index.
(texts, metadatas) keyed by id, exactly as a build would store them.
{id: 'indexed' | 'stale' | 'missing'} for each expected id.
The single comparison behind both the build (which skips 'indexed' rows) and the admin app's index status: stored with the same text, metadata and embedding model = 'indexed'; stored differently = 'stale'; absent = 'missing'. collection is a raw chromadb collection. Raises on a read error -- callers decide whether to fail open.
Ids already stored with the same text, metadata and embedding model.
Fail-open: any error reading the collection returns an empty set, i.e. everything is (re-)embedded exactly as before this optimisation existed.
The raw chromadb collection, or None when it does not exist.
For status reads: unlike _get_chroma_store it never creates the
collection and never builds an embedding client (reading needs neither,
and a status page must work while the embedding model is unconfigured).
Not strictly read-only on disk: opening a PersistentClient updates
chroma.sqlite3 (chromadb's own housekeeping, observed live), though no
collection or row is written.
A raw chromadb client on the configured index (no embedding client).
Index status of the business-dictionary rows in df for one Chroma collection.
df has the columns the _fetch_<source> helpers return. The expected
documents come from the same builders as build_vector_index (ids,
truncation, metadata, embedding model), so a row reported 'indexed' is one
the next build would skip. Returns {'collection', 'exists', 'rows':
{id: status}, 'orphans': [ids stored but no longer expected], 'count'}.
Raises on a read error. store: see chroma_collection_name (None = the
primary).
What the local Chroma index holds for store, without reading the database.
{'store', 'collections': {source: {'name', 'exists', 'count'}}, 'documents'}.
Never creates a collection and needs no embedding model. A collection that
cannot be read reports exists=None with the error, whole.
_upsert_documents_batched(store: Chroma, docs: list[Document], ids: list[str], progress=None) -> tuple
Upsert documents into a Chroma store in batches (add new, update existing).
Rows already stored with the same text, metadata and embedding model are skipped -- re-embedding them would produce the same vector, and on a large business dictionary that is most of the cost of a routine rebuild.
Respects tdfs4ds.EMBEDDING_BATCH_SIZE and the inter-batch delay
tdfs4ds.EMBEDDING_BATCH_DELAY (for vLLM/TEI overload prevention); the
delay is applied between batches, not after the final batch. An expired
embedding credential is offered for renewal and the current batch retried
(see tdfs4ds.utils.credential_renewal).
progress, when given, is called as progress(done, total) once before
the first batch and after each one (total = documents to embed). A
failing callback never stops the build.
Returns (n_embedded, n_skipped).
Build or refresh the three vector-store collections from the business dictionary tables.
Dispatches to the configured backend (tdfs4ds.VECTOR_STORE_BACKEND):
'chroma' (default) or 'teradata'.
On Chroma, each feature store has its own three collections
(chroma_collection_name). stores lists the stores to index (database
names or descriptors; default: every store in tdfs4ds.read_stores(), the
primary first). A store that is not the primary is read through its public
FS_V_BUSINESS_DICTIONARY_* views, so read rights are enough, and nothing
is written to its database. stores is ignored by the Teradata backend,
which indexes the primary only.
progress, when given, is called with one dict per step: store,
source, collection (1-based), n_collections, done, total
(documents to embed in that collection, None before it is known) and
detail. A failing callback never stops the build.
Returns, on Chroma, {'stores': [{'store', 'collections': {source:
{'embedded', 'skipped', 'error'}}}]}.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
force_rebuild
|
bool
|
If True, delete existing collections and re-index from
scratch. On Chroma this needs no special privileges. On Teradata
it destroys and recreates each collection, which requires admin
privileges ( |
False
|
Delete from the collection the ids no longer documented; their number.
Fail-open (0): a read error never stops a build. The ids compared are the ones the build itself writes, so nothing else in the collection is touched.
search_vector_index(query: str, sources: list[str] = ('objects', 'sections', 'columns'), section_names: Optional[List[str]] = None, top_k: int = 10, k: Optional[int] = None) -> pd.DataFrame
Search the requested Chroma collections using similarity search.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
query
|
str
|
Natural language search query. |
required |
sources
|
list[str]
|
Which collections to search (subset of 'objects', 'sections', 'columns'). |
('objects', 'sections', 'columns')
|
section_names
|
Optional[List[str]]
|
If provided and 'sections' is in sources, filter by SECTION_NAME. |
None
|
top_k
|
int
|
Number of results per collection. |
10
|
k
|
Optional[int]
|
Alias for |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with columns: SOURCE, OBJECT_NAME, SECTION_NAME, COLUMN_NAME, TEXT, SCORE. |