{% extends "base.html" %}
{% block title %}Vaults & Documents - Talkpipe Vault{% endblock %}
{% block content %}
{# In folder mode the open "vault" is a folder's own index; the named-vault
form below must not offer it as the vault to index into. #}
{% set vault_path = "" if folder_mode else vault_path %}
{# The retrieval filter belongs to one index, so the panel sits with what
it applies to: the Folders box for the one open folder, the vault panel
for a named vault. #}
{% macro filter_panel(retrieval_filter) %}
A ChatterLang script that filters or transforms the results
retrieved from the open {{ retrieval_filter.kind }} —
Each retrieved result flows through your script one at a time;
emit it to keep it, drop it to filter it, modify it to transform
it. Every result is a dictionary shaped like:
In
For the common case — is this text inside that
field? —
Any registered ChatterLang segment works — including LLM
transforms — but the script must be a single pipeline of
segments: no
Retrieval filter for this {{ retrieval_filter.kind }} (advanced)
{% if retrieval_filter.error %}
not compiling
{% elif retrieval_filter.active %}
active
{% elif retrieval_filter.script %}
saved, not enabled
{% endif %}
{{ retrieval_filter.target }} — before they reach
the Ask answer. It applies to this {{ retrieval_filter.kind }} only.
The script is stored with its index and travels with it; whether it
runs is decided here, on this machine.
How to write a filter
{"doc_id": "…", "score": 0.87,
"document": {"content": "…", "source": "…", "title": "…"}}
lambdaFilter/lambda expressions the
result is item, as everywhere in TalkPipe —
item['document'], item['score'],
item['doc_id']. (Because the result is a
dictionary, TalkPipe also exposes its top-level keys as bare
names, so document and score work
too.) Starter recipes:
| lambdaFilter[expression="'some term' not in item['document'].get('content', '').lower()"]
drop results whose content mentions a term
| lambdaFilter[expression="'/archive/' not in item['document'].get('source', '')"]
drop results from a folder, by source path
| lambdaFilter[expression="item['score'] > 0.2"]
drop low-scoring results (only when your
embedding backend reports similarity scores — the default
model2vec embedder scores all but the closest match or two
near 0, so there this drops most results, not just weak
ones)
isIn and isNotIn
say the same thing without an expression. Name the field with a
dotted path (document.content,
document.source) and the text to look for:
| isNotIn[field="document.content", value="some term"]
drop results whose content mentions a term
(the first recipe above, without the expression)
| isIn[field="document.source", value="/notes/"]
keep only results whose source path
contains a folder
isIn/isNotIn match exactly, so they are
case-sensitive, and the named field has to exist on every result
— a result missing it makes the script fail, which means
unfiltered results (or, with Strict checked, a failed
question). Reach for lambdaFilter when the field may
be absent (.get('source', '')), when the match
should ignore case (.lower()), or when the test is
anything more than containment.
INPUT FROM source, loops, or forks.
When active, the filter runs on Ask retrieval for this vault
(keyword and semantic streams are filtered independently, then
merged) and can be applied on the search pages with their
“Apply custom transform” checkbox.
Choose the folders you want to ask about. Each folder, and everything under it, is indexed in the background — keywords first, then meaning — and kept up to date as files change. You can search and ask right away; until indexing finishes, results may miss documents.
Click Browse… to choose a folder (or type its path); Add folder becomes available once the folder is found.
{% if folders %}{{ folder.root }}
{{ folder.progress }}
{{ folder.root }}
{{ folder.progress }}
{% if folder.vector_error %}{{ folder.vector_error }}{% endif %}
{% if folder.error %}{{ folder.error }}{% endif %}
{% if folder.role == 'writer' and folder.embedding_differs %}
{% endif %}
{% if folder.role == 'writer' and folder.chunking_differs %}
{% endif %}
{{ confirm_vault_path }} holds {{ confirm_entry_count }} item{{ 's' if confirm_entry_count != 1 }} that {{ 'are' if confirm_entry_count != 1 else 'is' }} not vault data. A vault is the folder that stores the search index — not the documents you want to search. Creating a vault here will write index files alongside your existing files, and deleting the vault later would delete this entire folder.
To make these documents searchable, put the vault in a new, empty folder instead and index this folder into it.
Pick the documents to index and the vault to index them into. Documents are embedded with {{ models.embedding_source }}/{{ models.embedding_model }} (change this under Settings).
{% if not folder_mode and retrieval_filter and retrieval_filter.available %} {{ filter_panel(retrieval_filter) }} {% endif %}No vaults opened yet — indexing documents creates your first one.
{% endif %}