Ingestion & MCP

Suck in content from anywhere — including any MCP server.

A source is any iterable of records. ingest() chunks, embeds, and upserts them.

from dynavec.ingest import ingest, IterableSource

src = IterableSource([
    {"id": "doc1", "text": long_text, "metadata": {"src": "wiki"}},
])
ingest(db, src, namespace="kb", chunk_size=1000, overlap=150)

From a Markdown or text directory

MarkdownSource reads UTF-8 .md and .txt files recursively. Install dynavec[ingest] for YAML front matter support.

from dynavec.ingest import MarkdownSource, ingest

source = MarkdownSource("./notes")
# Or select files with a root-relative glob:
source = MarkdownSource("./notes", glob="guides/**/*.md")
ingest(db, source, namespace="notes")

A Markdown file may begin with a YAML mapping between two --- lines:

---
title: Deployment guide
topic: aws
tags: [deployment, rag]
---
# Deploying the service
The document body starts here.

Front matter becomes metadata and is removed from the text before chunking. Use storage-compatible values (strings, numbers, booleans, lists, and mappings); quote dates to keep them as strings. Malformed or unclosed front matter raises an error naming the file. Text files are read verbatim.

IDs are root-relative paths such as guides/deploy.md; generated chunks retain this path in source_id. Metadata includes source="file" and path, which take precedence over front matter. Use separate namespaces for unrelated directory roots to avoid ID collisions. Discovery order is sorted, and files are read one at a time.

Try python examples/ingest_markdown.py ./notes --preview to inspect records without AWS calls. The example also supports ingestion into an existing index using an OpenAI embedder.

From any MCP server

MCPResourceSource turns an MCP server's resources (Notion, Confluence, Drive, your own) into an embeddable corpus — no per-source code.

from dynavec.ingest import ingest, MCPResourceSource

ingest(db, MCPResourceSource(mcp_session), namespace="kb")

Chunk ids are "{record_id}#chunk{n}" with source_id / chunk metadata, so you can group or delete a whole document later.