# kenso — Full Reference

> kenso turns a folder of Markdown files into a searchable knowledge base for AI agents. BM25 keyword search over SQLite FTS5, zero config, no infrastructure. Install with pip, index with one command, connect any MCP client.

kenso is an MCP server. AI agents search your docs through four tools: `search_docs`, `search_multi`, `get_doc`, and `get_related`. Results are ranked excerpts (~600 tokens per chunk) instead of entire files (3,000–8,000+). The document graph lets agents navigate typed relations between documents.

No vector database, no embedding model, no API keys. Two runtime dependencies: `mcp` and `aiosqlite`.

---

## Quick start

```bash
pip install kenso[yaml]      # Python 3.11+, MIT license
kenso ingest ./docs/          # index your Markdown files
```

That's it. Connect your editor — the MCP client starts kenso automatically.

```bash
# optional — verify the index before connecting an editor
kenso search "deployment pipeline"
kenso stats
```

---

## MCP Tools

### search_docs

Keyword search with BM25 ranking, deduplication, and relation re-ranking.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `query` | str | yes | — | Search query text |
| `category` | str \| None | no | None | Filter by exact category ("all", "", or None = no filter) |
| `limit` | int | no | 5 | Max results (capped to KENSO_SEARCH_LIMIT_MAX) |

Returns: JSON array of `{file_path, title, category, content_preview, score, tags?, related_count?, highlight?}`.

On empty query: returns `{"error": "Empty query."}`.

### search_multi

Multi-query search with Reciprocal Rank Fusion (RRF).

Each query runs independently through `search_docs`. Results are merged using RRF: `score = sum(1 / (K + rank))` with K=60. Queries capped at 5.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `queries` | list[str] | yes | — | List of search query strings (max 5) |
| `category` | str \| None | no | None | Filter by exact category |
| `limit` | int | no | 5 | Max results after merge (capped to KENSO_SEARCH_LIMIT_MAX) |

Returns: same shape as `search_docs`.

### get_doc

Retrieve full document content by file path.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `path` | str | yes | — | Document file path |
| `max_length` | int \| None | no | None | Max characters to return |

Returns: JSON object `{title, content, category, audience, tags, truncated?}`.

On non-existent path: returns `{"error": "No document found at path: <path>"}`.

### get_related

Navigate document graph with configurable depth and relation type filter.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `path` | str | yes | — | Document file path |
| `depth` | int | no | 1 | Hops to traverse (1=direct, max 3) |
| `relation_type` | str \| None | no | None | Filter by relation type |

Returns: JSON array of `{related_path, relation_type, direction, depth}`. Direction is "incoming" or "outgoing". Cycle-safe via visited set.

On non-existent path: returns empty array `[]`.

---

## Frontmatter

Parsed with PyYAML if installed (`pip install kenso[yaml]`), regex fallback otherwise.

### Supported fields

| Field | Type | Indexed | Description |
|-------|------|---------|-------------|
| `title` | str | 10× weight | Document title — strongest ranking signal |
| `category` | str | 5× weight | Category (fallback: parent directory name, then "general") |
| `tags` | str or list | 7× weight | Tags (comma-separated or YAML list) |
| `aliases` | list[str] | in searchable_content | Injected as "Also known as: ..." |
| `answers` | list[str] | in searchable_content | Injected as "Questions this document answers: ..." |
| `description` | str | in searchable_content | Summary text appended to searchable content |
| `relates_to` | str, list, or list of dicts | links table | Document relationships |

### Full frontmatter example

```yaml
---
title: Settlement Lifecycle for Equity Trades
category: post-trade
tags: [settlement, clearing, T+2, DVP, CNMV]
aliases:
  - trade settlement
  - post-trade processing
  - liquidación de operaciones
answers:
  - How are equity trades settled?
  - What is the T+2 settlement cycle?
  - What happens when a settlement fails?
description: >
  Covers the complete settlement lifecycle from trade execution
  to final DVP, including T+2 cycles and CNMV reporting.
relates_to:
  - path: order-management/matching-engine.md
    relation: receives_from
  - path: compliance/cnmv-reporting.md
    relation: triggers
---
```

### relates_to formats

```yaml
# Simple string
relates_to: guides/setup.md

# List of paths (all get relation_type "related")
relates_to:
  - guides/setup.md
  - compliance/reporting.md

# Typed relations
relates_to:
  - path: guides/setup.md
    relation: feeds_into
  - path: compliance/reporting.md
    relation: triggers
```

Glob patterns (`*`, `?`) in paths are silently ignored.

### Relation type conventions

| Type | Meaning |
|------|---------|
| `feeds_into` | Upstream data flow |
| `receives_from` | Downstream dependency |
| `triggers` | Causal relationship |
| `contains` | Parent-child hierarchy |
| `part_of` | Inverse of contains |
| `monitors` | Observational relationship |
| `implements` | Specification to implementation |
| `related` | Default fallback |

---

## CLI commands

### kenso ingest \<path\>

Scan directory for `.md` files and load into database. Files under 50 characters are skipped. Unchanged files (by SHA-256 hash of raw content including frontmatter) are skipped for incremental re-indexing.

### kenso search \<query\>

Search from terminal. Returns top 5 results with score, path, title, and highlight.

### kenso stats

Show database statistics: document count, chunk count, storage size, categories with counts, and link count.

### kenso lint \<path\>

Analyze Markdown files for retrieval quality issues. Works directly on files — no database required.

```bash
kenso lint <path>              # summary with score and prioritized fixes
kenso lint <path> --detail     # per-file violations
kenso lint <path> --json       # JSON output for CI integration
```

18 rules across three severity levels:

- **Errors** (exit code 1): file too short (KS001), broken relates_to links (KS002), tiny sections dropped during chunking (KS010), glob patterns in links (KS017)
- **Warnings**: missing tags (KS003), missing/generic titles (KS004/KS005), no preamble (KS006), generic headings (KS007), no H2 sections (KS008), orphan documents (KS012), inconsistent categories (KS014), too many links (KS018)
- **Info**: oversized sections (KS009), few tags (KS011), redundant tags covered by stemmer (KS013), weak lead sentences (KS015), dangling pronouns (KS016)

Each file scores 0–100. Overall score is the average across all files. `--detail` and `--json` are mutually exclusive.

### kenso serve

Start MCP server. Usually started automatically by your editor's MCP client — you rarely need to run this manually.

---

## Database

### Resolution cascade

1. `KENSO_DATABASE_URL` env var — explicit override, always wins
2. `.kenso/docs.db` in the current directory — project-local (default)
3. `~/.local/share/kenso/docs.db` — global fallback

Add `.kenso/` to your `.gitignore`. SQLite runs in WAL mode — multiple readers are safe.

### Environment variables

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `KENSO_DATABASE_URL` | str | (cascade above) | SQLite database path override |
| `KENSO_CHUNK_SIZE` | int | `4000` | Max chunk size in characters |
| `KENSO_CHUNK_OVERLAP` | int | `0` | Overlap between consecutive chunks |
| `KENSO_CONTENT_PREVIEW_CHARS` | int | `200` | Preview length in search results |
| `KENSO_SEARCH_LIMIT_MAX` | int | `20` | Maximum results per search |
| `KENSO_TRANSPORT` | str | `stdio` | Transport: stdio or streamable-http |
| `KENSO_HOST` | str | `127.0.0.1` | Server bind address |
| `KENSO_PORT` | str | `8000` | Server port |
| `KENSO_LOG_LEVEL` | str | `INFO` | Logging level |

---

## Editor configuration

kenso is a standard MCP server. For any client not listed here, set `command` to `kenso` with args `["serve"]`.

**Cursor** `.cursor/mcp.json`:
```json
{ "mcpServers": { "kenso": { "command": "kenso", "args": ["serve"] } } }
```

**VS Code** `.vscode/mcp.json`:
```json
{ "servers": { "kenso": { "command": "kenso", "args": ["serve"] } } }
```

**Claude Code** `.claude/mcp.json`:
```json
{ "mcpServers": { "kenso": { "command": "kenso", "args": ["serve"] } } }
```

**Codex CLI** `~/.codex/config.toml`:
```toml
[mcp_servers.kenso]
command = "kenso"
args = ["serve"]
```

**Gemini CLI** `~/.gemini/settings.json`:
```json
{ "mcpServers": { "kenso": { "command": "kenso", "args": ["serve"] } } }
```

If installed in a virtualenv, use the full path: `/path/to/.venv/bin/kenso`

For remote deployment (shared team access):
```bash
KENSO_TRANSPORT=streamable-http KENSO_HOST=0.0.0.0 KENSO_PORT=8000 kenso serve
```

Clients connect by URL:
```json
{ "mcpServers": { "kenso": { "url": "https://kenso.your-domain.com/mcp" } } }
```

---

## Internals

### Search cascade: AND → NEAR → OR

1. **AND** — all terms must be present (highest precision)
2. **NEAR/10** — terms within 10 tokens of each other (2–4 terms only)
3. **OR** — any term present (highest recall)

Stops at the first stage returning ≥ min(3, limit) results. If no stage meets the threshold, the last stage's results are returned.

### FTS5 column weights

| Column | Weight | Source |
|--------|--------|--------|
| `title` | 10× | Document title + section heading |
| `section_path` | 8× | Heading hierarchy ("Doc Title > Section") |
| `tags` | 7× | Frontmatter tags |
| `category` | 5× | Frontmatter category or parent directory |
| `searchable_content` | 1× | Chunk text + aliases + answers + description + tags + source path |

BM25 parameters: SQLite defaults (k1=1.2, b=0.75).

### Tokenizer

`porter unicode61 remove_diacritics 2`

- Porter stemmer handles English inflections (deploy/deployment/deploying)
- Unicode61 with Latin diacritics removal (José → jose)
- Case folding (API → api)
- Compound term expansion on queries: camelCase, snake_case, hyphens, dots

### Query processing

1. Strip FTS5 special characters: `"*()+-^:`
2. Wrap each term in double quotes for literal matching
3. Expand compound terms (camelCase/snake_case/hyphens/dots)
4. Single words ≥ 3 chars get prefix fallback: `"word"` → `"word"*`
5. Execute cascade (AND → NEAR → OR)

### Deduplication

Keeps only the highest-scoring chunk per document.

### Relation re-ranking

Documents with `relates_to` links to other documents in the result set get boosted:
`score = score × (1 + 0.15 × connection_count)`

Both directions count. Skipped if fewer than 2 results.

### Reciprocal Rank Fusion (search_multi)

Each query runs independently. Results merged with:
`rrf_score = Σ(1 / (60 + rank))` per query where the document appears.

Per-query fetch limit: `limit × 2`. Max 5 queries.

### searchable_content construction

```
[chunk content]

Also known as: [aliases, comma-joined]

Questions this document answers: [answers, pipe-joined]

[description]

Keywords: [tags, comma-joined]

Source: [relative file path]
```

Only non-empty fields are included.

---

## Chunking

1. Split by H2 headings (primary boundary)
2. Content before first H2 → preamble chunk titled "Document Title — Overview" (min 50 chars, otherwise merged into first section)
3. Sections under 20 characters are dropped
4. Oversized H2 sections split at H3, then H4 (max heading level 4)
5. Still oversized → split at paragraph boundaries (`\n\n`)
6. Code blocks (` ``` `, `~~~`) and tables (`|...|`) are protected from splitting
7. Overlap: prepend `KENSO_CHUNK_OVERLAP` chars from previous chunk (skips first and preamble chunks)

---

## SQLite schema

**chunks** — main content table:
```sql
CREATE TABLE chunks (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    file_path TEXT NOT NULL,
    chunk_index INTEGER NOT NULL DEFAULT 0,
    title TEXT,
    section_path TEXT,
    content TEXT NOT NULL,
    searchable_content TEXT,
    category TEXT,
    audience TEXT DEFAULT 'all',
    tags TEXT,
    content_hash TEXT,
    created_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
    updated_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
    UNIQUE (file_path, chunk_index)
);
```

**chunks_fts** — FTS5 virtual table:
```sql
CREATE VIRTUAL TABLE chunks_fts USING fts5(
    title, section_path, tags, category, searchable_content,
    content='chunks', content_rowid='id',
    tokenize='porter unicode61 remove_diacritics 2'
);
```

Auto-synced via triggers on INSERT, UPDATE, DELETE.

**links** — document relationships:
```sql
CREATE TABLE links (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    source_path TEXT NOT NULL,
    target_path TEXT NOT NULL,
    relation_type TEXT NOT NULL DEFAULT 'related',
    created_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
    UNIQUE (source_path, target_path, relation_type)
);
```

---

## Concurrency

SQLite WAL mode. Multiple readers safe. Concurrent writes serialized by SQLite — second writer waits, doesn't fail. Multiple `kenso serve` instances reading the same database is safe.
