Metadata-Version: 2.4
Name: noomexai-raven
Version: 0.2.1
Summary: RAVEN — Retrieval Augmented Adaptive Epistemic Navigation. Privacy-first, RAG backend.
License-Expression: MIT
Project-URL: Homepage, https://github.com/NoomexAI/raven_pypi
Project-URL: Repository, https://github.com/NoomexAI/raven_pypi
Project-URL: Issues, https://github.com/NoomexAI/raven_pypi/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.txt
License-File: ThirdPartyNotices.txt
Requires-Dist: llama-index-core<0.15,>=0.14
Requires-Dist: llama-index-llms-ollama<0.11,>=0.10
Requires-Dist: llama-index-embeddings-ollama<0.10,>=0.9
Requires-Dist: llama-index-llms-litellm<0.8,>=0.7
Requires-Dist: llama-index-embeddings-litellm<0.6,>=0.5
Requires-Dist: llama-index-vector-stores-qdrant<0.11,>=0.10
Requires-Dist: qdrant-client>=1.12
Requires-Dist: ollama>=0.5
Requires-Dist: litellm<2,>=1.41
Requires-Dist: pydantic>=2.0
Requires-Dist: docling<3.0,>=2.0
Provides-Extra: server
Requires-Dist: fastapi>=0.115; extra == "server"
Requires-Dist: uvicorn[standard]>=0.30; extra == "server"
Requires-Dist: python-multipart>=0.0.9; extra == "server"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: httpx2<3,>=2; extra == "dev"
Dynamic: license-file

# RAVEN

**Retrieval Augmented Adaptive Epistemic Navigation**

RAVEN is an asynchronous RAG backend for building applications around stored
documents. Applications can browse, search, and reconstruct sources directly,
or start a conversation in which the agent can use the same retrieval tools
and follow up on the user's manual work. The Python API and optional FastAPI
server expose durable operations and replayable events for both workflows.

RAVEN is local-first and privacy-conscious, but it is not limited to offline
models. Applications can use a local Ollama server, cloud models through
LiteLLM, or a mixture of the two.

RAVEN supports two primary integration paths:

```text
Python application  ───────────────────────>  Raven facade

Browser / desktop UI  ─>  FastAPI server  ─>  Raven facade
```

The high-level `Raven` class is the recommended entry point for library use.
The underlying components remain public for applications that need more direct
control.


## Contents

- [Choose how to use RAVEN](#choose-how-to-use-raven)
- [Key features](#key-features)
- [How RAVEN fits together](#how-raven-fits-together)
- [Glossary](#glossary)
- [Requirements](#requirements)
- [Installation](#installation)
- [Library quick start](#library-quick-start)
- [Operation-first API](#operation-first-api)
- [Events and streaming](#events-and-streaming)
- [Errors](#errors)
- [Model providers](#model-providers)
- [Knowledge and ingestion](#knowledge-and-ingestion)
- [Retrieval](#retrieval)
- [Conversations, sessions, and memory](#conversations-sessions-and-memory)
- [Manual session activity](#manual-session-activity)
- [Agent harness](#agent-harness)
- [Source reconstruction](#source-reconstruction)
- [FastAPI server quick start](#fastapi-server-quick-start)
- [Using RAVEN from a web or desktop UI](#using-raven-from-a-web-or-desktop-ui)
- [HTTP API overview](#http-api-overview)
- [HTTP request reference](#http-request-reference)
- [Configuration](#configuration)
- [Persistence and recovery](#persistence-and-recovery)
- [Upgrading to v0.2.1](#upgrading-to-v021)
- [Backup, restore, and cleanup](#backup-restore-and-cleanup)
- [Security and deployment responsibilities](#security-and-deployment-responsibilities)
- [Server troubleshooting](#server-troubleshooting)
- [Advanced component API](#advanced-component-api)
- [Development and testing](#development-and-testing)
- [Current limitations](#current-limitations)
- [License and third-party notices](#license-and-third-party-notices)


## Choose how to use RAVEN

RAVEN supports three deployment paths. They share the same backend behavior,
but differ in who creates the `Raven` instance and who supplies user identity.

| You are building | Start here | Who owns `Raven` | User identity |
| --- | --- | --- | --- |
| A Python application | [Library quick start](#library-quick-start) | Your Python process | Pass `user_id` to `Raven`; the stable default is suitable for one local user. |
| A desktop application with a web UI | [FastAPI server quick start](#fastapi-server-quick-start) | The bundled RAVEN server | Normally uses the stable default user; the desktop bridge keeps the launch bearer token private. |
| A hosted multi-user service | [Using RAVEN from a web or desktop UI](#hosted-applications) | RAVEN's runtime registry | Your host authenticates the account and forwards its internal UUID on every request. |

Use the library when Python code is the application boundary. Use the server
when another process, browser UI, or network service needs an HTTP and SSE
contract. The server is not a different backend: it validates HTTP input,
selects the correct user-scoped `Raven` runtime, and serializes the same
operations and events exposed by the library.

If this is your first time using RAVEN, follow this order:

1. Install Ollama 0.34.2 and start its server. RAVEN does not install or
   manage Ollama for you.
2. Install the package and verify the import.
3. Start RAVEN and configure one LLM plus one embedding model.
4. Create a knowledge and ingest a document.
5. Run a scoped search and inspect or reconstruct the returned sources.
6. To continue with the agent, create a conversation and a temporary session.
   The session can record manual actions before the first turn.
7. Consume the turn events, then close the session and RAVEN cleanly.

The detailed sections explain each step, including cancellation, retry,
reconnection, persistence, and failures.


## Key features

- **Document-first retrieval** — choose specific files, whole knowledges, or
  the entire knowledge base, then search and inspect results from Python or HTTP.
- **Local and cloud models** — use Ollama locally, LiteLLM-compatible cloud
  providers, or different providers for generation and embeddings.
- **Durable asynchronous operations** — long-running work has observable
  status, task results, cancellation, persisted events, and explicit retry.
- **Replayable streaming** — operation events can be consumed directly in
  Python or transported to a UI through replayable Server-Sent Events (SSE).
- **Persistent knowledge** — parse, semantically split, embed, retrieve, and
  navigate documents stored in isolated knowledge databases.
- **Multiple retrieval strategies** — embedded, hierarchical,
  agreement-based, and vector-conditioned retrieval. The agent can choose a
  strategy automatically or use the one requested for the turn.
- **Persistent conversations** — retain messages, tool calls, tool results,
  compacted context, vector memory, preferences, titles, and turn identity.
- **Manual session activity** — keep references to searches, source opens,
  browsing, and reconstruction so the agent can follow up in the same conversation.
- **Transparent evidence** — reconstruct the source sections used during a
  response so users can inspect and verify the supporting material.
- **Crash-aware storage** — interrupted operations are detected after restart,
  retryable work remains discoverable, and ingestion state is reconciled.
- **Per-user isolation** — each user UUID receives a separate data,
  operations, settings, and temporary-storage tree.
- **Application-ready server** — the optional FastAPI layer provides bearer
  authentication, OpenAPI schemas, upload controls, operation endpoints, and
  SSE delivery.


## How RAVEN fits together

```text
Python caller                         Browser / desktop UI
      |                                       |
      |                               Authenticated FastAPI
      +-------------------+-------------------+
                          |
                      Raven facade
                          |
                 Operations and events
                          |
       +------------------+-------------------+
       |                  |                   |
 Knowledge pipeline   Conversations      Agent harness
       |              and sessions            |
       +------------------+-------------------+
                          |
               SQLite and local Qdrant
                          |
                 Ollama or LiteLLM
```

The main ownership boundaries are:

| Component | Responsibility |
| --- | --- |
| `Raven` | Composes the backend and exposes its high-level, operation-first API. |
| `KnowledgeBase` | Discovers and manages persistent `Knowledge` resources. |
| `Knowledge` | Owns one knowledge database, its source records, and its vectors. |
| `ConversationManager` | Discovers and manages persistent `Conversation` resources. |
| `Conversation` | Owns its fixed retrieval scope, messages, turns, manual activity, memory, and preferences. |
| `Session` | Runs manual actions and agent turns against one persistent conversation. |
| `AgentHarness` | Defines model behavior, available tools, streaming, and evidence handling. |
| `Provider` | Resolves provider-neutral model specifications to Ollama or LiteLLM adapters. |
| `Operation` | Owns the lifecycle and event stream of one user-visible action. |
| `OperationTask` | Represents one invocation, result, cancellation, and retry boundary inside an operation. |

Knowledge and conversations are durable resources. Create a session when you
want to work against a conversation, then close it when finished.
Different conversations may run concurrently, while RAVEN prevents overlapping
turns from mutating the same conversation.


## Glossary

| Term | Meaning |
| --- | --- |
| RAVEN home | The application-controlled root directory. User-specific data is stored below it. |
| User runtime | One user-scoped `Raven` instance with isolated paths, settings, operations, knowledges, and conversations. |
| Knowledge | A persistent collection of source files, semantic sections, and vectors. |
| Knowledge base | The registry and lifecycle manager for all knowledges belonging to one user. |
| Conversation | A fixed retrieval scope with persistent messages, manual activity, memory, and preferences. |
| Session | A temporary object for running manual actions and agent turns in a conversation. |
| Agent harness | The component that builds model behavior, exposes permitted tools, runs the agent loop, and translates output into events. |
| Operation | The status, event stream, and durable history of one user-visible action. |
| Operation task | One invocation inside an operation, with its own result, status, parent relationship, and retry metadata. |
| Event | An ordered, persisted observation emitted while an operation runs. |
| Retrieval scope | The selection of knowledges and files eligible for a search. |
| Manual activity | A saved record of a completed session action and its source references. |
| Reconstruction | Reloading stored section sequences and highlighting the selected sections in their surrounding context. |


### Runtime lifecycle

The normal library lifecycle is:

```text
construct Raven
      |
      v
await raven.start()
      |
      +--> recover interrupted operation records
      +--> discover persistent knowledge and conversations
      +--> load per-user runtime settings
      |
      v
configure model pair
      |
      v
run knowledge, retrieval, or conversation work
      |
      v
close temporary sessions
      |
      v
await raven.close()
```

`Raven.start()` does not configure models. Model configuration is explicit
because the LLM and embedding model are application choices. `Raven.close()`
cancels active work, flushes operation state, closes open Qdrant and SQLite
resources, and closes provider adapters. It does not stop an external Ollama
server.

`start()` and `close()` are lifecycle methods, not operation-producing methods.
Do not create new work after `close()`.

Runtime inspection is immediate:

```python
print(raven.is_started)
print(raven.models_loaded)
print(raven.runtime_status())
```

`runtime_status()` returns the user ID, lifecycle flags, model-loaded flag,
operation-store health, runtime-settings revision, and the configured LLM and
embedding identities. It never returns API-key values or provider clients.


## Requirements

- Python 3.11 or newer.
- A writable RAVEN home directory supplied by the application.
- For local models, a separately installed and running
  [Ollama 0.34.2](https://ollama.com/download) server.
- For cloud models, the relevant provider API key available through an
  environment variable.
- Enough system memory, accelerator memory, and storage for the models and
  document collections selected by the application.

Installing `noomexai-raven` does not install the Ollama application or binary.
RAVEN connects to Ollama but does not start, stop, update, or supervise the
Ollama process. The desktop application or deployment host is responsible for
installing Ollama 0.34.2 and managing its lifecycle and host-level resources.

RAVEN does not require a particular GPU API or a fixed amount of VRAM. Those
requirements depend on the selected local model and the way Ollama is deployed.


## Installation

Install the Python library with both Ollama and LiteLLM provider support:

```bash
pip install noomexai-raven
```

Install the library and FastAPI server dependencies:

```bash
pip install "noomexai-raven[server]"
```

RAVEN uses different names at different integration boundaries:

| Context | Name |
| --- | --- |
| Product | RAVEN |
| PyPI distribution | `noomexai-raven` |
| Python package | `nraven` |
| Command-line program | `nraven` |
| High-level Python class | `Raven` |

For example:

```python
from nraven import Raven
```

Verify the installation before configuring external models:

```bash
python -c "import nraven; print(nraven.Raven)"
```

For an Ollama-backed setup, also verify that the external service is reachable
and that the required model names exist:

```bash
ollama list
```

RAVEN can pull a missing Ollama model later through an observable operation,
but the Ollama server itself must already be running. Cloud usage requires the
provider API-key environment variable to be present in the process that runs
RAVEN.


## Library quick start

The `Raven` facade is the recommended entry point for Python applications. The
example configures local Ollama models, ingests and searches one document, then
streams a turn in a conversation whose retrieval scope selects that file.

```python
import asyncio

from nraven import EventType, ModelRole, ModelSpec, Raven, RetrievalScope, RetrievalSelection


async def main() -> None:
    raven = Raven("./raven-home")
    session = None
    await raven.start()

    try:
        configure_task = await raven.configure_models(
            ModelSpec(
                provider="ollama",
                model="qwen3:8b",
                role=ModelRole.LLM,
            ),
            ModelSpec(
                provider="ollama",
                model="bge-m3",
                role=ModelRole.EMBEDDING,
            ),
        )
        await configure_task.result()

        create_knowledge_task = await raven.create_knowledge(
            "engineering",
            user_summary="Engineering reference documents",
        )
        await create_knowledge_task.result()

        ingestion_task = await raven.ingest(
            "engineering",
            "./documents/system-design.pdf",
        )
        ingested = await ingestion_task.result()

        scope = RetrievalScope(selections=(
            RetrievalSelection(knowledge_name="engineering", file_id=ingested["file_id"]),
        ))
        query = "What does the design say about thermal protection?"
        search = await raven.retrieve_embedded(query, scope=scope)
        sections = await search.result()
        print(f"Direct search returned {len(sections)} sections.")

        create_conversation_task = await raven.create_conversation(scope=scope)
        conversation = await create_conversation_task.result()

        session = raven.session(conversation)
        await session.start()
        run = await session.generate_response(
            query,
            retrieval_mode="auto",
        )

        async for event in run.stream:
            if event.type == EventType.CHAT_THINKING_DELTA:
                print(event.data.get("delta", ""), end="")
            elif event.type == EventType.CHAT_TOOL_CALL:
                print(f"\nTool: {event.data.get('name')}")
            elif event.type == EventType.CHAT_RESPONSE_DELTA:
                print(event.data.get("delta", ""), end="")

        result = await run.collect()
        print(f"\n\nFinal response: {result.response}")
    finally:
        if session is not None:
            await session.close()
        await raven.close()


asyncio.run(main())
```

Ollama must already be running and the selected models must be installed. Use
`Raven.pull_ollama_model()` when the application should pull a missing Ollama
model as an observable operation.

Save a source document at `./documents/system-design.pdf` or change the path
and query to match an existing document. This quick start assumes an empty
home. On a later process start, persistent
knowledges and conversations are discovered automatically. Use
`raven.get_knowledge("engineering")`, `raven.list_conversations()`, and
`raven.get_conversation(conversation_id)` instead of recreating resources that
already exist.

Operation-producing calls return an `OperationTask`, not the final domain
value. Await `task.result()` when the next step depends on that value, or
consume the operation's events when progress is the primary concern.


## Operation-first API

RAVEN represents work with two related objects:

- An `Operation` is the root lifecycle of one user-visible action. It owns the
  status and the ordered event stream.
- An `OperationTask` is one unit of work within that operation. It owns that
  invocation's result and retry metadata.

Nested component calls reuse the parent operation. This preserves one coherent
event stream while assigning each task its own UUID, name, status, and result.

Most high-level methods start their work immediately and return an
`OperationTask`:

```python
task = await raven.list_ollama_models()
models = await task.result()
```

Awaiting the method creates and schedules the task; it does not wait for the
operation to finish. `await task.result()` waits for completion and returns the
method's domain result. If the task failed or was cancelled, `result()` raises
the corresponding error instead of returning an ambiguous value.

An operation moves through these states:

```text
queued -> running -> completed
                  -> failed
                  -> cancelled
```

The states mean:

| Status | Meaning | Can more work be added? | `task.result()` behavior |
| --- | --- | --- | --- |
| `queued` | The operation or task has been durably registered but its worker has not started. | The root task is being established. | Waits. |
| `running` | The root worker is active and may create nested tasks. | Yes, while the operation remains running. | Waits. |
| `completed` | The root worker succeeded and its child tasks joined; explicitly handled child failures may remain in task history. | No. | Returns the task's native result. |
| `failed` | A worker raised an error and the failure was persisted. | No. | Raises the original `RavenError` or a safe reconstructed error. |
| `cancelled` | Cancellation was requested and terminal cancellation was persisted. | No. | Raises `RavenError` with `code="operation_cancelled"`. |

`Operation.status` describes the complete root action. Each `OperationTask`
also has a status because nested work can finish before the root operation.
The operation cannot complete successfully until its active child tasks have
joined.

Awaiting a child's `result()` surfaces its error and marks that result as
observed. A parent may catch and handle that error, then complete successfully;
the child's own status remains failed. Unobserved child failures or
cancellations prevent successful operation completion.

### Root, child, and sibling tasks

The first task run in a new operation is its root task. If that worker calls
another operation-aware component and passes the same `Operation`, the new
invocation becomes a child task. Two tasks started by the same parent are
siblings. RAVEN records `parent_task_id` for this relationship.

```text
operation: session.generate_response
|
+-- root task: session.generate_response
    |
    +-- child: conversation.get_context
    +-- child: retrieval.embedded
    +-- child: reconstruction.reconstruct
    +-- child: conversation.append_turn
```

This is why a session turn does not create disconnected retrieval and
reconstruction operations. The UI sees one ordered stream for the turn while
the library can still inspect or await each task independently.

The IDs serve different purposes:

| Identifier | Stability and scope | Use it for |
| --- | --- | --- |
| `operation_id` | One UUID for the complete user-visible action. | Status, event replay, cancellation, and grouping all nested work. |
| `task_id` | One UUID for one invocation inside that operation. | Obtaining that invocation's result, inspecting its error, and retrying eligible work. |
| `parent_task_id` | The task that directly initiated a child; `null` for the root. | Reconstructing task hierarchy for diagnostics or UI detail views. |

Applications can inspect and control work through the facade:

```python
operation = await raven.get_operation(task.operation_id)
record = await raven.get_operation_record(task.operation_id)
task_record = await raven.get_operation_task(task.operation_id, task.task_id)

await raven.wait_operation(task.operation_id)
await raven.cancel_operation(task.operation_id)
```

Operations and tasks are also pageable through `list_operations()` and
`list_operation_tasks()`. This lets a library caller or server recover status
without holding the original Python object.

History is returned newest first. Use the last ID from the previous page as
the next cursor:

```python
page = await raven.list_operations(limit=25, status="failed")

if page:
    next_page = await raven.list_operations(
        limit=25,
        status="failed",
        after_operation_id=page[-1].operation_id,
    )
```

For task history, call `list_operation_tasks()` with the owning operation ID
and use `after_task_id` in the same way. A cursor must identify an existing
record in the requested scope; it is not an arbitrary offset.

### Creating custom operations

Applications integrating their own long-running workers can use the same
contract:

```python
from nraven import Event, EventType


async def custom_worker(operation):
    await operation.publish(
        Event(
            type=EventType.WORK_PROGRESS,
            data={"stage": "indexing", "completed": 4, "total": 10},
        )
    )
    operation.raise_if_cancelled()
    return {"indexed": 10}


task = await raven.run_operation("my_app.index", custom_worker)
result = await task.result()
```

The example above uses `run_operation()` for a single worker: it creates the
operation, starts that worker, and gives you an `OperationTask` whose
`result()` waits for its return value.

For a workflow that calls several RAVEN components, create the operation
first and pass it to each call. This example assumes `raven` is started, its
models are configured, the source file exists, and `engineering` is a new
knowledge name:

```python
async def prepare_knowledge(operation):
    create_task = await raven.create_knowledge(
        "engineering",
        operation=operation,
    )
    knowledge = await create_task.result()

    ingest_task = await raven.ingest(
        knowledge.name,
        "./documents/system-design.pdf",
        operation=operation,
    )
    return await ingest_task.result()


operation = await raven.create_operation("my_app.prepare_knowledge")
root_task = await operation.run("my_app.prepare_knowledge", prepare_knowledge)
ingested_file = await root_task.result()
print(ingested_file["file_id"])
```

`operation.run()` starts the root task; its name must match the name passed to
`create_operation()`. Inside the worker, both component calls run as child
tasks under the same operation. Await each child's `result()` before depending
on its output. All three tasks share one operation ID and replayable event
stream, while `root_task.result()` waits for the whole workflow and returns
the ingestion result. Operation names must be non-empty stable strings;
application names should use a namespace such as `my_app.prepare_knowledge`
to avoid collisions with built-in names.

`submit_operation(name, worker)` is the event-oriented convenience form: it
creates and starts the root task but returns the owning `Operation`. Use it
when the caller primarily needs operation-level status/events. Use
`run_operation()` when the caller needs the returned `OperationTask` and its
native result.

### Built-in operation names

These stable names appear in task records and event correlation metadata:

| Area | Operation names |
| --- | --- |
| Runtime | `nraven.configure_models`, `runtime.settings.update`, `runtime.settings.reset` |
| Model provider | `model.check_connection`, `model.list`, `model.inspect`, `model.pull`, `model.delete`, `model.load_llm`, `model.load_embedding`, `model.preload_llm`, `model.preload_embedding`, `model.unload_llm`, `model.unload_embedding` |
| Knowledge | `knowledge.set_summary`, `knowledge.ingest`, `knowledge.delete_file`, `knowledge.create`, `knowledge.delete` |
| Ingestion | `ingestion.run`, `ingestion.cleanup` |
| Embedded retrieval | `retrieval.embedded`, `retrieval.embedded.files` |
| Hierarchical retrieval | `retrieval.hierarchical` |
| Agreement retrieval | `retrieval.agreement` |
| Vector-conditioned retrieval | `retrieval.vector_conditioned` |
| Reconstruction | `reconstruction.reconstruct`, `reconstruction.from_turn` |
| Conversation | `conversation.save_preference`, `conversation.remove_preference`, `conversation.update`, `conversation.generate_title`, `conversation.get_context`, `conversation.append_turn`, `conversation.reconcile_turn`, `conversation.create`, `conversation.update_metadata`, `conversation.delete` |
| Agent/session | `chat.generate_response`, `session.generate_response` |
| Manual session activity | `session.list_knowledges`, `session.list_files`, `session.list_sections`, `session.open_file`, `session.open_section`, `session.retrieve`, `session.reconstruct` |

Applications may display these names for diagnostics, but normal UI labels
should be friendlier and should not infer retryability from the name alone.

### Cancellation

Cancel the task directly when its handle is available:

```python
await task.cancel()
```

Or cancel its complete operation by ID:

```python
await raven.cancel_operation(task.operation_id)
```

Cancellation is cooperative. RAVEN records a terminal cancellation state and
allows cleanup or durable state transitions already in progress to finish
safely where required.

Cancelling a root task cancels the complete operation and its active children.
Cancelling a non-root task cancels only that task. Component workers check
cancellation between expensive stages, but a blocking provider request may not
stop until the provider returns control. Therefore cancellation means "stop as
soon as a safe boundary is reached," not "undo every committed side effect."

After cancellation, wait for `task.result()`, `run.wait()`, or an operation
terminal event before assuming cleanup is complete.

### User-confirmed retry

Only task types with an explicit retry policy are retryable. RAVEN currently
uses user-confirmed retry for operations such as ingestion, reconstruction,
and session generation rather than automatically repeating arbitrary model or
storage work.
The persisted `retry_policy` field is exactly `"never"` or
`"user_confirmed"`; it is not an instruction to auto-retry on startup.

| Retryable task type | Maximum attempts | What is reused |
| --- | --- | --- |
| `ingestion.run` | 3 | Knowledge name, retained source/snapshot information, and ingestion parameters. |
| `reconstruction.reconstruct` | 3 | The normalized evidence/retrieval input. |
| `session.generate_response` | 3 | Conversation ID, stable turn ID, query, retrieval mode, and run settings. |

An attempt is retryable only when it failed with an approved recoverable error,
has JSON-serializable retry input, has attempts remaining, and has not already
been retried. Completed, cancelled, non-retryable, and already-retried tasks
are rejected.

```python
retryable = await raven.list_retryable_tasks()

if retryable:
    failed = retryable[0]
    retry_task = await raven.retry_task(
        failed.operation_id,
        failed.task_id,
    )
    result = await retry_task.result()
```

A retry creates a new operation linked to the failed task. It does not rewrite
the original operation's history. The new task exposes
`retry_of_operation_id`, `retry_of_task_id`, and an incremented `attempt`.
Retries are idempotency-aware: ingestion does not duplicate a committed file,
and a session retry reuses an already committed turn with the same `turn_id`
instead of writing a second response.


## Events and streaming

Events are RAVEN's primary observability and streaming contract. Every event is
associated with an operation and may also identify the task that emitted it.

Conceptually, an event contains:

```json
{
  "type": "chat.tool_call",
  "data": {
    "name": "embedded_retrieval",
    "step": 1
  },
  "operation_id": "7e6d796c-6711-4fc7-a967-78738ecac96a",
  "task_id": "b3e7752e-64db-4924-a171-4c761f1594c0",
  "task_name": "session.generate_response",
  "event_id": 8,
  "timestamp": "2026-09-20T10:30:00+00:00",
  "is_final": false
}
```

`event_id` increases monotonically within an operation. `task_id` and
`task_name` correlate events from nested work without splitting the operation
into disconnected streams.

The remaining envelope fields have fixed semantics:

| Field | Type | Meaning |
| --- | --- | --- |
| `type` | `EventType` / string | Stable machine-readable event name. In JSON and SSE it is serialized as its string value. |
| `data` | object | Event-specific payload. Treat unknown additive fields as forward-compatible. |
| `operation_id` | UUID | Owning operation. RAVEN assigns it when the event is published. |
| `task_id` | UUID or `null` | Emitting task. Domain events published inside a task inherit the active task ID. |
| `task_name` | string or `null` | Stable built-in or application task name. |
| `event_id` | integer | One-based sequence within this operation; use it as the replay cursor. |
| `timestamp` | UTC datetime | Time RAVEN created the event. |
| `is_final` | boolean | `true` only for the operation's terminal event. No later event can be published. |

Consume events from a task while it runs:

```python
task = await raven.pull_ollama_model("qwen3:8b")

async for event in task.events():
    print(event.event_id, event.type.value, event.data)

await task.result()
```

Important event families include:

- `chat.thinking_delta`, `chat.tool_call`, `chat.tool_result`,
  `chat.response_delta`, and `chat.completed`;
- ingestion start, progress, commit, cleanup, and completion events;
- retrieval strategy events;
- `reconstruction.file` and reconstruction completion events;
- context compaction and conversation turn-commit events;
- model connection, pull, load, preload, unload, and failure events;
- operation and task terminal events.

### Replay after disconnection

Events are persisted, not limited to the lifetime of a subscriber. Store the
last successfully processed event ID and resume after it:

```python
last_event_id = 12

async for event in raven.operation_events(
    operation_id,
    after_event_id=last_event_id,
):
    last_event_id = event.event_id or last_event_id
    print(event)
```

The same cursor model powers the FastAPI SSE endpoint and its
`Last-Event-ID` reconnection behavior. A consumer receives retained events
after its cursor, then waits asynchronously for new events until the operation
reaches a terminal state.

Use `read_operation_events()` when a finite snapshot is more appropriate than
a live iterator.

```python
events = await raven.read_operation_events(
    operation_id,
    after_event_id=0,
    limit=100,
)
```

`after_event_id` is exclusive: a cursor of `12` asks for event 13 onward. A
cursor of `0` starts at the beginning. A negative cursor is invalid. A cursor
ahead of the retained operation history raises `event_history_gap` rather than
silently pretending that no events exist.

`task.events()` filters the operation stream to that task and its descendants.
For a root task this is the complete operation stream; for a child task it is
the corresponding subtree. `operation.events()` always exposes the entire
operation.

### Event reference

Event names are grouped by what a consumer normally does with them:

| Family | Event types | Typical consumer behavior |
| --- | --- | --- |
| Operation lifecycle | `operation.queued`, `operation.started`, `operation.completed`, `operation.failed`, `operation.cancelled` | Update the overall job state. The last three are terminal; the terminal operation event has `is_final=true`. |
| Task lifecycle | `operation.task.queued`, `operation.task.started`, `operation.task.completed`, `operation.task.failed`, `operation.task.cancelled` | Update one task row or nested-stage indicator. The payload contains the task `name`; failures include `error`. |
| Generic progress | `work.progress` | Render application-defined progress from `data`. |
| Chat stream | `chat.thinking_delta`, `chat.response_delta` | Append `data.delta` to separate thinking and answer buffers. `chat.delta` remains a generic/compatibility event type; the current harness emits the two explicit variants instead. |
| Chat tools | `chat.tool_call`, `chat.tool_result` | Correlate with `data.call_id`; render the step, tool name, bounded `ui_summary`, evidence, and recoverable error. |
| Chat outcome | `chat.max_iterations`, `chat.result_reused`, `chat.completed`, `chat.failed` | Mark iteration fallback, an idempotently reused turn, final response/evidence, or failure. `chat.completed` is a domain event; wait for the operation terminal event before releasing all operation state. |
| Ingestion pipeline | `ingestion.started`, `ingestion.progress`, `ingestion.vectors_written`, `ingestion.metadata_committed`, `ingestion.completed`, `ingestion.failed` | Render stages and counters, then show the committed file result or failure. |
| Ingestion cleanup | `ingestion.cleanup.started`, `ingestion.cleanup.completed`, `ingestion.cleanup.failed` | Explain rollback or crash-recovery cleanup. |
| Knowledge lifecycle | `knowledge.create.started`, `knowledge.create.completed`, `knowledge.create.failed`, `knowledge.updated`, `knowledge.delete.started`, `knowledge.delete.completed`, `knowledge.delete.failed` | Refresh knowledge metadata or remove a deleted item. |
| Knowledge file work | `knowledge.ingest.started`, `knowledge.ingest.progress`, `knowledge.ingest.completed`, `knowledge.ingest.failed`, `knowledge.file_delete.started`, `knowledge.file_delete.completed`, `knowledge.file_delete.failed` | Track storage-level ingestion and file deletion nested inside a higher-level operation. |
| Conversation lifecycle | `conversation.create.started`, `conversation.create.completed`, `conversation.create.failed`, `conversation.update.started`, `conversation.update.completed`, `conversation.update.failed`, `conversation.delete.started`, `conversation.delete.completed`, `conversation.delete.failed` | Refresh conversation metadata or remove a deleted conversation. |
| Context compaction | `conversation.memory_compaction.started`, `conversation.memory_compaction.completed`, `conversation.memory_compaction.failed` | Show that model context is being summarized; canonical UI history remains intact. |
| Turn persistence | `conversation.turn_commit.started`, `conversation.turn_commit.completed`, `conversation.turn_commit.reused`, `conversation.turn_commit.failed` | Mark durable turn commit or idempotent reuse. |
| Manual activity | `conversation.activity.recorded` | Read the new `activity_id`, `sequence`, `kind`, and `conversation_id`. For browsing, source opens, and retrieval, `data.result` contains the action output. Reconstruction streams its files separately. |
| Vector memory | `conversation.memory_index.started`, `conversation.memory_index.completed`, `conversation.memory_index.failed` | Track semantic indexing of committed user/assistant messages. |
| Embedded retrieval | `retrieval.embedded.started`, `retrieval.embedded.completed`, `retrieval.embedded.failed` | Show direct vector retrieval progress and result count/failure. |
| Hierarchical retrieval | `retrieval.hierarchical.started`, `retrieval.hierarchical.read`, `retrieval.hierarchical.completed`, `retrieval.hierarchical.failed` | Show knowledge/file reads and LLM-assisted scoring progress. |
| Agreement retrieval | `retrieval.agreement.started`, `retrieval.agreement.completed`, `retrieval.agreement.failed` | Show the combined embedded/hierarchical comparison. |
| Vector-conditioned retrieval | `retrieval.vector_conditioned.started`, `retrieval.vector_conditioned.completed`, `retrieval.vector_conditioned.failed` | Show vector narrowing followed by hierarchical selection. |
| Standalone HTTP retrieval result | `retrieval.result` | Read ranked sections or the native agreement result in `data.result`. Wait for the terminal operation event to confirm success. |
| Reconstruction | `reconstruction.started`, `reconstruction.file`, `reconstruction.completed`, `reconstruction.failed` | Render each reconstructed file from `reconstruction.file`, then mark the source set complete. |
| Ollama connection | `model.connection.started`, `model.connection.completed`, `model.connection.failed` | Display connection-check state. |
| Ollama inventory | `model.list.started`, `model.list.completed`, `model.list.failed`, `model.inspect.started`, `model.inspect.completed`, `model.inspect.failed` | Refresh or inspect the external Ollama model inventory. |
| Ollama mutation | `model.pull.started`, `model.pull.progress`, `model.pull.completed`, `model.pull.failed`, `model.delete.started`, `model.delete.completed`, `model.delete.failed` | Display pull progress or refresh inventory after deletion. |
| LLM adapter | `model.load_llm.started`, `model.load_llm.completed`, `model.load_llm.failed` | Show LLM adapter validation/construction for local or cloud models. |
| Embedding adapter | `model.load_embedding.started`, `model.load_embedding.completed`, `model.load_embedding.failed` | Show embedding-adapter validation/construction for local or cloud models. |
| LLM residency | `model.preload_llm.started`, `model.preload_llm.completed`, `model.preload_llm.failed`, `model.unload_llm.started`, `model.unload_llm.completed`, `model.unload_llm.failed` | Show local LLM residency changes. Cloud models report a no-local-residency result instead. |
| Embedding residency | `model.preload_embedding.started`, `model.preload_embedding.completed`, `model.preload_embedding.failed`, `model.unload_embedding.started`, `model.unload_embedding.completed`, `model.unload_embedding.failed` | Show local embedding residency changes. Cloud models report a no-local-residency result instead. |

Failure payloads use the same safe error shape as the library and HTTP API:

```json
{
  "error": {
    "code": "knowledge_not_found",
    "message": "Knowledge 'engineering' does not exist.",
    "details": {}
  }
}
```

Clients should branch on `code`, display `message`, and treat `details` as
structured context. Do not parse human-readable messages to determine logic.


## Errors

Expected failures use `RavenError`. Its stable fields are:

```python
from nraven.core.errors import RavenError


try:
    await task.result()
except RavenError as error:
    print(error.code.value)
    print(error.message)
    print(error.details)
```

| Field | Meaning |
| --- | --- |
| `code` | Stable `ErrorCode` enum used for program logic. |
| `message` | Safe human-readable explanation. |
| `details` | Optional structured context such as IDs, limits, or revision values. |

Unexpected exceptions are normalized at transport boundaries to
`internal_error`; raw exception text is not exposed to clients. Important
error groups are:

| Group | Common codes | What the caller should do |
| --- | --- | --- |
| Resource lookup | `knowledge_not_found`, `conversation_not_found`, `file_not_found`, `section_not_found`, `conversation_turn_not_found` | Refresh the relevant list and correct the identifier. |
| Resource conflict | `knowledge_already_exists`, `file_already_exists`, `conversation_turn_active`, `conversation_turn_conflict` | Reuse the existing resource, choose another name, or wait for the active turn. |
| Model configuration | `llm_model_required`, `embedding_model_required`, `invalid_model_spec`, `model_api_key_not_found`, `model_capability_missing`, `model_reload_in_progress` | Configure both roles, supply the referenced environment variable, or wait for the current reconfiguration. |
| Provider/runtime | `model_provider_failed`, `ollama_unavailable`, `ollama_operation_failed` | Check provider credentials/connectivity or the external Ollama service, then retry eligible work. |
| Ingestion input | `source_file_not_found`, `source_file_changed`, `source_file_unreadable`, `source_file_too_large`, `unsupported_source_file`, `document_parse_failed` | Correct or re-upload the source; do not retry unchanged invalid input indefinitely. |
| Embeddings | `invalid_embedding_result`, `embedding_dimension_mismatch`, `embedding_identity_mismatch` | Use a working embedding adapter; use the same embedding identity for an existing knowledge or rebuild that knowledge deliberately. |
| Retrieval/agent | `invalid_retrieval_mode`, `retrieval_mode_not_allowed`, `retrieval_scoring_invalid`, `agent_max_iterations` | Correct the mode/scope, inspect provider output, or adjust the bounded runtime setting. |
| Operation | `operation_not_found`, `operation_task_not_found`, `operation_cancelled`, `operation_interrupted`, `operation_finished` | Refresh history; retry only if the task record reports `can_retry=true`. |
| Retry | `operation_task_not_retryable`, `operation_task_already_retried`, `invalid_retry_input`, `upload_retry_expired` | Follow the existing linked retry or resubmit the original request/source. |
| Persistence | `persistence_failed`, `operation_sync_failed`, `operation_database_failed`, `operation_database_in_use`, `operation_database_corrupted` | Stop new writes, inspect filesystem ownership/health, and restore or repair before resuming service. |
| Configuration | `invalid_system_config`, `invalid_runtime_config`, `runtime_config_conflict`, unsupported schema-version codes | Correct the field; for a revision conflict, read current settings and reapply the intended change. |
| Server boundary | `authentication_required`, `invalid_user_id`, `request_body_too_large`, `request_query_too_large`, capacity-exceeded codes | Correct credentials/input or wait for capacity. |
| Event replay | `invalid_event_cursor`, `invalid_event_page_size`, `event_history_gap`, `event_stream_closed`, `event_stream_finished` | Correct the cursor/page size, refresh operation status, or treat an expired/deleted history as unavailable. |

Whether an error is retryable is determined by the persisted task's retry
policy—not merely by this table. Always inspect `can_retry` or use
`list_retryable_tasks()` before presenting a Retry action.

### Complete error-code catalogue

The following codes are the stable public vocabulary. Several related codes
can lead to the same caller action, but keeping them distinct makes logs,
events, and UI messages precise.

| Resource and conversation code | Condition |
| --- | --- |
| `invalid_knowledge_name` | A knowledge name normalizes to no valid characters. |
| `knowledge_already_exists` | The normalized knowledge identity is already present. |
| `knowledge_not_found` | The requested knowledge is not registered. |
| `knowledge_closed` | Work was attempted on a permanently closed knowledge handle. |
| `knowledge_not_started` | Storage-dependent work was attempted before the knowledge opened. |
| `invalid_resource_cache_size` | A knowledge/conversation resource-cache bound is invalid. |
| `conversation_already_exists` | A conversation identity conflicts with an existing resource. |
| `conversation_not_found` | The requested conversation is not registered. |
| `conversation_closed` | Work was attempted through a closed conversation or session. |
| `conversation_not_started` | Session/conversation work began before startup. |
| `conversation_memory_not_initialized` | Context or vector memory was used before model-backed initialization. |
| `context_token_limit_too_small` | The configured memory budget cannot support the memory implementation. |
| `conversation_turn_conflict` | An existing turn ID is associated with different input. |
| `conversation_turn_active` | Another session currently owns the conversation's turn lock. |
| `conversation_turn_not_found` | The requested committed turn does not exist. |
| `conversation_turn_result_missing` | A committed/reused turn lacks a valid stored agent result. |
| `activity_in_use` | A running turn has captured the activity. |
| `activity_committed` | The activity is already linked to a committed turn. |
| `reconstruction_evidence_not_found` | Persisted turn data contains no usable reconstruction evidence where evidence was required. |
| `foreign_conversation` | A `Conversation` from another `Raven` instance was supplied to `session()`. |
| `invalid_preference_id` | A preference ID is not a valid UUID. |
| `preference_not_found` | No preference has the supplied ID. |
| `invalid_conversation_id` | The conversation identifier is malformed. |
| `invalid_conversation_title` | The supplied title violates title requirements. |
| `invalid_message_cursor` | A message pagination cursor is invalid. |
| `file_already_exists` | The knowledge already contains or is ingesting that file name. |
| `file_not_found` | The requested stored file or file cursor does not exist. |
| `file_id_name_mismatch` | The stored name of a selected file differs from the name supplied with its ID. |
| `section_not_found` | The section does not exist or does not belong to the asserted file. |

| Model, retrieval, and ingestion code | Condition |
| --- | --- |
| `invalid_retrieval_mode` | The supplied retrieval mode string is unknown. |
| `retrieval_mode_not_allowed` | The tool's strategy differs from the strategy selected for this turn. |
| `retrieval_scoring_invalid` | Hierarchical model scoring remained invalid after bounded retries. |
| `llm_model_required` | An operation requires a configured LLM. |
| `embedding_model_required` | An operation requires a configured embedding model. |
| `invalid_model_spec` | A `ModelSpec`, role pairing, option, or reserved field is invalid. |
| `model_api_key_not_found` | The environment variable named by `api_key_ref` is absent or empty. |
| `model_provider_failed` | A non-Ollama provider or adapter operation failed. |
| `model_capability_missing` | The loaded adapter lacks a required async model capability. |
| `model_reload_in_progress` | Model-dependent work was requested during pair reconfiguration. |
| `agent_max_iterations` | The agent exhausted its allowed iterations and could not finish safely. |
| `invalid_chunking` | Chunk size, overlap, or related splitting values are invalid. |
| `no_chunks_produced` | Valid document sections produced no embedding chunks. |
| `invalid_embedding_result` | The embedding adapter returned missing, malformed, or non-finite vectors. |
| `embedding_dimension_mismatch` | A vector dimension differs from the established collection dimension. |
| `embedding_identity_mismatch` | The configured embedding identity differs from the one recorded for a knowledge. |
| `source_file_not_found` | The ingestion source path does not identify a file. |
| `source_file_changed` | A retry/snapshot hash no longer matches the submitted source. |
| `source_file_unreadable` | RAVEN cannot read the source. |
| `source_file_too_large` | The source exceeds the effective size limit. |
| `trusted_ingestion_disabled` | The server's trusted local-path endpoint is disabled. |
| `ingestion_path_not_allowed` | A trusted path is outside configured allowed roots. |
| `upload_retry_expired` | A retained upload required for retry no longer exists. |
| `invalid_upload_filename` | A browser upload name is empty, unsafe, or otherwise invalid. |
| `unsupported_source_file` | The file extension is not one of the supported document types. |
| `document_parse_failed` | Native or Docling parsing failed. |
| `no_sections_produced` | Parsing/splitting produced no semantic sections. |
| `section_metadata_extraction_failed` | LLM metadata extraction failed after bounded retries. |
| `ingestion_reconciliation_failed` | Pending vectors/metadata could not be reconciled safely. |

| Operation and event code | Condition |
| --- | --- |
| `invalid_operation_name` | An operation name is empty/invalid or a root task name does not match it. |
| `operation_not_found` | The operation UUID has no retained record. |
| `operation_cancelled` | A result was requested from cancelled work. |
| `operation_interrupted` | Startup recovery found non-terminal work whose process worker no longer exists. |
| `operation_finished` | New work/publication was attempted after terminal state. |
| `invalid_operation_id` | An operation ID is malformed. |
| `invalid_operation_status` | A status filter/value is unknown. |
| `invalid_operation_page_size` | An operation/task history page size is invalid. |
| `invalid_list_page_size` | A resource-list page size is invalid. |
| `invalid_operation_cache_size` | The finished-operation in-memory cache bound is invalid. |
| `operation_manager_closed` | Work was requested after the manager closed. |
| `operation_task_not_found` | The task UUID is absent or belongs to another operation. |
| `operation_task_not_retryable` | Status, type, error, retry input, or attempts disallow retry. |
| `operation_task_already_retried` | A linked retry already exists for that task. |
| `invalid_retry_input` | Durable retry input is missing, malformed, or not JSON serializable. |
| `event_stream_closed` | The stream is closed or reserved for cleanup. |
| `event_stream_finished` | Publication was attempted after its final event. |
| `invalid_event_cursor` | The cursor is negative or otherwise invalid. |
| `invalid_event_page_size` | The requested event page size is invalid. |
| `event_history_gap` | The requested cursor is ahead of the recovered retained history. |
| `operation_sync_failed` | SQLite operation/event state could not be checkpointed durably. |
| `operation_database_failed` | General operation-store access failed. |
| `operation_database_in_use` | Another owner holds the operation database. |
| `operation_database_corrupted` | Integrity validation found a corrupt operation database. |
| `unsupported_operation_database_version` | The database schema is newer/unsupported. |
| `invalid_operation_sync_interval` | The periodic durability interval is invalid. |
| `invalid_retention` | An operation/upload retention value is invalid. |
| `invalid_cleanup_batch_size` | A bounded cleanup batch size is invalid. |

| Configuration, server, and general code | Condition |
| --- | --- |
| `invalid_metadata` | Persisted or caller-supplied structured data violates its contract. |
| `unsupported_metadata_version` | Knowledge/conversation metadata uses an unsupported schema version. |
| `persistence_failed` | A durable resource update could not be committed. |
| `invalid_system_config` | A process-wide configuration field is invalid. |
| `unsupported_system_config_version` | The system-settings schema version is unsupported. |
| `invalid_runtime_config` | A per-user runtime setting/update is invalid. |
| `unsupported_runtime_config_version` | The runtime-settings schema version is unsupported. |
| `runtime_config_conflict` | `expected_revision` does not match the current revision. |
| `invalid_user_id` | User context is missing where required or is not a UUID. |
| `authentication_required` | The bearer token is missing or invalid. |
| `server_already_running` | Another RAVEN server already owns the selected home. |
| `runtime_registry_closed` | A server request reached a closed runtime registry. |
| `runtime_capacity_exceeded` | The server cannot create/lease another user runtime under its configured bound. |
| `operation_capacity_exceeded` | Per-user or global active-operation capacity is exhausted. |
| `request_body_too_large` | The HTTP request body exceeds the configured maximum. |
| `request_query_too_large` | The URL query string exceeds the configured maximum. |
| `ollama_unavailable` | The configured Ollama server cannot be reached. |
| `ollama_operation_failed` | Ollama failed a requested model action. |
| `internal_error` | An unexpected exception was hidden behind a safe boundary error. |


## Model providers

RAVEN separates model configuration from model implementation through three
public types:

- `ModelSpec` describes a model without constructing its adapter.
- `ModelRole` distinguishes the generation model from the embedding model.
- `Provider` routes each specification to Ollama or LiteLLM.

`ModelSpec` contains:

| Field | Type | Required | Meaning and constraints |
| --- | --- | --- | --- |
| `provider` | string | Yes | `"ollama"` for local Ollama, or the LiteLLM provider identifier required by the selected service. It is trimmed, normalized to lowercase, and cannot be empty. |
| `model` | string | Yes | Provider-specific model name. It is trimmed but otherwise preserved and cannot be empty. For Ollama, use the name shown by `ollama list`, including its tag where applicable. |
| `role` | `ModelRole` | Yes | Exactly `ModelRole.LLM` (`"llm"`) or `ModelRole.EMBEDDING` (`"embedding"`). The two arguments to `configure_models()` must have the corresponding roles. |
| `api_key_ref` | string or `None` | No | Name of an environment variable containing the API key. It is resolved when a non-Ollama adapter is loaded; the value is not stored in the spec. Omit it when the provider uses ambient credentials or no key. |
| `options` | object | No | Non-secret LiteLLM adapter options. Defaults to `{}`. Ollama specs currently reject non-empty options. Reserved model/provider/key fields and credential-like keys are rejected. |

`ModelSpec` is immutable and rejects unknown fields. It describes one role,
not a complete model pair. RAVEN requires one LLM spec and one embedding spec:

```python
llm_spec = ModelSpec(
    provider="ollama",
    model="qwen3:8b",
    role=ModelRole.LLM,
)

embedding_spec = ModelSpec(
    provider="ollama",
    model="bge-m3",
    role=ModelRole.EMBEDDING,
)
```

> **WARNING — Choose your embedding model before ingesting data.** Knowledge
> vectors and conversation vector memory are generated in that model's vector
> space. A different embedding model cannot reliably search those existing
> vectors, even if it produces vectors with the same dimensions. RAVEN records
> the embedding identity and rejects mismatches; configuring another model
> does **not** convert stored embeddings. To change models after data exists,
> plan an explicit rebuild/re-embedding of the affected knowledge and memory
> stores from their source data. Changing the LLM does not have this particular
> vector-compatibility constraint.

Secrets must not be placed in `options`. RAVEN rejects common credential fields
there and resolves `api_key_ref` from the process environment when the model is
loaded.

### What model configuration does

`Raven.configure_models(llm_spec, embedding_spec)` is deliberately more than
assigning two names. It performs this serialized sequence:

1. Validate that the first spec has role `llm` and the second has role
   `embedding`.
2. Build or reuse both LlamaIndex adapters through `Provider`.
3. Verify that the LLM exposes asynchronous chat, streaming chat, and structured
   prediction, and that the embedding model exposes asynchronous query and
   batch text embedding plus similarity calculation.
4. If the pair changed, validate the embedding model against existing
   knowledge stores. Existing vector collections cannot silently switch to an
   incompatible embedding identity or dimension.
5. Install the validated pair into RAVEN and rebuild model-dependent retrieval
   pipelines. A session's harness is created later by `Raven.session()`.
6. Unload replaced Ollama models that are no longer part of the active pair.
7. Preload the configured Ollama embedding model and LLM. Cloud adapters skip
   local residency actions.

The call returns an `OperationTask`; model-dependent work must wait for its
result:

```python
configure_task = await raven.configure_models(llm_spec, embedding_spec)
configured = await configure_task.result()
```

While configuration is in progress, new sessions are rejected with
`model_reload_in_progress`. If adapter construction, capability validation, or
embedding compatibility fails before installation, the previously configured
pair remains active. Configuration itself is runtime state: applications
should configure the required pair again after restarting RAVEN.

### Local models with Ollama

The following configures both roles from a running Ollama server:

```python
import asyncio

from nraven import ModelRole, ModelSpec, Raven


async def main() -> None:
    raven = Raven("./raven-home")
    await raven.start()

    try:
        task = await raven.configure_models(
            ModelSpec(
                provider="ollama",
                model="qwen3:8b",
                role=ModelRole.LLM,
            ),
            ModelSpec(
                provider="ollama",
                model="bge-m3",
                role=ModelRole.EMBEDDING,
            ),
        )
        configured = await task.result()
        print(configured)
    finally:
        await raven.close()


asyncio.run(main())
```

By default RAVEN connects to `http://127.0.0.1:11434`. Set `OLLAMA_HOST` in the
RAVEN process environment before constructing `Raven` to use another Ollama
endpoint. RAVEN does not choose Ollama's CPU/GPU backend; that is a property of
the externally managed Ollama process.

Configuration verifies both adapters and checks the embedding model against
existing knowledge stores before installing the new pair. When no models are
configured, RAVEN installs the pair and preloads its Ollama models. When a pair
is already configured, RAVEN installs the new pair and wires its pipelines,
unloads replaced Ollama models that are no longer selected, and preloads the
configured embedding model followed by the LLM. If pre-installation validation
fails, the previous configured pair remains active. Unload or preload failures
can occur after the new pair is installed: configuration and external Ollama
residency are not one atomic transaction.

RAVEN also exposes operation-based Ollama administration for connection checks,
listing, inspection, pulling, and deletion. These methods connect to the
Ollama server; they do not own its process.

```python
# Returns None on success; raises ollama_unavailable on failure.
connection = await raven.check_ollama_connection()
await connection.result()

# Returns the model records supplied by Ollama.
listing = await raven.list_ollama_models()
models = await listing.result()

# Returns Ollama's detailed model record.
inspection = await raven.inspect_ollama_model("qwen3:8b")
details = await inspection.result()

# Returns None. Progress is available as model.pull.progress events and through
# the optional callback.
pull = await raven.pull_ollama_model("qwen3:8b")
async for event in pull.events():
    if event.type == EventType.MODEL_PULL_PROGRESS:
        print(event.data)
await pull.result()

# Permanently deletes the model from Ollama and clears its cached adapters.
deletion = await raven.delete_ollama_model("unused-model:latest")
await deletion.result()
```

Deleting an Ollama model is different from unloading it. Deletion removes the
downloaded model from Ollama storage. Unloading only releases runtime residency
and keeps the model installed.

### Cloud models through LiteLLM

Set the provider's API key in the environment managed by your application or
deployment platform. `api_key_ref` contains only the variable's name:

```bash
# Shell syntax varies by platform. The important part is that the variable is
# present in the process environment before Python or the RAVEN server starts.
GEMINI_API_KEY=replace-with-your-secret
```

RAVEN reads `os.environ`; it does not automatically parse a `.env` file. If an
application uses `python-dotenv`, call `load_dotenv()` before creating/configuring
RAVEN. In production, prefer the host platform's secret/environment mechanism.

```python
import asyncio

from nraven import ModelRole, ModelSpec, Raven


async def main() -> None:
    raven = Raven("./raven-home")
    await raven.start()

    try:
        task = await raven.configure_models(
            ModelSpec(
                provider="gemini",
                model="gemini-2.5-flash",
                role=ModelRole.LLM,
                api_key_ref="GEMINI_API_KEY",
                options={"temperature": 0.2},
            ),
            ModelSpec(
                provider="ollama",
                model="bge-m3",
                role=ModelRole.EMBEDDING,
            ),
        )
        configured = await task.result()
        print(configured)
    finally:
        await raven.close()


asyncio.run(main())
```

The LLM and embedding model do not need to use the same provider. Provider
availability, capabilities, prices, and rate limits remain properties of the
selected external service.

For LiteLLM-backed models, `provider` is forwarded as LiteLLM's custom provider
identifier and `model` is forwarded as the provider's model name. The exact
identifier is provider-dependent; consult LiteLLM's provider documentation and
the model vendor's current model catalogue. For embeddings, RAVEN constructs
the provider-qualified model name expected by the LiteLLM embedding adapter.

Examples of non-secret options include generation temperature, timeout, API
base URL where supported, and embedding batch size. Options are adapter
specific: an option valid for an LLM may not be valid for an embedding model.
RAVEN reserves fields that it controls itself, including model name, API key,
and custom-provider selection.

### Switching configured models

Call `Raven.configure_models()` with a new valid pair to switch models. The
replacement is serialized and validated before it becomes active:

**Keep the same embedding model for existing knowledge and vector memory.**
The warning above applies to model switches too: RAVEN does not re-embed old
data during configuration, and an incompatible replacement is rejected.

```python
task = await raven.configure_models(new_llm_spec, new_embedding_spec)
configured = await task.result()
```

The result is JSON-compatible configuration metadata:

```json
{
  "llm": {
    "provider": "gemini",
    "model": "gemini-2.5-flash",
    "role": "llm"
  },
  "embedding": {
    "provider": "ollama",
    "model": "bge-m3",
    "role": "embedding"
  }
}
```

The response intentionally describes configured identity, not credentials or
provider client internals. API-key values are never returned.

### Ollama model residency

Configured Ollama models can be explicitly preloaded into or released from
Ollama's runtime memory without changing the configured adapter:

```python
preload = await raven.preload_configured_model(
    ModelRole.LLM,
    keep_alive="10m",
)
print(await preload.result())

unload = await raven.unload_configured_model(ModelRole.LLM)
print(await unload.result())
```

`keep_alive` accepts an Ollama duration string or a finite number of seconds.
A negative value requests indefinite residency. Zero is rejected for preload;
call `unload_configured_model()` when the intended action is unloading.

For a cloud model, these methods complete without attempting local residency
and report that the provider has no local residency to manage.

The residency result has this shape:

```json
{
  "provider": "ollama",
  "model": "qwen3:8b",
  "role": "llm",
  "action": "preload",
  "performed": true
}
```

When `performed` is `false`, the result also includes a `reason`, such as
`"provider_has_no_local_residency"`.

### Provider troubleshooting

| Symptom/code | Meaning | Corrective action |
| --- | --- | --- |
| `model_api_key_not_found` | The environment variable named by `api_key_ref` is absent or empty. | Set it in the process environment and configure again. Do not place the value in `options`. |
| `invalid_model_spec` | Roles are reversed, required strings are empty, Ollama options were supplied, or options contain reserved/secret fields. | Correct the spec using the field table above. |
| `model_capability_missing` | The selected adapter does not provide the async LLM or embedding capability RAVEN requires. | Select a compatible chat or embedding model/adapter. |
| `embedding_identity_mismatch` | Existing knowledge was created with another embedding identity. | Restore the original embedding model or deliberately rebuild/re-ingest that knowledge. |
| `embedding_dimension_mismatch` | Produced vectors do not match the stored Qdrant collection dimension. | Use the original compatible embedding model or rebuild the knowledge. |
| `ollama_unavailable` | RAVEN cannot reach the configured Ollama host. | Start/check the external Ollama server and `OLLAMA_HOST`, then run the connection check. |
| `ollama_operation_failed` | Ollama rejected or failed an inspect, pull, delete, load, preload, or unload action. | Inspect Ollama logs and the model name, then retry if appropriate. |
| `model_provider_failed` | A LiteLLM/provider adapter failed to load or execute. | Verify provider name, model name, credentials, network access, options, quota, and rate limits. |


## Knowledge and ingestion

A knowledge is one persistent collection of related documents and vectors.
`KnowledgeBase` manages the collection of knowledge resources; each `Knowledge`
owns its own metadata, source-section records, and local Qdrant storage.

Create a knowledge before ingesting documents into it:

```python
create_task = await raven.create_knowledge(
    "engineering",
    user_summary="Engineering specifications and design documents.",
)
knowledge = await create_task.result()
```

Knowledge names identify their storage directories and must be unique within
the current user's RAVEN home. The user summary helps whole-base, reasoning-based
retrieval decide which knowledge collections are relevant.

Names are trimmed, every character outside letters, numbers, `_`, and `-` is
replaced with `_`, and leading/trailing underscores are removed. The normalized
name must contain at least one valid character. Because the normalized name is
the persistent identity, names such as `Engineering Docs` and
`Engineering_Docs` resolve to the same resource and cannot both be created.

Creation returns an opened `Knowledge` object. Its persisted metadata is:

```json
{
  "schema_version": 1,
  "created_at": "2026-09-20T10:30:00+00:00",
  "name": "engineering",
  "user_summary": "Engineering specifications and design documents."
}
```

### Knowledge API reference

Inspection methods return their final value directly; mutations return an
`OperationTask` so their events and terminal state remain observable.

| Method | Inputs | Return/result | Side effect |
| --- | --- | --- | --- |
| `create_knowledge(name, user_summary="")` | Name and optional summary | Task result: `Knowledge` | Creates and registers a persistent knowledge directory. |
| `get_knowledge(name)` | Knowledge name | Open `Knowledge` handle | Opens or reuses the resource; no content mutation. |
| `get_knowledge_details(name)` | Knowledge name | Metadata object shown above | None. |
| `list_knowledges()` | None | All knowledge metadata records | None. |
| `list_knowledges_page(limit, after_name=None)` | Positive page size and optional prior-page name | One metadata page | None. |
| `update_knowledge(name, user_summary=...)` | Knowledge name and replacement summary | Task result: `None` | Atomically replaces `user_summary`; read it with `get_knowledge_details()`. |
| `list_knowledge_files(name)` | Knowledge name | File records | None. |
| `list_knowledge_files_page(name, limit, after_file_id=None)` | Knowledge, page size, optional file cursor | One file page in insertion order | None. |
| `list_file_sections(name, file_name)` | Knowledge and exact stored file name | Sections in document order | None. |
| `list_file_sections_page(name, file_name, limit, after_section_index=0)` | Knowledge, file, page size, last section index | One section page | None. |
| `get_knowledge_section(name, section_id, file_name=None)` | Knowledge, section ID, optional file assertion | Complete section record | None. |
| `count_knowledge_files(name)` | Knowledge name | Integer file count | None. |
| `count_knowledge_vectors(name)` | Knowledge name | Integer vector-point count | None. |
| `delete_knowledge_file(name, file_id)` | Knowledge and file ID—not file name | Task result: deleted file identity | Removes vectors and file/section records. |
| `delete_knowledge(name)` | Knowledge name | Task result on completion | Closes and removes the complete knowledge resource. |

Page cursors are stable resource identities, not numeric offsets. Use the final
item from one page to request the next:

```python
files = await raven.list_knowledge_files_page("engineering", limit=50)

if files:
    next_files = await raven.list_knowledge_files_page(
        "engineering",
        limit=50,
        after_file_id=files[-1]["file_id"],
    )
```

### Supported document formats

RAVEN currently accepts:

| Format | Extensions | Parsing behavior |
| --- | --- | --- |
| Plain text | `.txt` | Read natively as UTF-8 text. |
| Markdown | `.md`, `.markdown` | Read natively while preserving the textual source. |
| PDF | `.pdf` | Parsed through Docling into ordered document elements and page provenance. |
| Word | `.docx` | Parsed through Docling into ordered document elements and available page provenance. |

The parser normalizes text, headings, list items, tables, captions, formulas,
and picture elements. Tables are projected to Markdown where possible. The
current ingestion path indexes the document's textual projection; it does not
send raw image data to a vision model.

PDF and DOCX elements may carry page ranges. Plain-text and Markdown files use
`navigation_type="none"` because they do not have an intrinsic page system.
These provenance fields later allow reconstructed sections to link back to the
appropriate part of the source.

### Ingestion flow

```text
Source validation and snapshot
              |
              v
     Provenance-aware parsing
              |
              v
       Semantic sectioning
              |
              v
   LLM section-metadata extraction
              |
              v
       Chunking and embedding
              |
              v
  Qdrant vectors + SQLite file records
```

The parser first produces ordered elements. The semantic splitter then creates
sections while retaining the contributing element IDs and combined source
range. Each stored section receives an ID shaped as:

```text
<file-id>-<section-index>
```

For example, `a83fd91c20b4-3` identifies the third semantic section of that
file. A stored section has a summary, keywords, conditions, definitions,
source element IDs, a source range for navigation, and separate raw content.

The complete stored file record has this shape:

```json
{
  "file_id": "a83fd91c20b4",
  "file_name": "system-design.pdf",
  "section_count": 8,
  "chunk_count": 21,
  "ingested_at": "2026-09-20T10:35:00+00:00",
  "navigation_type": "page"
}
```

A complete section record returned by the high-level API or FastAPI navigation
has this shape (the agent's `list_sections` tool returns a smaller view):

```json
{
  "file_id": "a83fd91c20b4",
  "file_name": "system-design.pdf",
  "section_index": 3,
  "section_id": "a83fd91c20b4-3",
  "summary": "Thermal protection requirements",
  "keywords": ["thermal", "shutdown"],
  "conditions": ["temperature exceeds the configured limit"],
  "definitions": [],
  "raw_content": "...",
  "source_element_ids": ["a83fd91c20b4-7", "a83fd91c20b4-8"],
  "source_range": [2, 3]
}
```

`navigation_type` belongs to the file because all of its sections share the
same navigation system. `source_range` belongs to each section. The integer
suffix in `section_id` is the section index; it is not an independently
generated ID.

Start ingestion through the high-level API after configuring the models:

```python
from nraven import EventType


ingestion = await raven.ingest(
    "engineering",
    "./documents/system-design.pdf",
)

async for event in ingestion.events():
    if event.type == EventType.INGESTION_PROGRESS:
        print(event.data)

ingested_file = await ingestion.result()
print(ingested_file)
```

The result contains the knowledge name, file name, generated file ID, section
count, chunk count, and original source path. Progress events identify stages
such as source validation, snapshotting, parsing, sectioning, metadata
extraction, chunking, embedding, vector storage, and metadata commit.

Chunking and semantic-splitting values use the current per-user runtime
settings unless explicitly overridden for the call:

```python
ingestion = await raven.ingest(
    "engineering",
    "./documents/system-design.pdf",
    breakpoint_percentile_threshold=92,
    chunk_size=768,
    chunk_overlap=64,
)
```

Every `Raven.ingest()` argument is:

| Argument | Type | Default when omitted | Constraint/effect |
| --- | --- | --- | --- |
| `knowledge_name` | string | Required | Must identify an existing knowledge. |
| `source_path` | string or `Path` | Required | Must identify a readable supported file. Library paths are trusted application input; server path ingestion has additional policy checks. |
| `breakpoint_percentile_threshold` | integer or `None` | Runtime setting, initially `95` | `1..100`. Lower values generally create more semantic boundaries. |
| `buffer_size` | integer or `None` | Runtime setting, initially `1` | Positive sentence-window buffer used when comparing semantic units. |
| `max_extraction_retries` | integer or `None` | Runtime setting, initially `3` | Positive number of LLM metadata-extraction attempts per section. |
| `chunk_size` | integer or `None` | Runtime setting, initially `512` | Positive embedding chunk size. |
| `chunk_overlap` | integer or `None` | Runtime setting, initially `50` | Non-negative and strictly smaller than `chunk_size`. |
| `max_source_size_bytes` | integer or `None` | System limit, initially `100 MiB` | Positive per-call limit that cannot exceed the process-wide system ceiling. |
| `max_document_pages` | integer or `None` | System limit, initially `1,000` | Positive per-call limit that cannot exceed the process-wide system ceiling. |
| `operation` | `Operation` or `None` | New operation | Pass an existing operation only when composing ingestion into a larger workflow. |

The task result is:

```json
{
  "knowledge": "engineering",
  "file": "system-design.pdf",
  "file_id": "a83fd91c20b4",
  "section_count": 8,
  "chunk_count": 21,
  "source_path": "documents/system-design.pdf"
}
```

An idempotently reused committed retry also includes
`"already_committed": true`.

### Ingestion progress contract

Every `ingestion.progress` payload includes the knowledge, file, generated
file ID, stage, status, and optional counters:

```json
{
  "knowledge": "engineering",
  "file": "system-design.pdf",
  "file_id": "a83fd91c20b4",
  "stage": "metadata_extraction",
  "status": "running",
  "completed": 3,
  "total": 8,
  "section_index": 3
}
```

Stages occur in this order:

| Stage | What is happening | Useful progress fields |
| --- | --- | --- |
| `source_validation` | File existence, readability, extension, and size are checked. | `size_bytes` on completion. |
| `source_snapshot` | An immutable private copy is created for the run/retry boundary. | Stage status. |
| `parsing` | TXT/Markdown native parsing or Docling PDF/DOCX parsing creates normalized elements. | Element count. |
| `sectioning` | Embedding-assisted semantic splitting creates provenance-aware sections. | Section count. |
| `metadata_extraction` | The LLM extracts summary, keywords, conditions, and definitions for each section. | Completed sections, total sections, current `section_index`. |
| `chunking` | Stored sections are divided into embedding-sized chunks. | Completed and total work as supplied by the event. |
| `embedding` | Chunk vectors are generated in batches. | Completed and total chunks. |
| `vector_storage` | Qdrant points are written. | Completed and total points. |
| `metadata_commit` | File and section records are committed to SQLite. | Commit status. |

Counters may be `null` for stages that cannot report meaningful totals. A UI
should display the stage/status text even when a percentage cannot be
calculated.

### Navigation and management

RAVEN exposes the same source hierarchy to applications that it exposes to the
agent's navigation tools:

```python
knowledges = await raven.list_knowledges()
files = await raven.list_knowledge_files("engineering")
sections = await raven.list_file_sections(
    "engineering",
    "system-design.pdf",
)
section = await raven.get_knowledge_section(
    "engineering",
    sections[0]["section_id"],
)
```

The section lookup returns metadata together with `raw_content`, so an
application can inspect a source directly without performing semantic
retrieval. Pageable variants are available for knowledge, file, and section
listings.

```python
details = raven.get_knowledge_details("engineering")

files = await raven.list_knowledge_files("engineering")
first_file = files[0]

sections = await raven.list_file_sections(
    "engineering",
    first_file["file_name"],
)

full_section = await raven.get_knowledge_section(
    "engineering",
    sections[0]["section_id"],
    file_name=first_file["file_name"],
)
print(full_section["raw_content"])
```

File deletion and knowledge deletion are operation-based:

```python
delete_file = await raven.delete_knowledge_file(
    "engineering",
    ingested_file["file_id"],
)
await delete_file.result()
```

### Ingestion safety and retry

RAVEN snapshots the submitted source before parsing so the file cannot silently
change underneath a running ingestion. Concurrent ingestion of the same file
is rejected, and an already committed file is not duplicated during a retry.

Vector writes happen before the SQLite metadata commit. On a normal error or
cancellation, RAVEN settles the in-flight write and removes uncommitted vectors.
Graceful shutdown also reconciles tracked pending writes. A hard process
termination bypasses that cleanup: retry uses the ingestion's stable `file_id`
to remove uncommitted vectors or reuse an already committed file. Startup marks
abandoned operations as interrupted; it does not perform a knowledge-wide
orphan-vector sweep.

Ingestion tasks persist the information required for user-confirmed retry. A
server upload remains in its private retry area only for the configured
retention window; after that window the source must be uploaded again.

Library ingestion snapshots are removed when the run exits because the
original trusted source path remains part of retry input. Server uploads are
different: the browser's original file is unavailable to the backend after the
request, so the server retains its private staged copy for
`upload_retry_retention_seconds`. A retry after that deadline fails with
`upload_retry_expired`; the user must upload the file again.

Duplicate behavior is based on the stored file name within one knowledge:

- a concurrent ingestion claim for that file is rejected;
- an already committed file raises `file_already_exists` during a new
  ingestion;
- retry reconciliation recognizes the same committed `file_id` and returns it
  with `already_committed=true` instead of duplicating vectors;
- deleting a file removes its vectors and records before that name can be
  ingested as a new file again.

Each knowledge stores `metadata.json`, `file.sqlite3`, and its local Qdrant
data in the user-scoped knowledge directory. Do not edit those files while
RAVEN is running. Corrupt or unsupported metadata is reported through
`list_discovery_issues()` instead of preventing unrelated valid resources from
starting.

RAVEN persistently stores parsed semantic sections and vectors, not a permanent
copy of the original uploaded document. The ingestion snapshot is temporary.
Applications that need original-file download or page navigation must retain
the original file in their own managed storage and associate it with the
returned `file_id`/file name. Reconstruction uses RAVEN's stored section text
and provenance; it does not require the original binary.


## Retrieval

RAVEN provides four retrieval strategies. Each searches the documents eligible
under a `RetrievalScope` within the current user's knowledge base.

| Strategy | Best suited for | Relative cost |
| --- | --- | --- |
| Embedded | Focused scientific, factual, or numeric questions where semantic precision matters most. | Lowest |
| Hierarchical | Broader context, narrative structure, and relationships that benefit from LLM metadata scoring. | High |
| Vector-conditioned | Vector narrowing followed by hierarchical scoring; a practical compromise for contextual questions. | Medium to high |
| Agreement | Comparing embedded and hierarchical results when an expensive cross-check is justified. | Highest |

The strategies perform different work:

- **Embedded** embeds the query and searches eligible chunks in Qdrant,
  deduplicates chunk hits by section, and returns the highest-ranked complete
  sections.
- **Hierarchical** scores section metadata in bounded batches. It first uses
  knowledge summaries to shortlist candidates when the scope covers the whole
  knowledge base, `full_retrieval=False`, and at least two knowledges have
  eligible files. Otherwise it skips knowledge shortlisting. The result includes
  the full content of the selected sections.
- **Vector-conditioned** shortlists distinct eligible files by their best
  matching chunk, then scores sections within those files.
- **Agreement** independently runs embedded and hierarchical retrieval and
  gives both branches the same resolved scope and result ceiling.

### Define a retrieval scope

Each `RetrievalSelection` has a `knowledge_name` and optional `file_id` and
`file_name`. Combine selections across knowledges. If they overlap, each
eligible file is included once.

| Selection | What it includes |
| --- | --- |
| `RetrievalScope()` | The user's entire knowledge base. |
| `knowledge_name` only | All files in the named knowledge, including files added later. |
| `knowledge_name` and `file_id` | One file by its stable ID. |
| `knowledge_name` and `file_name` | The current file with that name in the knowledge. |
| `knowledge_name`, `file_id`, and `file_name` | The file by ID, with the saved name required to match. |

Conversation creation validates explicit selections. Each search resolves the
selection again against the current catalog and fails if an explicitly
selected knowledge or file is missing. If both file fields are supplied and
they disagree, RAVEN raises `file_id_name_mismatch`.

```python
from nraven import RetrievalScope, RetrievalSelection

scope = RetrievalScope(selections=(
    RetrievalSelection(knowledge_name="engineering", file_name="system-design.pdf"),
    RetrievalSelection(knowledge_name="requirements"),
))
```

### Run retrieval directly

Use one of the four `Raven` retrieval methods. Each accepts a non-empty query,
a keyword-only `scope`, and an optional shared `operation`. Configure models
before searching. Await the returned task for the result.

```python
retrieval = await raven.retrieve_embedded(
    "What operating temperature does the controller require?",
    scope=scope,
    top_k=3,
)
sections = await retrieval.result()
```

| Method | Optional strategy arguments |
| --- | --- |
| `retrieve_embedded()` | `top_k` |
| `retrieve_hierarchical()` | `top_k_section`, `top_k_knowledge`, `full_retrieval`, `max_per_read`, `anchor_count` |
| `retrieve_agreement()` | `top_k`, `top_k_knowledge`, `full_retrieval`, `max_per_read`, `anchor_count` |
| `retrieve_vector_conditioned()` | `top_k_files`, `top_k_sections`, `max_per_read`, `anchor_count` |

The `top_k` arguments are ceilings. A search can return fewer results, and a
valid search with no evidence can return an empty list. For agreement,
`top_k` applies to each branch. For vector-conditioned retrieval,
`top_k_files` limits the shortlist and `top_k_sections` limits the final
ranked sections. `top_k_knowledge` is used only when whole-base hierarchical
shortlisting runs. Omitted arguments use the [runtime defaults](#runtimeconfig).

Except for agreement retrieval, every strategy returns the same bounded list:

```json
[
  {
    "knowledge_name": "engineering",
    "file_name": "system-design.pdf",
    "section_id": "a83fd91c20b4-3",
    "raw_content": "..."
  }
]
```

This is deliberately the complete model-facing retrieval contract. Internal
scores, chunks, and vector identifiers do not leak into the result.

### Agreement retrieval

Agreement retrieval preserves the relationship between its two source
strategies:

```json
{
  "agreement_type": "Disagreement",
  "retrieved_content": {
    "embedded_retrieval": [
      {
        "knowledge_name": "engineering",
        "file_name": "system-design.pdf",
        "section_id": "a83fd91c20b4-3",
        "raw_content": "..."
      }
    ],
    "hierarchical_retrieval": [
      {
        "knowledge_name": "engineering",
        "file_name": "requirements.pdf",
        "section_id": "fd05a621bd91-2",
        "raw_content": "..."
      }
    ]
  }
}
```

When both strategies select the same nonempty ranked sections, `agreement_type` is
`"Strong Agreement"` and `retrieved_content` is a single section list. Weak
agreement and disagreement retain the two lists so the model can reason about
their differences. Reconstruction accepts both shapes.

Agreement is classified as follows:

| Value | Meaning |
| --- | --- |
| `Strong Agreement` | Both branches return the same nonempty sequence of section identities. `retrieved_content` is one list. |
| `Weak Agreement` | The ranked section lists differ and share at least one source file. Both lists are retained. |
| `Disagreement` | The branches share no source files, including when either list is empty. Both lists are retained. |

Agreement is not a truth score. It reports whether two retrieval mechanisms
selected the same evidence. The model or application must still evaluate the
content.

Standalone calls choose the strategy by method or endpoint. Manual session
retrieval requires `embedded`, `hierarchical`, `agreement`, or
`vector_conditioned`. An agent turn also accepts `"auto"` so the model can
choose a strategy.

### Retrieval events and failures

Every strategy emits `started`, `completed`, and `failed` events for its own
family. Nested work—such as the embedded and hierarchical stages of agreement
or vector-conditioned retrieval—emits its own correlated child-task events in
the same operation stream. Completion payloads include scope, result count or
agreement type, and selected section IDs where applicable.

Common failures are:

- `knowledge_not_found`, `file_not_found`, or `file_id_name_mismatch` for a
  selection that cannot be resolved;
- `invalid_retrieval_mode` for an unknown session retrieval mode;
- `embedding_identity_mismatch` or `embedding_dimension_mismatch` when the
  configured embedding model is incompatible with stored vectors;
- `retrieval_scoring_invalid` when hierarchical structured output cannot be
  validated after its bounded retries;
- provider errors when embedding or LLM scoring fails.

Direct retrieval failures terminate its task. Inside the agent, recoverable
tool failures are returned to the model with a corrective `next_action`, as
described in [Tool results and recoverable errors](#tool-results-and-recoverable-errors).


## Conversations, sessions, and memory

A conversation and a session are intentionally different resources:

- A `Conversation` is persistent. It owns metadata, canonical messages,
  committed turns, manual activities, compacted context, vector memory, and
  preferences.
- A `Session` is temporary. It lets the caller run manual actions and agent
  turns against that conversation.

A session is never restored from disk. To continue an existing conversation,
load the conversation and create a new session around it.

### Create a conversation with a retrieval scope

Omit `scope` or pass `RetrievalScope()` to search the user's whole knowledge
base. Pass explicit selections to limit retrieval to selected knowledges or
files.

```python
create_task = await raven.create_conversation()
conversation = await create_task.result()
```

This example limits retrieval to a whole knowledge.

```python
from nraven import RetrievalScope, RetrievalSelection

scope = RetrievalScope(selections=(RetrievalSelection(knowledge_name="engineering"),))
create_task = await raven.create_conversation(scope=scope)
conversation = await create_task.result()
saved_scope = conversation.scope.model_dump(mode="json", exclude_none=True)
```

Conversation metadata stores `scope` alongside the conversation ID and
display fields.

```json
{
  "schema_version": 2,
  "conversation_id": "c36c45ac5575",
  "scope": {
    "selections": [
      {"knowledge_name": "engineering", "file_id": null, "file_name": null}
    ]
  },
  "title": "New Conversation",
  "is_titled": false,
  "pinned": false,
  "created_at": "2026-09-20T11:00:00+00:00"
}
```

Set the selection at creation. A whole-knowledge selection continues to
include newly ingested files. Create a new conversation to use a different
selection. Navigation, explicit source inspection, and reconstruction can
access the user's whole knowledge base.

### Conversation API reference

| Method | Inputs | Return/result | Notes |
| --- | --- | --- | --- |
| `create_conversation(scope=None)` | Optional `RetrievalScope` | Task result: `Conversation` | Omit the scope to search the whole knowledge base. |
| `get_conversation(id)` | Conversation ID | `Conversation` handle | Opens or reuses persistent storage lazily. |
| `get_conversation_details(id)` | Conversation ID | Metadata object | No mutation. |
| `list_conversations()` | None | All metadata records | No mutation. |
| `list_conversations_page(limit, after_conversation_id=None)` | Positive size and optional cursor | One metadata page | Use the final ID as the next cursor. |
| `update_conversation(id, title=None, pinned=None)` | One or both mutable fields | Task result: updated `Conversation` | A manual title sets `is_titled=true`; call `.to_dict()` for metadata. Supplying neither field is a no-op. |
| `rename_conversation(id, title)` | New title | Task result: updated `Conversation` | Marks the conversation titled. |
| `set_conversation_pinned(id, pinned)` | `True` to pin or `False` to unpin | Task result: updated `Conversation` | Updates the pin state. |
| `delete_conversation(id)` | Conversation ID | Task result on completion | Closes resources and removes persistent data. |
| `get_conversation_messages(id)` | Conversation ID | Complete canonical `ChatMessage` list | Never returns compacted-only model context. |
| `get_conversation_messages_page(id, limit, after_message_id=0)` | Conversation, positive size, numeric cursor | Durable message records | Suitable for UI history pagination. |
| `get_conversation_turn(id, turn_id)` | Conversation and turn UUID | Turn record | Includes stored `AgentRunResult` when available. |
| `get_conversation_turn_messages(id, turn_id)` | Conversation and turn UUID | Canonical messages for one turn | Includes user, tool, and assistant messages. |
| `get_conversation_turn_messages_page(id, turn_id, limit, after_message_id=0)` | Conversation, turn, positive size, numeric cursor | Durable message records for that turn | Uses the same global durable message IDs as full-history pagination. |
| `get_conversation_preferences(id)` | Conversation ID | Preference records | No mutation. |
| `save_conversation_preference(id, text)` | Non-empty preference text | Task result: preference record | Exact duplicate text reuses the existing record. |
| `remove_conversation_preference(id, preference_id)` | Stable preference UUID | Task result: removed record | Fails if the ID is invalid or absent. |
| `session(conversation, ...)` | A `Conversation` owned by this `Raven` | Temporary `Session` | Use `manual_only=True` for a session without an agent. |

`raven.list_discovery_issues()` returns validation problems found while
discovering both knowledge and conversation directories. A malformed resource
is excluded from normal lists, but other valid resources remain available.

```python
update = await raven.update_conversation(
    conversation.conversation_id,
    title="Thermal protection review",
    pinned=True,
)
updated_conversation = await update.result()

page = raven.list_conversations_page(limit=25)
if page:
    next_page = raven.list_conversations_page(
        limit=25,
        after_conversation_id=page[-1]["conversation_id"],
    )
```

A paged canonical message record is:

```json
{
  "message_id": 17,
  "turn_id": "7ad218ba-44c9-4f3d-b6d5-a870cbcb509c",
  "message_order": 2,
  "message": {
    "role": "tool",
    "content": "{\"ok\":true,\"evidence\":[...]}"
  }
}
```

`message_id` is a durable numeric pagination cursor. `message_order` is the
position inside that turn. In the Python API, `message` is a LlamaIndex
`ChatMessage` object; the JSON above illustrates its serialized representation.
Use `message.model_dump(mode="json")` before encoding it as JSON. Assistant
tool-call messages may use structured blocks rather than plain `content`.

A turn record is:

```json
{
  "turn_id": "7ad218ba-44c9-4f3d-b6d5-a870cbcb509c",
  "operation_id": "7e6d796c-6711-4fc7-a967-78738ecac96a",
  "user_query": "What does the design specify for thermal protection?",
  "result": {
    "operation_id": "7e6d796c-6711-4fc7-a967-78738ecac96a",
    "thinking": "...",
    "response": "...",
    "iteration_limit_reached": false,
    "evidence": [],
    "tool_calls": [],
    "reconstructed_sources": []
  },
  "memory_indexed": true,
  "committed_at": "2026-09-20T11:05:00+00:00"
}
```

### Running a turn

```python
from nraven import EventType


session = raven.session(conversation)
await session.start()

try:
    run = await session.generate_response(
        "What does the design specify for thermal protection?",
        retrieval_mode="auto",
    )

    async for event in run.stream:
        if event.type == EventType.CHAT_THINKING_DELTA:
            print(event.data.get("delta", ""), end="")
        elif event.type == EventType.CHAT_TOOL_CALL:
            print("Tool:", event.data.get("name"))
        elif event.type == EventType.CHAT_TOOL_RESULT:
            print("Tool result:", event.data.get("ui_summary"))
        elif event.type == EventType.RECONSTRUCTION_FILE:
            print("Source:", event.data.get("file_name"))
        elif event.type == EventType.CHAT_RESPONSE_DELTA:
            print(event.data.get("delta", ""), end="")

    result = await run.collect()
    print(result.response)
finally:
    await session.close()
```

`SessionRun` is the turn handle:

| Member | Meaning |
| --- | --- |
| `operation_id` | UUID used for status, events, cancellation, and server correlation. |
| `turn_id` | Stable UUID used to persist or idempotently reuse the conversation turn. |
| `status` | Current `OperationStatus` of the session task. |
| `stream` / `events(after_event_id=0)` | Fresh replayable iterator over this run and its nested tasks. |
| `collect()` | Waits for the session task and returns its `AgentRunResult`, after turn persistence. The agent assembles that result from events; collecting does not start another inference. |
| `wait()` | Waits and returns terminal status without returning/raising the run result. |
| `cancel()` | Cancels the turn, waits for terminal cancellation, and releases the conversation claim. |

`collect()` returns:

```json
{
  "operation_id": "7e6d796c-6711-4fc7-a967-78738ecac96a",
  "thinking": "accumulated thinking deltas",
  "response": "final assistant response",
  "iteration_limit_reached": false,
  "evidence": [
    {
      "knowledge_name": "engineering",
      "file_name": "system-design.pdf",
      "section_id": "a83fd91c20b4-3"
    }
  ],
  "tool_calls": [
    {
      "call_id": "call-1",
      "step": 1,
      "name": "embedded_retrieval",
      "ok": true,
      "ui_summary": {}
    }
  ],
  "reconstructed_sources": []
}
```

For a new turn, the session worker assembles `AgentRunResult` from the
persisted agent events and saves the turn before returning the result.
`run.collect()` waits for that session task and returns its result. It can be
called after consuming `run.stream` and does not start another inference.

If the task fails or is cancelled, `run.collect()` raises. Use the events
already received for partial UI display and inspect the operation or task error
for the cause.

`session.generate_response()` accepts a non-empty `user_query`, an optional
`retrieval_mode`, and—only for component composition—an existing `operation`.
Omitting `retrieval_mode` or passing `"auto"` exposes all four retrieval tools.
Passing an exact strategy exposes that retrieval tool as described in
[Tool exposure and retrieval constraints](#tool-exposure-and-retrieval-constraints).

`Raven.session()` controls per-session defaults:

| Argument | Default | Constraint/effect |
| --- | --- | --- |
| `conversation` | Required | Must be the actual `Conversation` object owned by this `Raven` instance; an object from another runtime raises `foreign_conversation`. |
| `manual_only` | `False` | Set `True` to open a session for manual actions without an agent. Manual retrieval still requires configured models. |
| `max_iterations` | Runtime setting, initially `10` | Positive and no greater than the system maximum, initially `50`. |
| `memory_token_limit` | Runtime setting, initially `4,000` | Positive model-context memory budget. Extremely small values can fail memory initialization. |
| `memory_top_k` | Runtime setting, initially `5` | Positive maximum number of matched memory nodes per search; currently one node represents one turn and may return both user and assistant messages. |

Starting a session opens the conversation's storage. Memory initialization and
reconciliation using the configured model pair happen when
`generate_response()` submits a turn, before its session task is created.
Closing a session cancels its active turns and manual tasks. The shared
conversation remains owned by `ConversationManager`.

Each run also has a stable `turn_id`. A committed turn contains the user
message, assistant tool-call messages, tool-result messages, and final
assistant response. Internal thinking is streamed for the caller but is not
inserted into the canonical model-compatible message sequence.

The canonical message sequence for a tool-using turn is:

```text
user message
assistant tool-call message
tool result message
[additional assistant tool-call and tool result messages]
assistant final response
```

Persisting tool calls and results lets later model context remember what was
used and lets `reconstruct_from_turn()` recover evidence. Thinking is excluded
from those canonical messages, not from all persistent storage: it remains in
operation events and may also be present in the turn's saved `AgentRunResult`
snapshot in `messages.sqlite3`.

After a turn is committed, the session attempts a separate title completion if
the conversation is still untitled. That completion does not enter the chat
history. Once a title has been generated or set explicitly, later turns do not
regenerate it. Applications may update the title or pinned state explicitly.

Title generation is best-effort: a title failure does not discard an otherwise
successful answer or turn commit. Until generation succeeds or the application
sets a title, metadata remains `title="New Conversation"` and
`is_titled=false`. A later completed turn can retry generation using that turn's
query. A manual title update marks the conversation titled so automatic
generation will not overwrite it.

### Turn serialization

Multiple temporary sessions may reference the same conversation, which is
useful when the same account has several browser tabs or application windows.
Only one turn may run against a conversation at a time. A competing turn is
rejected instead of racing message, memory, and preference mutations.

Turns in different conversations can run concurrently.

A competing same-conversation turn fails with `conversation_turn_active`. The
caller should wait for, cancel, or observe the active operation before
submitting another turn. RAVEN does not queue an unbounded backlog for one
conversation because the second turn's context would otherwise be ambiguous.

Turn commits are atomic and idempotent. If retry encounters an already
committed `turn_id`, RAVEN verifies the query, reconciles memory indexing, and
reuses the stored result rather than generating and storing a duplicate turn.

### Context and vector memory

RAVEN maintains two complementary memory views:

- **Compacted context** uses LlamaIndex's summary memory to keep recent
  messages and a summary of older history within the configured token limit.
  This is the context passed to the model.
- **Vector memory** indexes the original user query and final assistant
  response for semantic recall. The manual activity attached to a user
  message stays in the canonical transcript and activity log.

The complete canonical history remains available to the application even when
the model-facing context has been compacted:

```python
messages = await raven.get_conversation_messages(
    conversation.conversation_id,
)
```

Compaction does not replace or delete canonical messages in
`messages.sqlite3`. It updates the derived model-context summary used by the
LlamaIndex memory layer. Vector memory keeps its entries for the original
query and response across compactions.

The agent uses `search_memory(query)` to recall an earlier discussion. Each
entry in the tool's `result` list contains a `turn_id` and the recalled
`messages` for that turn. If the turn has linked activities, the entry also
includes an `activity_action` instruction with the exact ID to pass to
`list_activities_from_turn`.

For example, asking “Which file did I open before we discussed safe-hold?”
gives the agent a topic to search for. A match with linked activities looks
like this:

```json
{
  "turn_id": "4dbef684-e9c5-4b4f-8a87-0b390745912c",
  "messages": [
    {"role": "user", "content": "Let's review the safe-hold deadline."},
    {"role": "assistant", "content": "The deadline is 95 milliseconds."}
  ],
  "activity_action": "This turn has recorded user activities. If the user asks what they did, call list_activities_from_turn with turn_id \"4dbef684-e9c5-4b4f-8a87-0b390745912c\" before answering."
}
```

The activity lookup reads the saved records even after compaction. The hint
contains no activity records itself. Memory results support conversation
recall. Document claims still require source evidence.

Compaction emits started, completed, and failed events so a UI can explain a
delay instead of appearing stalled.

### Conversation preferences

Preferences are local to one conversation. Each preference has a stable UUID
and its text:

```json
{
  "preference_id": "90297ef3-5336-4d09-a155-edf21408e242",
  "text": "Use concise answers unless I ask for detail."
}
```

The agent can list, save, and remove preferences when the user explicitly asks
it to do so. Removal uses `preference_id`, avoiding fragile exact-text
matching. Preference changes invalidate the conversation's cached prompt and
apply on the next agent run.

Preferences are conversation-local, not global. They are injected into the
conversation's system policy when that prompt is next built. Saving a
preference does not rewrite prior turns, and removing one does not erase text
that already appears in canonical messages.

Applications can manage the same records directly:

```python
save_task = await raven.save_conversation_preference(
    conversation.conversation_id,
    "Use concise answers unless I ask for detail.",
)
preference = await save_task.result()

remove_task = await raven.remove_conversation_preference(
    conversation.conversation_id,
    preference["preference_id"],
)
await remove_task.result()
```


## Manual session activity

Use a session when the agent should be able to follow up on work the user does
manually. Direct `Raven` retrieval, navigation, and reconstruction methods work
outside conversation history.

| Session method | Inputs | Result |
| --- | --- | --- |
| `list_knowledges()` | No required arguments | Task returning knowledge metadata. |
| `list_files(knowledge_name)` | Knowledge name | Task returning file metadata. |
| `list_sections(knowledge_name, file_id, ...)` | Knowledge name and stable file ID. Optional `limit` (default `20`, maximum `50`), `after_section_index` (default `0`), and `get_content` (default `False`). | Task returning a page of section records. Set `get_content=True` to include `raw_content`. |
| `open_file(knowledge_name, file_id, ...)` | Knowledge name and file ID. Optional `origin_activity_id` and `origin_turn_id`. | Task returning the file record. |
| `open_section(knowledge_name, section_id, ...)` | Knowledge name and section ID. The same optional origin IDs apply. | Task returning the stored section, including its text. |
| `retrieve(query, retrieval_mode=...)` | Query and one of the four explicit strategies | Task returning the native retrieval result within the saved conversation scope. |
| `reconstruct(retrieval_result)` | A section list or agreement result | Task returning reconstructed files. |
| `remove_from_activity(activity_id)` | Activity UUID | `True` if a pending entry was deleted, `False` if absent. Captured and committed entries raise an error. |

The task methods accept an optional `operation`. Successful manual actions
are recorded. To refresh an activity panel, call
`raven.list_activities(conversation_id)` directly. For passive catalog
browsing, use the direct knowledge APIs.

```python
from nraven import Conversation, Raven


async def search_then_chat(raven: Raven, conversation: Conversation) -> str:
    session = raven.session(conversation)
    await session.start()
    try:
        search = await session.retrieve(
            "What are the thermal protection requirements?",
            retrieval_mode="embedded",
        )
        sections = await search.result()
        if sections:
            hit = sections[0]
            opened = await session.open_section(hit["knowledge_name"], hit["section_id"])
            await opened.result()

        run = await session.generate_response("What did I just search for and open?")
        return (await run.collect()).response
    finally:
        await session.close()
```

### How activities reach a turn

Each activity has an `activity_id`, `sequence`, `operation_id`, `kind`,
`started_at`, `recorded_at`, `details`, and an optional `turn_id`. The activity
database stores compact references. For instance, a retrieval entry keeps the
query, strategy, scope selection, settings, and ordered section references.

When a turn starts, RAVEN fits a batch of pending activities into its context
budget. The original user message for that turn stores the same JSON supplied
to the model.

```json
{
  "activities": [
    {
      "activity_id": "855a028b-f40b-40e2-b220-0b460c23adec",
      "context_text": "Opened section engineering/system-design.pdf/a83fd91c20b4-3."
    }
  ],
  "current_query": "What did I just open?"
}
```

On commit, the captured activities receive that turn's ID. Entries recorded
after capture, entries that did not fit, and entries from a turn cancelled or
failed before commit remain pending. A pending entry can be removed until a
running turn captures it. The saved turn remains the source of truth for its
activity links after compaction or restart.

Use `await raven.list_activities(conversation_id)` to read pending and linked
actions from Python. Pass `turn_id` to read only actions linked to that turn.
The method returns `Activity` records in ascending sequence order.
`limit` defaults to `50` and accepts `1` through `200`. For the next page,
set `after_sequence` to the last record's `sequence` and keep the same turn
filter. A page shorter than `limit` marks the end of the available records.


## Agent harness

The agent harness owns model behavior; it does not own conversation storage.
For each run it creates a fresh LlamaIndex `FunctionAgent`, builds the current
conversation policy and prompt, exposes only permitted tools, translates
workflow output into RAVEN events, tracks evidence, reconstructs sources, and
normalizes failures.

RAVEN does not classify the user's query with a separate brittle intent
classifier. The model decides whether it should:

- answer directly;
- search conversation memory;
- inspect the conversation's retrieval scope;
- navigate knowledge names, files, and sections;
- retrieve relevant sections;
- inspect a specific section;
- modify an explicit conversation preference;
- or use several tools sequentially.

Tool calls are sequential by default. The system prompt tells the model to use
the smallest sufficient tool sequence and not to call additional retrieval or
navigation tools merely to reconfirm evidence it already has.

### Tool exposure and retrieval constraints

Tool availability is enforced by construction, not merely requested in the
system prompt:

| Conversation/run | Retrieval tools exposed |
| --- | --- |
| `"auto"` or omitted | All four retrieval tools. |
| `embedded`, `hierarchical`, `agreement`, or `vector_conditioned` | Only the corresponding retrieval tool. |

`"auto"` and `None` both mean automatic mode; there is no `auto` retrieval
tool. Automatic mode changes which concrete tools are exposed and lets the
agent choose among them. A restricted mode physically omits the other
retrieval tools from the `FunctionAgent` configuration.

> **Small-model behavior:** Automatic tool selection depends on the selected
> model's instruction-following and function-calling ability. Small models may
> occasionally answer from their general knowledge, claim that they cannot
> access the knowledge base, or refuse a safety-sensitive question—especially
> a critical medical question—without calling a retrieval tool. When grounding
> matters, tell the model explicitly to use retrieval and answer only from the
> retrieved evidence. Set the conversation scope to the documents intended for
> that search. Selecting a concrete `retrieval_mode` limits the available
> strategy. The model still decides whether to call it. Retrieved medical material
> is still reference information and is not a substitute for qualified medical
> judgment.

The exact retrieval tool names and arguments are:

| Tool | Arguments | Use |
| --- | --- | --- |
| `embedded_retrieval` | `query` | Specific facts, exact details, definitions, numbers, and requirements. |
| `hierarchical_retrieval` | `query` | Narrative context, chronology, themes, and relationships. |
| `vector_conditioned_retrieval` | `query` | General-purpose balance of factual relevance, context, and compute cost. |
| `agreement_retrieval` | `query` | Expensive cross-check when independent searches should corroborate evidence. |

The wrappers supply the conversation's scope and the turn's runtime retrieval
settings. Every tool below stays available when the caller selects a specific
retrieval strategy. Navigation covers the user's entire knowledge base.

| Tool | Arguments | Result or effect |
| --- | --- | --- |
| `get_retrieval_scope` | None | Returns the saved `RetrievalScope`. A whole knowledge is shown by name, and an empty `selections` list means the entire knowledge base. |
| `list_knowledges` | None | Returns knowledge metadata. |
| `list_files` | `knowledge_name` | Returns stored file records in the named knowledge. |
| `list_sections` | `knowledge_name`, `file_name`. Optional `get_content`, `after_section_index`, `limit`. | Returns section IDs and metadata without `raw_content` by default. Default page size `20`, maximum `50`. With `get_content=true`, default `3`, maximum `5`. Use `ui_summary.next_after_section_index` to continue. |
| `get_sections` | `knowledge_name`, `section_ids` (1–5 IDs) | Returns sections in requested order with metadata and raw text. Successful lookups supply evidence. |
| `search_memory` | `query` | Returns semantically relevant messages grouped by `turn_id`. Matches with linked activities include an `activity_action` hint for looking them up. |
| `list_activities_from_turn` | Required `turn_id`, optional `after_sequence` | Reads up to five recorded actions linked to a specific turn. Use `ui_summary.next_after_sequence` with the same turn ID for another page. |
| `list_all_activities` | Optional `after_sequence` | Reads up to five recorded actions in conversation sequence order, including pending and turn-linked actions. Use `ui_summary.next_after_sequence` for another page. |
| `list_preferences` | None | Returns this conversation's preference IDs and text. |
| `save_preference` | `text` | Persists an explicit enduring preference for future runs. |
| `remove_preference` | `preference_id` | Removes the exact saved preference. |

The activity tools read the current conversation and return records in
ascending `sequence` order. Use `list_activities_from_turn` when the question
refers to a particular exchange. Use `list_all_activities` for general history,
including actions that are still pending. Both return up to five records per
call. If `ui_summary.next_after_sequence` is a number, pass it as
`after_sequence` on the next call. For a turn-specific lookup, pass the same
`turn_id` again. A `null` cursor marks the last page.

The harness exposes these tools to the model. Application code reads activity
history through `Raven.list_activities()` or
`GET /api/v1/conversations/{conversation_id}/activities`. Reading the history
creates no activity entry. The high-level Raven navigation methods return
complete stored section records.

For example, an agent can inspect a section metadata page after
section index 2 without sending raw section content to the model:

```json
{"knowledge_name": "engineering", "file_name": "system-design.pdf", "after_section_index": 2, "limit": 1}
```

The `list_sections` tool returns a `ToolResult` like:

```json
{
  "ok": true,
  "result": [
    {
      "section_id": "a83fd91c20b4-3",
      "section_index": 3,
      "summary": "Thermal protection requirements",
      "keywords": ["thermal", "shutdown"],
      "conditions": ["temperature exceeds the configured limit"],
      "definitions": [],
      "source_range": [2, 3]
    }
  ],
  "ui_summary": {
    "kind": "navigation",
    "operation": "list_sections",
    "knowledge_name": "engineering",
    "file_name": "system-design.pdf",
    "section_ids": ["a83fd91c20b4-3"],
    "has_more": true,
    "next_after_section_index": 3
  },
  "evidence": []
}
```

When `has_more` is true, the next call can pass
`"after_section_index": 3`. The cursor is exclusive. `get_content=true` adds `raw_content` to each
listed section and makes those returned sections evidence; it is intended for
explicit sequential inspection, not general question answering.

To read selected sections instead, pass IDs from the metadata page to
`get_sections` (up to five IDs from one knowledge per call):

```json
{"knowledge_name": "engineering", "section_ids": ["a83fd91c20b4-3", "a83fd91c20b4-4"]}
```

Its `result` is a list in the requested order. Each item includes its own
`section_id`, `file_name`, `knowledge_name`, metadata, and complete
`raw_content`. Its `evidence` and `ui_summary.sections` contain only the
knowledge/file/section references. Invalid page sizes or more than five section IDs return
`ok=false`; a missing ID also fails the whole `get_sections` call rather than
returning a partial list.

The prompt directs general document questions to retrieval and explicit
browsing or source inspection to navigation. An unknown retrieval mode is
rejected before model execution.

### Tool results and recoverable errors

Inside the agent, every tool result separates model-facing content from bounded
UI metadata:

```json
{
  "ok": true,
  "result": [
    {
      "knowledge_name": "engineering",
      "file_name": "system-design.pdf",
      "section_id": "a83fd91c20b4-3",
      "raw_content": "..."
    }
  ],
  "ui_summary": {
    "kind": "retrieval",
    "section_count": 2,
    "sections": [
      {
        "knowledge_name": "engineering",
        "file_name": "system-design.pdf",
        "section_id": "a83fd91c20b4-3"
      }
    ]
  },
  "evidence": [
    {
      "knowledge_name": "engineering",
      "file_name": "system-design.pdf",
      "section_id": "a83fd91c20b4-3"
    }
  ]
}
```

| Field | Model receives it | Published in `chat.tool_result` | Meaning |
| --- | --- | --- | --- |
| `ok` | Yes | Yes | Whether the tool completed its requested action. |
| `result` | Yes | No | Full model-facing data, including raw section content when needed. |
| `ui_summary` | Yes | Yes | Bounded metadata safe for ordinary UI rendering. |
| `evidence` | Yes | Yes | Deduplicable knowledge/file/section references. |
| `error` | On failure | On failure | Stable safe error payload. |
| `next_action` | When useful | When useful | Concrete correction the model should take after a recoverable error. |

The model-facing result can contain the complete context needed to answer. The
bounded `ui_summary` gives the frontend safe metadata for rendering. The
published `chat.tool_result` event contains `ok`, `ui_summary`, evidence, and
error information when applicable; it deliberately omits the full model-facing
`result` so raw sections are not duplicated in the event stream.

Recoverable mistakes—such as an invalid knowledge name, file name, section ID,
or preference ID—are returned to the model as `ok=false` with an error and a
concrete `next_action`. This lets the model correct its arguments and retry.
Cancellation, infrastructure failures, and unrecoverable persistence failures
terminate the operation.

### Evidence and completion behavior

Successful retrieval and direct section inspection produce evidence references
containing only `knowledge_name`, `file_name`, and `section_id`. The harness
deduplicates these references and uses them for source reconstruction.

When the agent has attempted to use knowledge tools but obtains no valid
evidence, RAVEN does not accept an unsupported knowledge answer. It emits a safe
insufficient-evidence response instead. Conversation memory is not treated as
evidence for a knowledge-base fact.

The harness enforces a maximum number of agent iterations. If the limit is
reached, it records that state and asks the agent engine for an early final
response. Failure to produce that response becomes an observable failed
operation rather than an incomplete success.

Reaching the limit emits `chat.max_iterations` and sets
`AgentRunResult.iteration_limit_reached=true` if an early final response is
successfully produced. If early finalization fails, `collect()` raises and the
operation ends as failed. The harness never reports a silently truncated
successful answer.

### Prompt and preference lifecycle

The harness builds a conversation-specific system policy from stable RAVEN
behavior plus that conversation's current preferences. It does not enumerate
the available tools in prompt text because LlamaIndex supplies the actual tool
schemas to the model. Scope and retrieval restrictions are enforced by the
tool catalogue itself.

The prompt is cached per conversation. Saving or removing a preference
invalidates that conversation's cached prompt; the next run rebuilds it. This
avoids rebuilding an unchanged prompt for every turn while ensuring preference
changes take effect at the correct boundary.


## Source reconstruction

Source reconstruction is RAVEN's transparency layer. It takes retrieval
evidence, groups section IDs by knowledge and file, reloads the complete stored
section sequence, and marks the sections used by the model.

A reconstructed file has this shape:

```json
{
  "knowledge_name": "engineering",
  "file_name": "system-design.pdf",
  "navigation_type": "page",
  "sections": [
    {
      "section_id": "a83fd91c20b4-1",
      "highlighted": false,
      "raw_content": "...",
      "source_range": [1, 2]
    },
    {
      "section_id": "a83fd91c20b4-3",
      "highlighted": true,
      "raw_content": "...",
      "source_range": [2, 2]
    }
  ]
}
```

For page-based documents, `[12, 12]` means the section belongs to page 12 and
`[12, 14]` means it spans pages 12 through 14. Text and Markdown sources use
`navigation_type="none"` and may have `source_range=null`.

Reconstruction preserves semantic sections and source navigation provenance;
it is not intended to reproduce the original file's pixel-perfect layout.

### Accepted reconstruction input

`Raven.reconstruct()` accepts three forms:

1. `None` or `[]`, which completes successfully with `[]`.
2. The common retrieval section list. `raw_content` may be present but is not
   required because reconstruction reloads canonical stored sections:

   ```json
   [
     {
       "knowledge_name": "engineering",
       "file_name": "system-design.pdf",
       "section_id": "a83fd91c20b4-3"
     }
   ]
   ```

3. An agreement retrieval object whose `retrieved_content` is either one
   section list or an object containing `embedded_retrieval` and
   `hierarchical_retrieval` lists.

Each reference must contain non-empty `knowledge_name`, `file_name`, and
`section_id` strings. Duplicate references are removed. References are grouped
by knowledge and file, then the complete stored section sequence for each file
is returned with matching IDs highlighted.

### Automatic reconstruction

During an agent turn, the harness automatically reconstructs files whenever a
successful tool result provides knowledge evidence. Each reconstructed file is
published as `reconstruction.file`, and `run.collect()` includes all files in
`reconstructed_sources`.

### Direct reconstruction

Any compatible retrieval result can be reconstructed independently of the
agent:

```python
from nraven import RetrievalScope

retrieval = await raven.retrieve_embedded(
    "What does the documentation say about thermal protection?",
    scope=RetrievalScope(),
)
retrieved_sections = await retrieval.result()

reconstruction = await raven.reconstruct(retrieved_sections)
files = await reconstruction.result()
```

The method accepts the common section-list contract and the agreement-retrieval
contract.

`reconstruction.file` is emitted once per reconstructed file and contains the
same object that appears in the task result. `reconstruction.completed`
contains `file_count` and the total number of sections included across the
reconstructed files.

### Reconstruction from a persisted turn

Tool results are stored with their conversation turn, so reconstruction can be
requested again later without repeating retrieval:

```python
reconstruction = await raven.reconstruct_from_turn(
    conversation.conversation_id,
    run.turn_id,
)
files = await reconstruction.result()
```

If the selected turn contains no successful knowledge evidence—for example, a
casual conversation with no tool calls—the method returns an empty list.

The turn lookup examines successful persisted tool-result messages and
deduplicates their evidence references. It does not repeat retrieval, and it
does not depend on the original session still existing.

### Reconstruction failures and retry

- malformed input raises `invalid_metadata` before source loading;
- a deleted or unknown knowledge raises `knowledge_not_found`;
- a deleted or unknown file raises `file_not_found`;
- a section ID that does not belong to the named file raises
  `section_not_found` and reports the missing IDs;
- an unknown conversation or turn raises the corresponding conversation/turn
  not-found error.

Direct reconstruction stores normalized evidence as retry input and is
eligible for user-confirmed retry after approved recoverable failures. A
previous turn can also be reconstructed again by calling
`reconstruct_from_turn()` because the evidence remains in canonical tool-result
messages.


## FastAPI server quick start

Install the server dependencies and start RAVEN with an explicit storage home:

```bash
pip install "noomexai-raven[server]"
nraven serve --home /path/to/raven-home
```

`--home` is mandatory. On its first start, the server creates the default
system configuration at:

```text
<raven-home>/system_settings/system_settings.json
```

Pass `--system-settings /path/to/system_settings.json` to use another settings
file instead. Host, port, and log level come from that immutable
`SystemConfig`; the defaults are `127.0.0.1`, `8765`, and `info`.

The CLI starts one Uvicorn worker. RAVEN's embedded Qdrant databases and local
SQLite stores are intentionally owned by one server process, so increasing the
Uvicorn worker count is not supported.

At launch, the server:

- obtains an exclusive lock for the supplied RAVEN home;
- creates a random bearer token;
- writes the token to `<raven-home>/temp/server_auth/bearer_token` for the
  owning application or deployment host;
- starts the user-runtime registry and recovery services;
- removes the token and releases the home lock on normal shutdown.

A second server cannot own the same home concurrently. Treat the bearer-token
file like a password and do not expose it through a public web root.

The generated API contract is available at:

- `http://127.0.0.1:8765/docs`
- `http://127.0.0.1:8765/openapi.json`

Those routes, along with liveness and readiness probes, are intentionally
available without the bearer token. Application routes require it.


## Using RAVEN from a web or desktop UI

The HTTP interface follows the same operation-first design as the Python API.
A normal UI flow is:

1. Start or connect to the RAVEN server.
2. Obtain the launch bearer token through the trusted application or host
   layer.
3. Configure the LLM and embedding-model pair.
4. Create a knowledge database and upload documents.
5. Consume the returned operation's SSE stream until ingestion finishes.
6. Run standalone retrieval and reconstruction, or create a conversation with
   a retrieval scope to retain manual activity.
7. Submit a turn when the user wants the agent to continue, and render its
   typed events as they arrive.
8. Reconnect with `Last-Event-ID` if the connection drops.
9. Cancel an active operation or explicitly retry an eligible failed task when
   needed.

### Desktop applications

The default server configuration uses RAVEN's persisted default user UUID, so
a single-user desktop application can omit `X-Raven-User-ID`. The native
application process reads the launch token locally and attaches it to requests
made by, or proxied for, its UI. A browser view should not independently read
credentials from the filesystem.

A desktop request therefore normally contains:

```http
POST /api/v1/conversations HTTP/1.1
Host: 127.0.0.1:8765
Authorization: Bearer <launch-token>
Content-Type: application/json

{}
```

### Hosted applications

In a hosted deployment, RAVEN is an internal backend behind the host's own
application server. The host authenticates the end user, authorizes access to
an internal UUID, and forwards that UUID in `X-Raven-User-ID`. Set
`require_user_id_header` to `true` in `SystemConfig` so requests without that
context are rejected.

The host-to-RAVEN request contains both credentials:

```http
POST /api/v1/conversations HTTP/1.1
Host: raven.internal:8765
Authorization: Bearer <launch-token>
X-Raven-User-ID: 4b9bd24b-61de-42ed-9176-d2db91b66702
Content-Type: application/json

{"scope":{"selections":[{"knowledge_name":"engineering"}]}}
```

This example assumes `raven.internal` has been added to `allowed_hosts` in the
server's `SystemConfig`.

RAVEN validates the UUID and isolates that user's files, databases,
operations, and settings beneath a separate directory. It does not authenticate
the public account or decide which UUID that account may use; those are host
application responsibilities.

### Streaming operation events

Native browser `EventSource` cannot attach the required `Authorization`
header. Use `fetch()` with an SSE parser, or proxy the stream through the host
application. This compact TypeScript example preserves the last successfully
received event ID for reconnection:

```typescript
type RavenEvent = {
  event_id: number;
  operation_id: string;
  task_id: string | null;
  task_name: string | null;
  type: string;
  timestamp: string;
  data: Record<string, unknown>;
  is_final: boolean;
};

async function streamOperation(
  eventsUrl: string,
  token: string,
  onEvent: (event: RavenEvent) => void,
  options: {
    userId?: string;
    lastEventId?: number;
    signal?: AbortSignal;
  } = {},
): Promise<number | undefined> {
  const headers: Record<string, string> = {
    Authorization: `Bearer ${token}`,
  };
  if (options.userId) headers["X-Raven-User-ID"] = options.userId;
  if (options.lastEventId !== undefined) {
    headers["Last-Event-ID"] = String(options.lastEventId);
  }

  const response = await fetch(eventsUrl, {
    headers,
    signal: options.signal,
  });
  if (!response.ok || !response.body) {
    throw new Error(`SSE request failed: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";
  let cursor = options.lastEventId;

  while (true) {
    const {value, done} = await reader.read();
    buffer += decoder.decode(value, {stream: !done});
    const frames = buffer.split(/\r?\n\r?\n/);
    buffer = frames.pop() ?? "";

    for (const frame of frames) {
      if (!frame || frame.startsWith(":")) continue;
      const data = frame
        .split("\n")
        .filter((line) => line.startsWith("data:"))
        .map((line) => line.slice(5).trimStart())
        .join("\n");
      if (!data) continue;

      const event = JSON.parse(data) as RavenEvent;
      onEvent(event);
      cursor = event.event_id;
    }

    if (done) return cursor;
  }
}
```

Persist `cursor` only after the UI has successfully handled the event. On
reconnect, send that value as `Last-Event-ID`; RAVEN replays retained events
after it and then waits for new events. SSE heartbeat comments keep idle
connections alive. The stream closes after a final event. A cursor ahead of
the operation's available event history returns `410 Gone`. A request for an
operation that cleanup removed returns `404 Not Found`. The application or
deployment host schedules operation cleanup. See
[Backup, restore, and cleanup](#backup-restore-and-cleanup).

Render event types independently: thinking is not response text, tool calls
and bounded tool results describe agent work, `reconstruction.file` supplies
verifiable sources, and response deltas form the user-visible answer.

For a desktop UI, the trusted native process obtains the launch token and
forwards the requests in this order: `POST /provider/models/configure`,
`POST /knowledges`, `POST /knowledges/{name}/files`, follow its `events_url`,
`POST /conversations`, `POST /conversations/{id}/turns`, follow that new
`events_url`, then `GET /conversations/{id}/turns/{turn_id}` and `/messages`
for durable display. All paths in this paragraph are under `/api/v1`.
The UI should store the returned `conversation_id` as its chat identity and
the returned `turn_id` as the durable identity of the generated turn; neither
is the SSE `event_id`.

For a hosted UI, the browser calls its **host backend**, which authenticates
the account and authorizes its UUID. The host backend supplies RAVEN's
per-launch bearer token and the authorized `X-Raven-User-ID` on each internal
request, including SSE reconnects. The browser does not choose that UUID or
receive the launch token. The same sequence above then applies per user.

Within one operation, `operation_id` selects the stream, `task_id` identifies
a root or nested task, `event_id` is the replay cursor, and `call_id` matches
an agent's `chat.tool_call` with its `chat.tool_result`. If a connection drops,
reopen the same operation stream with the last successfully handled event ID;
do not submit the turn a second time just to recover deltas.

HTTP clients can also search or reconstruct sources without a conversation.
Use one of the four strategy-specific retrieval routes or
`POST /api/v1/reconstruction`, then follow the returned `events_url` for
results. See [Standalone retrieval and reconstruction requests](#standalone-retrieval-and-reconstruction-requests).


## HTTP API overview

Every application route uses versioned paths and typed Pydantic contracts.
This table summarizes the implemented surface; use `/docs` or
`/openapi.json` for exact request fields, response schemas, query parameters,
and error bodies.

| Area | Representative routes |
| --- | --- |
| Health | `GET /api/v1/health/live`, `GET /api/v1/health/ready` |
| Runtime | Status, revision-safe settings read/update/reset, and discovery issues under `/api/v1/runtime` |
| Configured models | `GET /api/v1/provider/models/configured`, `POST /api/v1/provider/models/configure`, configured-role preload and unload routes |
| Ollama administration | Status, list, inspect, pull, and delete routes under `/api/v1/providers/ollama` |
| Operations | List and inspect operations/tasks, stream events, cancel operations, discover retryable tasks, and retry a task |
| Knowledge | CRUD, file listing/deletion, section navigation, statistics, multipart upload, and trusted-path ingestion under `/api/v1/knowledges` |
| Standalone search and reconstruction | Four strategy-specific POST routes under `/api/v1/retrieval` and `POST /api/v1/reconstruction` |
| Conversations | CRUD, saved scope, title, pin state, canonical messages, turns, preferences, and reconstruction under `/api/v1/conversations` |
| Manual activity | Browse, inspect sources, search, reconstruct, and manage activity history under `/api/v1/conversations/{conversation_id}/activities` |

### Authentication and errors

Send the launch token on protected routes:

```http
Authorization: Bearer <launch-token>
```

Include `X-Raven-User-ID: <uuid>` when hosted user context is enabled. Errors
share one machine-readable envelope and request ID:

```json
{
  "error": {
    "code": "knowledge_not_found",
    "message": "Knowledge 'engineering' does not exist.",
    "details": {}
  },
  "request_id": "632d4f96-1e5f-4a4b-b1fd-f6167a742f01"
}
```

The same request ID is returned in `X-Request-ID` for correlation with server
logs.

### Configuring models

Configure one validated LLM and embedding pair:

```http
POST /api/v1/provider/models/configure HTTP/1.1
Authorization: Bearer <launch-token>
Content-Type: application/json

{
  "llm": {
    "provider": "ollama",
    "model": "qwen3:8b",
    "role": "llm",
    "api_key_ref": null,
    "options": {}
  },
  "embedding": {
    "provider": "ollama",
    "model": "bge-m3",
    "role": "embedding",
    "api_key_ref": null,
    "options": {}
  }
}
```

Long-running commands return `202 Accepted` with an operation reference:

```json
{
  "operation_id": "cc9dce11-16ca-42be-b418-18588d03a571",
  "task_id": "8f708231-4a27-40cb-a51e-477339c167ad",
  "status": "running",
  "events_url": "/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/events",
  "retry_of_operation_id": null,
  "retry_of_task_id": null
}
```

The initial status may be `queued` or `running`. Follow `events_url` for
progress and terminal state, or inspect the task directly.

### Uploading and ingesting a document

Browser and desktop clients should use multipart upload:

```http
POST /api/v1/knowledges/engineering/files HTTP/1.1
Authorization: Bearer <launch-token>
Content-Type: multipart/form-data; boundary=...

--...
Content-Disposition: form-data; name="file"; filename="system-design.pdf"
Content-Type: application/pdf

<file bytes>
--...--
```

The returned operation reference points to ingestion progress and completion
events. The server removes the private staged upload after successful
ingestion. Failed or interrupted ingestion can reuse the staged file during
the `upload_retry_retention_seconds` window. The server sweeps expired uploads.
Trusted local paths use `POST /api/v1/knowledges/{name}/ingest-path` and are
disabled unless the server explicitly enables them and allowlists their roots.

### Submitting a turn

Create a conversation with `POST /api/v1/conversations`. Send `{}` for a
whole-base search or include a `scope` with the desired selections. Submit a
turn to the conversation ID returned by that request.

```http
POST /api/v1/conversations/<conversation-id>/turns HTTP/1.1
Authorization: Bearer <launch-token>
Content-Type: application/json

{
  "user_query": "What does the design say about thermal protection?",
  "retrieval_mode": "auto"
}
```

The `202 Accepted` response adds `conversation_id` and `turn_id` to the
operation reference. Use the operation's SSE endpoint while the turn is
running. The persisted turn and canonical messages can be fetched later from
their conversation routes.


## HTTP request reference

The following `curl` examples use POSIX shell syntax (`curl.exe` in
PowerShell). Set `RAVEN_URL=http://127.0.0.1:8765` and supply `RAVEN_TOKEN`
through your trusted desktop or host process. Each protected request explicitly
sends `Authorization: Bearer $RAVEN_TOKEN`. For hosted use, also send
`X-Raven-User-ID: 4b9bd24b-61de-42ed-9176-d2db91b66702` on every protected
request when `require_user_id_header` is enabled. The UUIDs below are example
path values; replace them with IDs returned by your server. Requests without
`-d` or `-F` have no body. JSON bodies use `Content-Type: application/json`.
Do not pass the process-wide launch token directly to an untrusted browser.

Each response example below identifies its HTTP status and useful fields.
`202 Accepted` commands return an operation reference, not the finished task
result. Most `202` commands use this complete body:

```json
{
  "operation_id": "cc9dce11-16ca-42be-b418-18588d03a571",
  "task_id": "8f708231-4a27-40cb-a51e-477339c167ad",
  "status": "running",
  "events_url": "/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/events",
  "retry_of_operation_id": null,
  "retry_of_task_id": null
}
```

The initial status may instead be `queued`. A turn submission uses a distinct
`202` body: `conversation_id`, `turn_id`, `operation_id`, `task_id`, `status`,
and `events_url` (without retry-link fields). The `200` and `201` routes below
wait for their work and return the resource. Errors use the common
`{"error":{"code":"...","message":"...","details":{}},"request_id":"..."}`
envelope; see [Errors](#errors). `/openapi.json` supplies the complete typed
response schemas. The examples use valid request formats; response snippets
marked *excerpt* show selected fields rather than the entire object.

### Health and runtime requests

Health and OpenAPI routes are unauthenticated; all runtime routes require the
bearer token. These requests have empty bodies unless `-d` is shown.

| Request | Success | Meaning |
| --- | --- | --- |
| `curl -i "$RAVEN_URL/api/v1/health/live"` | `200 {"status":"ok"}` | HTTP process is live. |
| `curl -i "$RAVEN_URL/api/v1/health/ready"` | `200 {"status":"ready"}` | Registry is ready; otherwise `503`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/runtime"` | `200` excerpt: `{"started":true,"models_loaded":false,"operation_store_healthy":true}` | User-runtime state; full response also has `user_id`, `closed`, `runtime_config_revision`, and `models`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/runtime/settings"` | `200` complete settings document, initially `{"revision":0,...}` | Read current defaults before editing. |
| `curl -i -X PATCH -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"expected_revision":0,"retrieval_top_k":5}' "$RAVEN_URL/api/v1/runtime/settings"` | `200` complete settings document, now `{"revision":1,"retrieval_top_k":5,...}` | Change only specified fields; stale revision gives `409`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"expected_revision":1}' "$RAVEN_URL/api/v1/runtime/settings/reset"` | `200` complete defaults, now `{"revision":2,"retrieval_top_k":3,...}` | Restore all runtime defaults. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/runtime/discovery-issues"` | `200 {"items":[]}` when no malformed resource was found | Each nonempty item has `resource_type`, `directory_name`, `code`, and `message`. |

The complete mutable field list and request semantics are in
[RuntimeConfig](#runtimeconfig). `operation_store_healthy: false` needs operator
attention before submitting further work. Invalid settings are `422`, not
background tasks.

### Model requests

Configured-model routes affect this user's `Raven` runtime. Ollama
administration affects the externally managed Ollama service and should be
restricted by a hosted application's authorization layer.

| Request | Success | Meaning |
| --- | --- | --- |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/provider/models/configured"` | `200 {"llm":null,"embedding":null}` before configuration | After configuration, each role has `provider`, `model`, and `role`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"llm":{"provider":"ollama","model":"qwen3:8b","role":"llm","api_key_ref":null,"options":{}},"embedding":{"provider":"ollama","model":"bge-m3","role":"embedding","api_key_ref":null,"options":{}}}' "$RAVEN_URL/api/v1/provider/models/configure"` | `202` operation reference | Validate and install one LLM/embedding pair. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/providers/ollama/status"` | `200 {"provider":"ollama","status":"available"}` | Check Ollama connection; unavailability gives `503`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/providers/ollama/models"` | `200 {"items":[]}` if no models are installed | Items can include `model`, `modified_at`, `digest`, `size`, and `details`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"model":"qwen3:8b"}' "$RAVEN_URL/api/v1/providers/ollama/models/inspect"` | `200` inspection object containing `model`, `template`, `modelfile`, `license`, `details`, `model_info`, `parameters`, and `capabilities` | Inspect one model. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"model":"qwen3:8b"}' "$RAVEN_URL/api/v1/providers/ollama/models/pull"` | `202` operation reference | Follow `model.pull.progress` events. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"model":"qwen3:8b"}' "$RAVEN_URL/api/v1/providers/ollama/models/delete"` | `202` operation reference | Permanently remove an installed model. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"keep_alive":"10m"}' "$RAVEN_URL/api/v1/provider/models/configured/llm/preload"` | `202` operation reference | Preload the configured role. Omit body to use Ollama's default; `0` is invalid. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/provider/models/configured/embedding/unload"` | `202` operation reference | Unload the configured role; no body. |

For preload/unload, `{role}` is `llm` or `embedding`; substitute either in
both examples. Cloud adapters have no Ollama VRAM residency. Model names for
inspect/pull/delete go in the JSON body, not a path segment. See
[Model providers](#model-providers) for role, API-key, and switching rules.

### Operation and SSE requests

Use the `operation_id` and `task_id` returned by a `202` command. In these
examples they are `cc9dce11-16ca-42be-b418-18588d03a571` and
`8f708231-4a27-40cb-a51e-477339c167ad`. The default list page size is the
`operation_page_size` system setting (50); `limit` must be positive and cannot
exceed it. Pass a non-null `next_cursor` back in the corresponding `after_*`
query parameter. List status filters accept the documented operation states.

| Request | Success | Meaning |
| --- | --- | --- |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operations?status=failed&limit=10"` | `200 {"items":[],"next_cursor":null}` if none failed | Newest-first operation page; optional `after_operation_id=<uuid>`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operation-tasks/retryable?limit=10"` | `200 {"items":[],"next_cursor":null}` if none eligible | Retryable task page; optional `after_task_id=<uuid>`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571"` | `200` operation object with `operation_id`, `name`, `status`, `last_event_id`, times, `error`, and `is_finished` | Inspect terminal state independently of SSE. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/tasks?status=failed&limit=10"` | `200 {"items":[],"next_cursor":null}` if none failed | Tasks within one operation; optional `after_task_id=<uuid>`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/tasks/8f708231-4a27-40cb-a51e-477339c167ad"` | `200` task object with `status`, `retry_policy`, `attempt`, `max_attempts`, `attempts_remaining`, `can_retry`, retry links, times, and error | Check a task's eligibility before retry. Retry inputs are never exposed. |
| `curl -i -N -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Last-Event-ID: 5' "$RAVEN_URL/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/events"` | `200 text/event-stream`, frames such as `id: 6`, `event: chat.response_delta`, `data: {...}` | Replay after event 5, then follow live updates. Omit `Last-Event-ID` to start at the beginning. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/cancel"` | `200` operation object; status may still be transitioning | Request cooperative cancellation; no request body. Recheck operation/task status. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/operations/cc9dce11-16ca-42be-b418-18588d03a571/tasks/8f708231-4a27-40cb-a51e-477339c167ad/retry"` | `202` new operation reference with `retry_of_operation_id` and `retry_of_task_id` populated | User-confirmed retry; no request body. The original operation is unchanged. |

An SSE frame has `id`, typed `event`, and a JSON `data` envelope with
`event_id`, `operation_id`, `task_id`, `task_name`, `type`, ISO timestamp,
payload `data`, and `is_final`. Heartbeats are SSE comments, not JSON events.
The stream closes after a final event. An invalid cursor gives `400`. A cursor
ahead of the available event history gives `410`. A cleaned-up operation gives
`404`. See
[Replay after disconnection](#replay-after-disconnection) and
[Streaming operation events](#streaming-operation-events) for the client flow.

### Knowledge, ingestion, and navigation requests

The `name` path value below is `engineering`. File deletion uses a `file_id`;
section navigation uses the stored `file_name`, which must be URL-encoded if
it contains spaces or reserved characters. List endpoints return
`{"items":[...],"next_cursor":null}` when the final page is reached.
Non-null cursors belong in the next request's matching `after_*` parameter.
`limit` defaults to `operation_page_size` and may not exceed it.

| Request | Success | Meaning |
| --- | --- | --- |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges?limit=10"` | `200 {"items":[],"next_cursor":null}` if none exist | Knowledge summaries; optional `after_name=engineering`. Items include `name`, `user_summary`, `count`, and `created_at`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"name":"engineering","user_summary":"Engineering reference documents"}' "$RAVEN_URL/api/v1/knowledges"` | `201` metadata with `schema_version`, `name`, `created_at`, and `user_summary` | Create; duplicate name gives `409`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering"` | `200` same knowledge metadata object | Inspect an existing knowledge. |
| `curl -i -X PATCH -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"user_summary":"Updated engineering documents"}' "$RAVEN_URL/api/v1/knowledges/engineering"` | `200` updated metadata object | Replace its summary; an empty string is allowed. |
| `curl -i -X DELETE -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering"` | `202` operation reference | Delete the knowledge and its files/vectors. Irreversible without backup. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering/files?limit=10"` | `200 {"items":[],"next_cursor":null}` if empty | Files; optional `after_file_id=<file-id>`. Each item has `file_id`, `file_name`, counts, `ingested_at`, and `navigation_type`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -F 'file=@system-design.pdf;type=application/pdf' "$RAVEN_URL/api/v1/knowledges/engineering/files"` | `202` operation reference | Browser-safe upload. The required multipart field is **`file`**; do not manually set a boundary. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"source_path":"/srv/raven-imports/system-design.pdf"}' "$RAVEN_URL/api/v1/knowledges/engineering/ingest-path"` | `202` operation reference | Trusted server-local path only; disabled by default and limited to allowed roots. No upload bytes in this request. |
| `curl -i -X DELETE -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering/files/a83fd91c20b4"` | `202` operation reference | Delete by **file ID**, not filename. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering/files/system-design.pdf/sections?limit=10&after_section_index=0"` | `200 {"items":[],"next_cursor":null}` if none | Sections in increasing index order. Use returned numeric cursor in `after_section_index`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering/files/system-design.pdf/sections/a83fd91c20b4-3"` | `200` full section with `section_id`, `file_id`, `file_name`, `section_index`, `raw_content`, `summary`, `keywords`, `conditions`, `definitions`, `source_element_ids`, and `source_range` | Inspect exact text and provenance; wrong file/section pairing gives `404`. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/knowledges/engineering/stats"` | `200 {"vector_count":42,"file_count":1}` | Count stored files and vector points. |

Accepted filenames are simple `.txt`, `.md`, `.docx`, or `.pdf` names; paths,
control characters, reserved device names, and unsupported extensions are
rejected. The configured source-file and request-size limits also apply. A
successful upload is removed from staging after ingestion. A failed upload
remains private and retryable only within `upload_retry_retention_seconds`;
after expiry, upload again. Duplicate file ingestion gives a conflict rather
than silently replacing data. A failed/interrupted ingestion may be retried
through its task if `can_retry` is true; RAVEN reconciles pending vector and
metadata state. See [Ingestion safety and retry](#ingestion-safety-and-retry).

### Conversation, turn, and preference requests

The examples use conversation ID `c36c45ac5575`, turn ID
`c4d242e9-f6e9-4a84-8cdc-e7bf5c8a3021`, and preference ID
`562fe22a-8eea-4f24-bb81-eeb0f663fd4d`. Use the IDs returned by your own
server. The scope is selected at creation. The server creates a temporary
session when a turn or manual action is submitted.

| Request | Success | Meaning |
| --- | --- | --- |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations?limit=10"` | `200 {"items":[],"next_cursor":null}` if none exist | Newest-first metadata; optional `after_conversation_id=<id>`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"scope":{"selections":[{"knowledge_name":"engineering"}]}}' "$RAVEN_URL/api/v1/conversations"` | `201` metadata with `schema_version`, `conversation_id`, `scope`, `title`, `is_titled`, `pinned`, and `created_at` | Search all files in `engineering`. Use `{}` for the whole knowledge base. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575"` | `200` full conversation metadata | Reload a conversation. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/scope"` | `200 {"selections":[{"knowledge_name":"engineering"}]}` | Read the saved `RetrievalScope`. File selections preserve their supplied `file_id` and `file_name` fields. |
| `curl -i -X PATCH -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"title":"Thermal design","pinned":true}' "$RAVEN_URL/api/v1/conversations/c36c45ac5575"` | `200` updated metadata | Supply `title`, `pinned`, or both; scope cannot be changed. |
| `curl -i -X PATCH -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"title":"Thermal design"}' "$RAVEN_URL/api/v1/conversations/c36c45ac5575/title"` | `200` updated metadata | Set a title and mark the conversation titled. |
| `curl -i -X PATCH -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"pinned":false}' "$RAVEN_URL/api/v1/conversations/c36c45ac5575/pinned"` | `200` updated metadata | Send `true` to pin or `false` to unpin. |
| `curl -i -X DELETE -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575"` | `202` operation reference | Delete conversation data; back up first if needed. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/messages?limit=10&after_message_id=0"` | `200 {"items":[],"next_cursor":null}` if empty | Canonical, uncompacted transcript. Messages include role, content, turn/order, tool calls, and tool-call identifiers. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/turns/c4d242e9-f6e9-4a84-8cdc-e7bf5c8a3021"` | `200` turn with `turn_id`, `operation_id`, `user_query`, `result`, `memory_indexed`, `committed_at` | Load a durable turn. Result includes thinking, response, evidence, tool calls, and reconstructed sources. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/turns/c4d242e9-f6e9-4a84-8cdc-e7bf5c8a3021/messages?limit=10&after_message_id=0"` | `200 {"items":[],"next_cursor":null}` if none | Canonical messages for only this turn, paginated by message ID. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"user_query":"What protects the motor from overheating?","retrieval_mode":"auto"}' "$RAVEN_URL/api/v1/conversations/c36c45ac5575/turns"` | `202` reference plus `conversation_id` and new `turn_id` | Start generation. `retrieval_mode` can be omitted, `auto`, or an allowed exact mode. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/turns/c4d242e9-f6e9-4a84-8cdc-e7bf5c8a3021/reconstruction"` | `202` operation reference | Reconstruct persisted turn evidence; no body. A casual turn may return an empty result. |
| `curl -i -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/preferences"` | `200 {"items":[]}` if none | Items contain exact `preference_id` and `text`. |
| `curl -i -X POST -H "Authorization: Bearer $RAVEN_TOKEN" -H 'Content-Type: application/json' -d '{"text":"Prefer concise explanations."}' "$RAVEN_URL/api/v1/conversations/c36c45ac5575/preferences"` | `201` preference with `preference_id` and `text` | Save conversation-scoped preference. |
| `curl -i -X DELETE -H "Authorization: Bearer $RAVEN_TOKEN" "$RAVEN_URL/api/v1/conversations/c36c45ac5575/preferences/562fe22a-8eea-4f24-bb81-eeb0f663fd4d"` | `200` removed preference object | Delete by stable ID, not matching text; no body. |

An accepted turn's `events_url` is the UI's live source of thinking, tool
calls/results, reconstruction, and answer deltas. Fetch the turn and messages
after completion for durable display. Two simultaneous turns on the same
conversation are rejected with `409`; different conversations may run at
once. See [Conversations, sessions, and memory](#conversations-sessions-and-memory).


### Standalone retrieval and reconstruction requests

Use these routes to search or reconstruct without a conversation. Every
retrieval body requires a non-empty `query` and a `scope`. Use
`"scope":{"selections":[]}` to search the whole knowledge base. Each route
accepts only its own strategy options, and omitted options use the runtime
defaults.

| Route | Example JSON body |
| --- | --- |
| `POST /api/v1/retrieval/embedded` | `{"query":"What is the safe-hold deadline?","scope":{"selections":[{"knowledge_name":"engineering"}]},"top_k":3}` |
| `POST /api/v1/retrieval/hierarchical` | `{"query":"What led to safe-hold?","scope":{"selections":[]},"top_k_section":3,"top_k_knowledge":2,"full_retrieval":false,"max_per_read":50,"anchor_count":4}` |
| `POST /api/v1/retrieval/agreement` | `{"query":"What is the safe-hold deadline?","scope":{"selections":[{"knowledge_name":"engineering","file_id":"a83fd91c20b4"}]},"top_k":3,"top_k_knowledge":2,"full_retrieval":false,"max_per_read":50,"anchor_count":4}` |
| `POST /api/v1/retrieval/vector-conditioned` | `{"query":"How do the fault response and power systems interact?","scope":{"selections":[]},"top_k_files":5,"top_k_sections":3,"max_per_read":50,"anchor_count":4}` |
| `POST /api/v1/reconstruction` | `{"sections":[{"knowledge_name":"engineering","file_name":"system-design.pdf","section_id":"a83fd91c20b4-3"}]}` |

Use real file and section IDs from your catalog. Each command returns `202`
with an `events_url`. On that stream, standalone retrieval emits
`retrieval.result` with the strategy's native output in `data.result`.
Reconstruction emits one `reconstruction.file` event per file and can run
without configured models. Use the section references from a retrieval result
to build the `sections` list. For an agreement result, choose references from
the appropriate branch or combine both branches. Wait for
`operation.completed` before treating either command as successful.
Invalid request bodies return `422`. If an accepted search cannot resolve its
scope, inspect the failed operation's events for the error.

### Manual activity requests

Prefix each path below with `/api/v1/conversations/{conversation_id}/activities`.
Every POST records a successful action. The activity-history GET reads the
conversation directly without starting a session or recording an activity.
The catalog GET routes under `/api/v1/knowledges` also browse without recording
activity.

| Request after `/activities` | Body or query parameters | Output |
| --- | --- | --- |
| `POST /knowledges` | No body | All knowledges for the current user. |
| `POST /knowledges/engineering/files` | No body | Files in `engineering`. |
| `POST /knowledges/engineering/files/a83fd91c20b4/sections?limit=10&after_section_index=0&get_content=false` | No body. All query parameters are optional. `limit` defaults to `20` and accepts `1`–`50`. `after_section_index` defaults to `0`. `get_content` defaults to `false`. | A page of section records. Use the last `section_index` to request the next page. |
| `POST /knowledges/engineering/files/a83fd91c20b4/open` | No body. Optional `origin_activity_id` and `origin_turn_id` query parameters accept UUIDs. | The file's stored metadata. |
| `POST /knowledges/engineering/sections/a83fd91c20b4-3/open` | No body. The same optional origin IDs apply. | One section record, including its raw text. |
| `POST /retrieval` | `{"query":"What is the safe-hold deadline?","retrieval_mode":"embedded"}` | Results in the conversation's saved scope. Choose `embedded`, `hierarchical`, `agreement`, or `vector_conditioned`. |
| `POST /reconstruction` | `{"sections":[{"knowledge_name":"engineering","file_name":"system-design.pdf","section_id":"a83fd91c20b4-3"}]}` | Reconstructed files with matching sections highlighted. |
| `GET ?limit=10&after_sequence=0` | No body. `limit` defaults to `50` and accepts `1`–`199`. `after_sequence` defaults to `0`. Optional `turn_id` filters by the committed turn UUID. | `200` with `items` and `next_cursor`. Use `next_cursor` as the next `after_sequence` and keep the same `turn_id` filter. A `null` cursor marks the last page. |
| `DELETE /855a028b-f40b-40e2-b220-0b460c23adec` | No body. Substitute the activity UUID. | `200 {"removed":true}` when removed, or `{"removed":false}` when absent. A captured or committed activity returns `409`. |

The POST routes return `202` with an `events_url`. Read the activity UUID from
`conversation.activity.recorded`. For browsing, source opens, and
retrieval, that event also carries the action output in `data.result`.
Reconstruction emits its files in `reconstruction.file` events. The retrieval
route uses current runtime settings. The scope applies to retrieval, and
navigation and reconstruction can access any source in the user's knowledge
base. Only retrieval requires configured models.


## Configuration

RAVEN separates deployment policy, user-scoped paths, and mutable operation
defaults into three configuration types.

### `SystemConfig`

`SystemConfig` is immutable for the lifetime of a running server. It controls
host, port, logging, allowed origins and hosts, request and upload limits,
SSE heartbeat timing, concurrency limits, cache bounds,
trusted-path ingestion, and deployment-wide safety ceilings.
It also stores operation-cleanup policy values that an application can use
when scheduling `OperationCleanupService`; those values do not enable
automatic operation cleanup on their own.

The server uses one shared `SystemConfig` for every user runtime. It is not a
per-user preference object and cannot be changed through the HTTP API.

#### Server loading and changes

Starting the server with only a home directory:

```bash
nraven serve --home /path/to/raven-home
```

loads this file:

```text
<raven-home>/system_settings/system_settings.json
```

If the file does not exist, RAVEN creates it with all default values. To change
system settings, stop the server, edit the file, and restart it. The file is
validated at startup and is not hot-reloaded.

An existing settings file elsewhere can be selected explicitly:

```bash
nraven serve \
  --home /path/to/raven-home \
  --system-settings /path/to/production-system-settings.json
```

An explicit file must already exist. RAVEN reads it in place and does not copy
it into the home directory.

The default generated file has this complete shape:

```json
{
  "schema_version": 1,
  "host": "127.0.0.1",
  "port": 8765,
  "log_level": "info",
  "operation_sync_interval_seconds": 1.0,
  "operation_retention_seconds": 86400.0,
  "upload_retry_retention_seconds": 86400.0,
  "operation_cleanup_interval_seconds": 300.0,
  "operation_cleanup_batch_size": 100,
  "event_replay_page_size": 256,
  "operation_page_size": 50,
  "finished_operation_cache_size": 256,
  "open_knowledge_limit": 16,
  "open_conversation_limit": 64,
  "llm_adapter_cache_size": 8,
  "embedding_adapter_cache_size": 8,
  "prompt_cache_size": 32,
  "sse_heartbeat_interval_seconds": 30.0,
  "max_source_file_bytes": 104857600,
  "max_upload_request_overhead_bytes": 65536,
  "max_request_body_bytes": 1048576,
  "max_query_string_bytes": 8192,
  "max_active_operations_per_user": 8,
  "max_active_operations": 64,
  "max_document_pages": 1000,
  "max_retrieval_top_k": 25,
  "max_agent_iterations": 50,
  "runtime_idle_seconds": 1800.0,
  "max_user_runtimes": 100,
  "cors_origins": ["http://localhost:3000"],
  "allowed_hosts": ["localhost", "127.0.0.1", "[::1]"],
  "require_user_id_header": false,
  "trusted_ingestion_enabled": false,
  "allowed_ingestion_roots": []
}
```

Unknown fields, invalid types, non-finite numbers, and unsupported
`schema_version` values cause startup to fail with a configuration error.

#### Server and request boundary fields

| Field | Default | Accepted value | Purpose |
| --- | ---: | --- | --- |
| `schema_version` | `1` | Exactly the supported schema version | Identifies the system-settings file format. It should not be changed manually. |
| `host` | `"127.0.0.1"` | Non-empty string | Address passed to Uvicorn. Keep the loopback default for desktop use; hosted deployments may bind another interface deliberately. |
| `port` | `8765` | Integer from `1` through `65535` | TCP port used by the FastAPI server. |
| `log_level` | `"info"` | `"critical"`, `"error"`, `"warning"`, `"info"`, or `"debug"` | Uvicorn logging level. |
| `sse_heartbeat_interval_seconds` | `30.0` | Finite number greater than zero | Maximum idle interval before the operation SSE endpoint sends a heartbeat comment. |
| `max_source_file_bytes` | `104857600` (100 MiB) | Positive integer | Maximum accepted source-document size. |
| `max_upload_request_overhead_bytes` | `65536` (64 KiB) | Positive integer | Maximum multipart framing and metadata overhead allowed beyond the uploaded file bytes. |
| `max_request_body_bytes` | `1048576` (1 MiB) | Positive integer | Maximum ordinary non-file request-body size. |
| `max_query_string_bytes` | `8192` (8 KiB) | Positive integer | Maximum encoded query-string size. |
| `cors_origins` | `["http://localhost:3000"]` | List of complete HTTP or HTTPS origins | Browser origins allowed by CORS. Entries may contain a scheme, host, and optional port, but no path, query, or fragment. |
| `allowed_hosts` | `["localhost", "127.0.0.1", "[::1]"]` | Non-empty list of hostnames or IP literals without ports | Host-header allowlist checked before request processing. Add an internal hostname explicitly before using it in hosted deployment. |
| `require_user_id_header` | `false` | Boolean | When true, protected requests must include a valid `X-Raven-User-ID`; when false, a missing header selects the default desktop user. |

#### Operations and event fields

| Field | Default | Accepted value | Purpose |
| --- | ---: | --- | --- |
| `operation_sync_interval_seconds` | `1.0` | Finite number greater than zero | Interval used by the operation manager's periodic durable-store synchronization service. Lifecycle checkpoints are still written immediately where required. |
| `operation_retention_seconds` | `86400.0` (24 hours) | Finite number greater than or equal to zero | Retention policy value for an application-created `OperationCleanupService`. Pass it as a `timedelta` to the service. Zero makes terminal history immediately eligible; passing the deadline alone does not delete it. |
| `operation_cleanup_interval_seconds` | `300.0` (5 minutes) | Finite number greater than zero | Interval an application can use to schedule operation cleanup. RAVEN does not schedule operation cleanup itself. This value also bounds the server's maintenance-loop interval for runtime eviction and upload cleanup. |
| `operation_cleanup_batch_size` | `100` | Positive integer | Batch-size policy value to pass to an application-created `OperationCleanupService`; limits the expired operations considered per pass. |
| `event_replay_page_size` | `256` | Positive integer | Number of persisted events loaded per page while replaying an operation stream. |
| `operation_page_size` | `50` | Positive integer | Default and maximum page size exposed by operation and resource-list HTTP endpoints. |
| `finished_operation_cache_size` | `256` | Non-negative integer | Maximum completed `Operation` objects retained in the in-memory LRU cache. Zero disables that completed-operation cache. Durable records remain in SQLite until retention cleanup. |
| `max_active_operations_per_user` | `8` | Positive integer no greater than `max_active_operations` | Maximum accepted non-terminal operation tasks for one user. |
| `max_active_operations` | `64` | Positive integer | Maximum accepted non-terminal operation tasks across the server. |

#### Resource and cache fields

| Field | Default | Accepted value | Purpose |
| --- | ---: | --- | --- |
| `open_knowledge_limit` | `16` | Positive integer | Maximum idle/open knowledge resources retained in one user's LRU registry before eligible resources are closed and evicted. |
| `open_conversation_limit` | `64` | Positive integer | Maximum idle/open conversation resources retained in one user's LRU registry. Active resources remain protected. |
| `llm_adapter_cache_size` | `8` | Positive integer | Maximum cached LLM adapters in the provider layer. |
| `embedding_adapter_cache_size` | `8` | Positive integer | Maximum cached embedding adapters in the provider layer. |
| `prompt_cache_size` | `32` | Positive integer | Maximum cached conversation-specific agent prompts. Preference changes invalidate the affected prompt. |
| `runtime_idle_seconds` | `1800.0` (30 minutes) | Finite number greater than zero | Time an unleased user runtime may remain idle before server eviction. Active tasks retain their runtime. |
| `max_user_runtimes` | `100` | Positive integer | Maximum user runtimes that may be resident or initializing simultaneously. Idle runtimes are evicted when possible before capacity is rejected. |

#### Ingestion and runtime ceiling fields

| Field | Default | Accepted value | Purpose |
| --- | ---: | --- | --- |
| `upload_retry_retention_seconds` | `86400.0` (24 hours) | Finite number greater than zero | Time a failed private staged browser upload remains available for user-confirmed ingestion retry. Successful uploads are removed after ingestion. |
| `max_document_pages` | `1000` | Positive integer | Maximum number of pages accepted from a page-based document parser. |
| `max_retrieval_top_k` | `25` | Positive integer | Server-wide ceiling for the three runtime retrieval counts and per-call `top_k` overrides. |
| `max_agent_iterations` | `50` | Positive integer | Server-wide ceiling for `RuntimeConfig.agent_max_iterations` and per-session overrides. |
| `trusted_ingestion_enabled` | `false` | Boolean | Enables the trusted local-path ingestion route for controlled host environments. Browser clients should normally use multipart upload. |
| `allowed_ingestion_roots` | `[]` | List of filesystem paths | Roots beneath which trusted local-path ingestion is permitted. Paths are expanded and resolved. At least one root is required when trusted ingestion is enabled. |

`max_active_operations_per_user` cannot exceed `max_active_operations`.
Trusted-path ingestion cannot be enabled with an empty
`allowed_ingestion_roots` list.

#### Library usage

Library applications can construct and pass the immutable configuration
directly:

```python
from nraven import Raven, SystemConfig


config = SystemConfig(
    max_active_operations_per_user=4,
    max_active_operations=32,
    max_source_file_bytes=50 * 1024 * 1024,
)
raven = Raven("./raven-home", system_config=config)
```

When `system_config` is omitted, `Raven` uses an in-memory `SystemConfig()` with
the defaults above. Unlike the server CLI, the `Raven` constructor does not
automatically load or create `system_settings.json`.

Library applications that want file-backed system settings can manage them
explicitly:

```python
from nraven import Raven, SystemConfig


config = SystemConfig(
    max_active_operations_per_user=4,
    max_active_operations=32,
)
config.save("./system_settings.json")

loaded_config = SystemConfig.load("./system_settings.json")
raven = Raven("./raven-home", system_config=loaded_config)
```

`SystemConfig.save()` uses an atomic temporary-file replacement and requests a
filesystem synchronization before returning. Because the dataclass is frozen,
changing a setting means creating or loading a new `SystemConfig`, then
constructing a new `Raven` or restarting the server with it.

### `PathConfig`

`PathConfig` is immutable for one RAVEN/user runtime. It validates the user ID
as a UUID and derives all user-specific storage paths from `raven_home` and
`user_id`:

```text
<raven-home>/
├── system_settings/
│   └── system_settings.json
├── temp/
│   └── server_auth/
│       ├── server.lock
│       └── bearer_token
└── <user-uuid>/
    ├── data/
    │   ├── knowledge_base/
    │   └── conversations/
    ├── operations/
    │   └── operations.sqlite3
    ├── runtime_settings/
    │   └── runtime_settings.json
    └── temp/
        └── uploads/
```

The server lazily creates one `Raven` runtime per validated UUID and evicts
idle runtimes according to `SystemConfig`. Active operation tasks retain their
runtime until they reach a terminal state.

### `RuntimeConfig`

`RuntimeConfig` stores each user's adjustable defaults in
`runtime_settings.json`. An individual configuration snapshot is immutable.

The complete settings document contains these fields:

| Field | Default | Accepted value | Purpose |
| --- | ---: | --- | --- |
| `schema_version` | `1` | Managed by RAVEN | Identifies the persisted runtime-settings schema. It is returned to callers but cannot be updated directly. |
| `revision` | `0` initially | Managed by RAVEN | Increases after every successful update or reset and provides optimistic concurrency control. It cannot be updated directly. |
| `breakpoint_percentile_threshold` | `95` | Integer from `1` through `100` | Percentile used to identify semantic distances that become section boundaries. Lower values generally produce more boundaries. |
| `buffer_size` | `1` | Positive integer | Number of neighboring semantic units included on each side when constructing embedding windows for boundary detection. |
| `max_extraction_retries` | `3` | Positive integer | Maximum attempts for LLM-based structured section-metadata extraction during ingestion. |
| `chunk_size` | `512` | Positive integer | Target chunk size supplied to LlamaIndex's sentence-aware splitter before embedding and vector storage. |
| `chunk_overlap` | `50` | Non-negative integer smaller than `chunk_size` | Overlap between adjacent embedding chunks. |
| `retrieval_top_k` | `3` | Positive integer up to `SystemConfig.max_retrieval_top_k` (`25` by default) | Maximum returned sections. Agreement uses it for each branch. |
| `retrieval_top_k_knowledge` | `2` | Same bounds as `retrieval_top_k` | Maximum shortlisted knowledges in hierarchical or agreement retrieval when whole-base knowledge shortlisting runs. |
| `retrieval_top_k_files` | `5` | Same bounds as `retrieval_top_k` | Maximum files shortlisted by vector-conditioned retrieval. |
| `full_retrieval` | `false` | Boolean | Skip knowledge-summary shortlisting in hierarchical and agreement retrieval. Explicit scopes already skip it. |
| `retrieval_max_per_read` | `50` | Positive integer | Maximum candidate records per LLM scoring batch. |
| `retrieval_anchor_count` | `4` | Non-negative integer | Maximum previously ranked candidates supplied as scoring anchors to the next batch. `0` disables anchors. |
| `agent_max_iterations` | `10` | Positive integer no greater than `SystemConfig.max_agent_iterations` (`50` by default) | Maximum number of agent workflow iterations allowed for one turn. |
| `memory_token_limit` | `4000` | Positive integer | Token budget for the model-facing compacted conversation context. |
| `memory_top_k` | `5` | Positive integer | Maximum matched memory nodes in conversation-memory search. Currently each node represents a turn and may return both user and assistant messages, so this is not a strict message-count limit. |

Ingestion and direct retrieval resolve omitted options when their Python
methods run. For standalone HTTP retrieval, the background worker calls the
Python retrieval method after submission. A settings change before that call
can affect the search.

Each new agent turn reads the current retrieval defaults once and uses them for
all its retrieval tool calls. Manual session retrieval reads them for each
search. A retried turn reuses its saved settings snapshot. Session constructor
defaults such as memory and iteration limits are set when the session is
created. Explicit per-call arguments can override defaults where the method
exposes them.

#### Python API

Update it atomically through the Python facade:

```python
current = raven.get_runtime_settings()

task = await raven.update_runtime_settings(
    {
        "retrieval_top_k": 5,
        "agent_max_iterations": 12,
    },
    expected_revision=current["revision"],
)
updated = await task.result()
```

Each successful change creates a new revision. Supplying
`expected_revision` prevents one caller from silently overwriting a concurrent
update. `Raven.reset_runtime_settings()` restores built-in defaults as another
revision.

#### HTTP API

The same revision-safe contract is available over HTTP:

```http
GET   /api/v1/runtime/settings
PATCH /api/v1/runtime/settings
POST  /api/v1/runtime/settings/reset
```

All three endpoints require the normal bearer authorization header. In hosted
mode they also use `X-Raven-User-ID`, so each user reads and changes only their
own runtime settings.

##### Read settings

```http
GET /api/v1/runtime/settings HTTP/1.1
Authorization: Bearer <launch-token>
X-Raven-User-ID: <user-uuid>
```

The GET endpoint has no request body. `X-Raven-User-ID` is required only when
the server has `require_user_id_header=true`.

Example response:

```json
{
  "schema_version": 1,
  "revision": 0,
  "breakpoint_percentile_threshold": 95,
  "buffer_size": 1,
  "max_extraction_retries": 3,
  "chunk_size": 512,
  "chunk_overlap": 50,
  "retrieval_top_k": 3,
  "retrieval_top_k_knowledge": 2,
  "retrieval_top_k_files": 5,
  "full_retrieval": false,
  "retrieval_max_per_read": 50,
  "retrieval_anchor_count": 4,
  "agent_max_iterations": 10,
  "memory_token_limit": 4000,
  "memory_top_k": 5
}
```

##### Update settings

```http
PATCH /api/v1/runtime/settings HTTP/1.1
Authorization: Bearer <launch-token>
Content-Type: application/json
```

The PATCH body requires `expected_revision` and at least one configurable
field. Include only the fields that should change:

```json
{
  "expected_revision": 0,
  "retrieval_top_k": 5,
  "agent_max_iterations": 12
}
```

The complete request-body shape is shown below. Apart from
`expected_revision`, every field is optional:

```json
{
  "expected_revision": 0,
  "breakpoint_percentile_threshold": 90,
  "buffer_size": 2,
  "max_extraction_retries": 4,
  "chunk_size": 768,
  "chunk_overlap": 75,
  "retrieval_top_k": 5,
  "retrieval_top_k_knowledge": 3,
  "retrieval_top_k_files": 6,
  "full_retrieval": false,
  "retrieval_max_per_read": 40,
  "retrieval_anchor_count": 3,
  "agent_max_iterations": 12,
  "memory_token_limit": 8000,
  "memory_top_k": 8
}
```

Every configurable field is optional independently, so these are also valid
partial updates:

```json
{
  "expected_revision": 4,
  "chunk_size": 768,
  "chunk_overlap": 75
}
```

```json
{
  "expected_revision": 5,
  "memory_token_limit": 8000
}
```

Do not include `schema_version` or `revision` in the body, and omit fields that
should remain unchanged. Unknown fields and an update containing no setting
are rejected with `422 Unprocessable Entity`. Setting values must be JSON
integers rather than strings or booleans. RAVEN validates the merged result, so
`chunk_overlap` must remain smaller than `chunk_size`, and retrieval or agent
limits must remain below their `SystemConfig` ceilings.

On success, PATCH returns the complete settings document with `revision`
incremented by one. If the current revision differs from
`expected_revision`, RAVEN returns `409 Conflict`; read the current settings
again before deciding whether to submit a new update.

##### Reset settings

```http
POST /api/v1/runtime/settings/reset HTTP/1.1
Authorization: Bearer <launch-token>
Content-Type: application/json

{
  "expected_revision": 6
}
```

Reset accepts no setting fields. It restores every configurable field to the
defaults listed above, increments the current revision, persists the result,
and returns the complete updated settings document. It never changes the
revision back to zero. A stale `expected_revision` returns `409 Conflict` just
as it does for PATCH.


## Persistence and recovery

RAVEN stores each user's state under:

```text
<raven-home>/<user-uuid>/
├── data/
│   ├── knowledge_base/
│   │   └── <knowledge-name>/
│   │       ├── metadata.json
│   │       ├── file.sqlite3
│   │       └── qdrant/
│   └── conversations/
│       └── <conversation-id>/
│           ├── metadata.json
│           ├── messages.sqlite3
│           ├── activities.sqlite3
│           ├── preferences.json
│           └── qdrant/
├── operations/
│   └── operations.sqlite3
├── runtime_settings/
│   └── runtime_settings.json
└── temp/
    └── uploads/
```

This persists:

- knowledge metadata, parsed source sections, and Qdrant vectors;
- conversation metadata, original messages, tool messages, turns, compacted
  context, vector memory, preferences, and manual activity records;
- operation records, task records, statuses, retry links, and events;
- per-user runtime settings.

### Recovery after interruption

On startup, RAVEN inspects persisted operations. If a previous process stopped
while an operation was non-terminal, the worker no longer exists, so RAVEN
marks the operation as interrupted instead of pretending that it completed or
leaving it permanently running.

Eligible failed tasks remain discoverable for explicit retry. A retry starts a
new linked operation and uses the task's persisted retry input; RAVEN does not
blindly resume arbitrary Python workers or automatically repeat external model
calls.

Ingestion uses pending records and reconciliation to remove or complete
partially committed storage changes. Conversation turn IDs and idempotent turn
commits prevent the same completed turn from being written twice during retry
or recovery.

If the process stops after committing a turn and before updating its activity
links, opening the conversation repairs those links. RAVEN reads the activity
IDs from the original user message in the `messages` table of
`messages.sqlite3` and updates the corresponding rows in `activities.sqlite3`.
The model's compacted context is stored separately in `context_messages`.

Model adapters, active network clients, and Ollama VRAM residency are runtime
state. They are rebuilt or reconfigured after process restart rather than being
treated as durable application state.


## Upgrading to v0.2.1

- Use `RetrievalScope` when creating conversations and making direct searches.
  A whole-knowledge selection uses only `knowledge_name`. An empty selection
  list searches the entire user's knowledge base.
- Replace the old generic `Raven.retrieve()` call with the appropriate
  strategy-specific method. Use strategy-only values such as `embedded` for
  `retrieval_mode`.
- For HTTP conversation creation, send `{}` or a `scope` object. Read the saved
  selection at `GET /api/v1/conversations/{conversation_id}/scope`.

Conversation metadata is now schema version `2`. Back up your v0.2.0 home
before switching versions. Version `1` conversations are rejected during
discovery and reported through discovery issues. Create new conversations in
v0.2.1 and keep the backup if you need to open the earlier histories with
v0.2.0.


## Backup, restore, and cleanup

The safest consistent backup is an **offline copy of the complete RAVEN home**:
stop accepting new work, wait for or cancel active operations, stop the one
RAVEN server process, and only then copy the home to protected storage. This
keeps each user's knowledge SQLite files, Qdrant directories, conversation
SQLite files, operations/event SQLite store, metadata, and runtime settings
together. Do not copy only `file.sqlite3` without its corresponding Qdrant
directory. Copy conversation metadata, messages, activities, preferences, and
vector memory together. A live file-by-file copy is not a transactionally consistent
backup across these stores. The externally managed Ollama installation and its
model cache are separate and are **not** included in a RAVEN-home backup.

To restore, stop RAVEN, copy the complete saved home into the intended home
path, ensure the server identity can read/write it, then start one server with
`nraven serve --home <restored-home>`. If you used an explicit
`--system-settings` path outside the home, restore that file separately. Verify
`/api/v1/health/ready`, the user runtime status, knowledge/conversation lists,
and operation discovery issues before opening traffic. The launch bearer token
is per start and must be obtained again; do not reuse a token from a backup.
Provider API keys must be restored through the host's environment/secret
manager. Reconfigure the runtime model pair after restart.

This is a backup/restore procedure for the **same storage schema**. Importing
one user's directory into a different account, combining two homes, or
migrating across incompatible schema versions is not an automatic public API.
Keep a pre-upgrade backup and test new releases on a copy before replacing a
production home. Malformed resources are reported by
`GET /api/v1/runtime/discovery-issues`; do not edit SQLite or Qdrant files
under a live process to repair them.

RAVEN provides the machinery for operation-history cleanup through
`OperationCleanupService`, but neither `Raven` nor the bundled FastAPI server
schedules it automatically. The desktop application's bridge layer or the
host's backend around RAVEN owns the retention policy and cleanup schedule.
Without that integration, persisted operations, tasks, retry inputs, and events
remain on disk even after the configured retention period has passed.

For a started `Raven` instance, an application can perform one cleanup pass:

```python
from datetime import timedelta

from nraven import OperationCleanupService

cleanup = OperationCleanupService(
    raven.operation_manager,
    retention=timedelta(seconds=raven.system_config.operation_retention_seconds),
    batch_size=raven.system_config.operation_cleanup_batch_size,
)
cleanup_result = await cleanup.run_once()
```

To make cleanup automatic, the bridge or host must schedule `run_once()`
periodically, for example using `operation_cleanup_interval_seconds`, and own
that scheduler's startup, shutdown, and error handling. The service uses the
owning runtime's operation manager; the bundled HTTP API does not expose an
operation-cleanup endpoint. Each user has a separate operation store, so cleanup
must be arranged for each user's store.

Each pass considers a bounded batch of terminal operations older than the
supplied retention period and reports deleted, skipped, and failed IDs.
Deleting an operation also removes its tasks, events, and retry inputs, so that
operation can no longer be replayed or retried. Retention expiry alone does not
make its HTTP resource disappear; deletion does.

Upload cleanup is separate and is implemented automatically in the server.
Private staged browser uploads have their own
`upload_retry_retention_seconds` window; successful uploads are removed after
ingestion, while failed sources expire and are cleaned up. After that, the
original task may still appear in history but its upload can no longer be
retried: the client must upload again. Never delete individual live database
files as a substitute for the cleanup service.


## Security and deployment responsibilities

RAVEN provides the security boundary needed between its API and a trusted
desktop application or deployment host. It does not attempt to replace a
public application's identity system.

### What RAVEN enforces

- A random per-launch bearer token protects application routes.
- Bearer-token comparison is constant-time, and credential values are not
  returned in API responses or events.
- Optional `X-Raven-User-ID` context must be a valid UUID.
- Each UUID receives a separate storage root and independently owned runtime.
- Request hosts and browser origins must match explicit `SystemConfig`
  allowlists.
- Request bodies, query strings, source files, multipart overhead, active
  operations, and list pages are bounded.
- Uploaded filenames are validated and staged inside controlled user-specific
  directories.
- Trusted-path ingestion is disabled by default and, when enabled, is limited
  to configured roots.
- Model specifications refer to API keys by environment-variable name;
  applications should never place secret values in `options`.
- Errors use stable codes and sanitized messages, while an `X-Request-ID`
  allows operators to correlate failures with private server logs.

Health and OpenAPI routes do not require the bearer token, but they remain
subject to host validation. All other routes require the launch token. The
default host and CORS configuration is loopback-only and should remain narrow
unless a deployment has a deliberate network boundary.

### What the host application owns

For a desktop product, the native application owns the server process, reads
the local launch token, and decides how its UI reaches RAVEN.

For a hosted product, the host application must:

- authenticate end users;
- authorize each account to one internal user UUID;
- inject that UUID and the private RAVEN bearer token into proxied requests;
- decide which model-administration and destructive routes each user may
  invoke;
- keep provider API keys in its environment or secret manager;
- terminate TLS and configure its reverse proxy safely;
- start, stop, monitor, and resource-limit Ollama when local models are used;
- back up and protect the RAVEN home at the filesystem level.

Do not expose the bearer-token file, RAVEN home, or an unrestricted internal
RAVEN port to untrusted clients. In a hosted deployment, browser requests
should normally go through the host backend rather than carrying RAVEN's
process-wide launch token directly.

Ollama model residency is process-global. Changing or unloading a configured
Ollama model can affect shared host resources, so a multi-user host should
restrict model-administration routes and choose an explicit residency policy.


## Server troubleshooting

Start with the HTTP status and the response `error.code`, then use
`X-Request-ID` to find the corresponding server log entry. For accepted
background work, inspect the operation and task status and their event stream;
the original `202` response is not proof of completion.

| Symptom | Check | Action |
| --- | --- | --- |
| Server will not start | `--home` was omitted, settings JSON is invalid, port is occupied, or another process owns the home lock | Supply a writable explicit home. Validate/edit `system_settings.json` while stopped. Keep one RAVEN worker per home; do not remove a lock while its owner still runs. |
| `/live` works but `/ready` gives `503` | Runtime registry not ready | Wait for startup or inspect startup logs. Do not begin ingestion or chat until ready. |
| `401 authentication_required` | Missing/wrong per-launch bearer token, or required hosted user ID omitted | Obtain the **current** token from the trusted owner and add the required `X-Raven-User-ID`. A restart rotates the token. |
| `400 invalid_user_id` | Hosted header is not a UUID | Fix the host's account-to-UUID mapping; do not accept a browser-supplied UUID as authorization. |
| `403 host_not_allowed` or `origin_not_allowed` | Request host or browser Origin not in immutable allowlist | Add the intended exact host/origin to system settings and restart; do not open a wildcard merely to silence the error. |
| `503 ollama_unavailable` | Ollama 0.34.2 is not running at configured `OLLAMA_HOST` | Start/check the separately managed Ollama service, its address, and firewall; then retry the provider check. |
| Cloud model fails | Provider key reference absent or key unavailable to the RAVEN process | Ensure the referenced environment variable is present **before** server startup; do not put the secret value in `ModelSpec.options`. Check provider quotas/capabilities separately. |
| `409` during ingestion | A file with the same identity/name is already present or active | Inspect `/files`; delete the existing file deliberately before re-ingesting, or wait for active ingestion to finish. |
| Upload gives `413`, `422`, or trusted path is refused | File/request limit, invalid filename, unsupported format, disabled trusted path, or path outside allowed roots | Check source extension/name and byte limits; for browser files use multipart `file`, not server-local path. Change immutable path policy only after operator review and restart. |
| Turn gives `409 conversation_turn_active` | Another session is already processing that conversation | Wait for its operation to finish/cancel, then submit a new turn. Separate conversations may run concurrently. |
| Operation gives `429` | Per-user or global active-operation limit reached | Wait/cancel old work; only increase limits after capacity planning. |
| SSE reconnect gives `400`, `410`, or `404` | Invalid `Last-Event-ID`, cursor ahead of available history, or missing operation | For `400`, send a non-negative decimal cursor. For `410`, check operation status and use its `last_event_id` to determine where replay can resume. For `404`, use the saved turn when available. |
| A server crash left an operation interrupted | No worker survives process termination | Inspect retry eligibility; explicitly retry eligible tasks. Do not assume a started-but-unfinished operation succeeded. |
| `503 operation_database_in_use`, storage failure, or corrupt resource | Another process owns the local database, or storage is unhealthy | Stop conflicting processes, check filesystem permissions/free space, make an offline backup, and inspect discovery issues. Do not manually modify live SQLite/Qdrant files. |

Readiness is not model readiness: `/ready` reports that the runtime registry
can serve requests; `/api/v1/runtime` shows whether **this user's** models
are configured. A hosted deployment should keep RAVEN behind its authenticating
backend and not expose the shared launch token or internal user UUID mapping
to a public browser.


## Advanced component API

`Raven` is the recommended composition root, but the package exports its major
components for applications that need lower-level integration or focused
testing:

| Area | Public components |
| --- | --- |
| Operations and events | `OperationManager`, `Operation`, `OperationTask`, records, statuses, retry policies, cleanup services, `EventStream`, `Event`, and `EventType` |
| Providers | `Provider`, `ModelSpec`, `ModelRole`, `OllamaManager`, and `LiteLLMManager` |
| Knowledge and conversations | `KnowledgeBase`, `Knowledge`, `ConversationManager`, `Conversation`, and `DiscoveryIssue` |
| Document processing | `DocumentParser`, parsed element/document contracts, `ProvenanceAwareSemanticSplitter`, and semantic unit/section contracts |
| Pipelines | `IngestionPipeline`, all four retrieval pipeline classes, the common `RetrievalPipeline`, and `Reconstructor` |
| Agent execution | `AgentPolicy`, `AgentHarness`, `Session`, and `SessionRun` |
| Configuration | `SystemConfig`, `PathConfig`, and `RuntimeConfig` |

These types preserve the same operation and event contracts used by the
facade. When composing them manually, the application becomes responsible for
their dependency order, lifecycle, shared operation context, and cleanup. Most
applications should begin with `Raven` and move to individual components only
when they need a boundary the facade intentionally does not expose.

### Additional `Raven` facade methods

The main workflows above introduce the commonly used methods. These public
facade methods support custom orchestration, pagination, inspection, and
administration without requiring direct component ownership:

| Method and required inputs | Result and use |
| --- | --- |
| `raven.runtime_config` | Current immutable `RuntimeConfig` snapshot; read-only. Use `update_runtime_settings()` to persist a change. |
| `await raven.create_operation(name)` | A new `Operation`; caller starts its tasks with `operation.run(name, worker)`. |
| `await raven.submit_operation(name, worker)` | Creates an operation and starts a root task; returns the `Operation`. `worker` is an async function taking that operation. Inspect events/status or await its completion. |
| `await raven.list_operation_tasks(operation_id, status=None, limit=None, after_task_id=None)` | A page of `OperationTaskRecord` values for one operation; use the final task ID as the next cursor. Unknown operation raises a not-found error. |
| `await raven.list_knowledges_page(limit=10, after_name=None)` | List of knowledge summary dictionaries, ordered for cursor pagination; the caller chooses a positive page size. |
| `await raven.update_knowledge(name, user_summary="...")` | `OperationTask`; await `.result()` before reading the revised summary. Missing knowledge fails. |
| `await raven.list_file_sections_page(knowledge_name, file_name, limit=10, after_section_index=0)` | Section dictionaries after the given index; an unknown file raises `file_not_found`. |
| `await raven.count_knowledge_vectors(knowledge_name)` | Integer number of vector points in that knowledge. |
| `await raven.count_knowledge_files(knowledge_name)` | Integer number of stored files. |
| `await raven.delete_knowledge(name)` | `OperationTask`; await `.result()` and observe events. Removes the knowledge's storage; back up first. |
| `raven.get_conversation_details(conversation_id)` | Persistent metadata dictionary, including scope and title; missing ID raises `conversation_not_found`. |
| `await raven.get_conversation_messages_page(conversation_id, limit=10, after_message_id=0)` | Canonical message records after the cursor, not compacted model context. |
| `await raven.get_conversation_turn(conversation_id, turn_id)` | Durable turn dictionary; missing turn raises `conversation_turn_not_found`. |
| `await raven.get_conversation_turn_messages(conversation_id, turn_id)` | LlamaIndex `ChatMessage` objects belonging to that turn. |
| `await raven.get_conversation_turn_messages_page(conversation_id, turn_id, limit=10, after_message_id=0)` | Canonical per-turn message records with IDs suitable for paging a UI. |
| `await raven.get_conversation_preferences(conversation_id)` | List of `{preference_id, text}` dictionaries for that conversation. |
| `await raven.delete_conversation(conversation_id)` | `OperationTask`; await `.result()` and observe events. Deletes the persisted transcript, activity history, memory, and preferences. |

For a long-running method, the returned `OperationTask` is the handle:
`await task.result()` retrieves completion or raises its error, while
`task.operation_id` selects its replayable events. Read-only methods return
their values directly. Pagination methods return a bounded list, not an HTTP
`{items,next_cursor}` envelope; that envelope is added by FastAPI.


## Development and testing

Clone the repository and install the package in editable mode with development
and server dependencies:

```bash
python -m venv .venv
python -m pip install -e ".[dev,server]"
```

Run the fast suite without external model services:

```bash
python -m pytest -m "not integration"
```

Run the complete pytest suite when the required external services and model
assets are available:

```bash
python -m pytest
```

Tests marked `integration` may require a running Ollama server, installed test
models, cloud-provider credentials, or real document assets. Use a dedicated
temporary RAVEN home for destructive, crash-recovery, and subprocess tests; do
not point them at application data.

The repository also contains focused end-to-end scripts for provider,
ingestion, retrieval, reconstruction, session, memory, retry, and hard-crash
behavior. Review a script's model and filesystem requirements before running
it.

Build distributions with:

```bash
python -m pip install build
python -m build
```

Before release, test the generated wheel in fresh environments in both
supported installation forms:

```bash
python -m pip install ./dist/noomexai_raven-0.2.1-py3-none-any.whl
python -m pip install "./dist/noomexai_raven-0.2.1-py3-none-any.whl[server]"
```

Use the wheel filename produced by your build.

The core wheel must cover both Ollama and LiteLLM/cloud provider paths. The
server smoke test must additionally verify CLI startup, bearer authentication,
OpenAPI generation, SSE replay, cancellation, and shutdown.


## Current limitations

- The server uses one Uvicorn worker. Embedded Qdrant and per-user SQLite
  stores are not a distributed multi-process backend.
- Durable operations and events are local to one RAVEN home. Running several
  replicas requires an external coordination and storage design that this
  release does not provide.
- RAVEN does not own the Ollama process. A desktop application or deployment
  host must supervise it and clean up its host-level resources.
- Model specifications and adapter/VRAM state are runtime configuration, not
  durable state. Configure models again after a server restart.
- Supported ingestion formats in this release are `.txt`, `.md`, `.pdf`, and
  `.docx`.
- Parsing and reconstruction preserve semantic content and navigation ranges,
  but do not reproduce the source's exact visual layout or bounding boxes.
- Crash recovery marks abandoned operations as interrupted and supports
  explicit retry for eligible tasks. It does not serialize arbitrary Python
  execution state or resume a model call at the exact interrupted instruction.
- Uploaded sources can be retried only within the configured private retention
  window. After expiry, the user must upload the file again.
- RAVEN validates user UUIDs and isolates their storage, but public account
  authentication, account-to-UUID authorization, TLS, and internet-facing
  policy belong to the host application.
- Cloud model capabilities, rate limits, availability, and pricing are defined
  by the selected provider and LiteLLM adapter.


## License and third-party notices

RAVEN is released under the [MIT License](LICENSE.txt).

The distribution depends on third-party open-source packages under their own
licenses. See [ThirdPartyNotices.txt](ThirdPartyNotices.txt) for the packages,
license identifiers, copyright notices, and license texts included with this
project.
