# Agents API — Frontend Reference

Complete reference for all three AI endpoints. Read alongside `CONTENT_TYPES.md`
and `ai-api-types.ts` for the full picture.

---

## Endpoints at a glance

| Endpoint | Purpose |
|----------|---------|
| `POST /ai/agents/{agent_id}` | Start a new conversation with an agent |
| `POST /ai/conversations/{conversation_id}` | Continue any conversation (turn 2+) |
| `POST /ai/conversations/{conversation_id}/tool_results` | Return client-executed tool results |
| `POST /ai/chat` | Raw config chat (no agent, full manual control) |
| `POST /ai/agents/{agent_id}/warm` | Pre-load agent into cache (optional, reduces first-turn latency) |
| `POST /ai/conversations/{conversation_id}/warm` | Pre-load conversation into cache (optional) |

All streaming endpoints return a **JSONL stream** — one JSON object per line:
```json
{ "event": "<event_type>", "data": { ... } }
```

---

## POST /ai/agents/{agent_id}

Start a new agent conversation.

```
POST /ai/agents/{agent_id}
Authorization: Bearer <supabase-jwt>
Content-Type: application/json
```

### Request body — `AgentStartRequest`

```typescript
interface AgentStartRequest {
  user_input?: string | ContentBlock[] | null;
  variables?: Record<string, unknown> | null;
  config_overrides?: LLMParams | null;
  stream?: boolean;         // Default: true
  debug?: boolean;          // Default: false
  client_tools?: string[];  // Default: []
  ide_state?: IdeState | null;
  context?: Record<string, ContextValue>;  // Default: {}

  // Organizational scope — see "Scope fields" section below.
  organization_id?: string | null;
  workspace_id?: string | null;
  project_id?: string | null;
  task_id?: string | null;
}
```

All fields are optional. A bare `{}` body is valid (the agent runs with its defaults
and no user message).

---

## POST /ai/conversations/{conversation_id}

Continue an existing conversation. Use this for every turn after the first.

### Request body — `ConversationContinueRequest`

```typescript
interface ConversationContinueRequest {
  user_input: string | ContentBlock[];   // Required
  config_overrides?: LLMParams | null;
  stream?: boolean;         // Default: true
  debug?: boolean;          // Default: false
  client_tools?: string[];  // Default: []
  ide_state?: IdeState | null;
  context?: Record<string, ContextValue>;  // Default: {}

  // Organizational scope — overrides the scope set on turn 1 for this turn only.
  // Typically omitted when the scope is stable across the conversation.
  organization_id?: string | null;
  workspace_id?: string | null;
  project_id?: string | null;
  task_id?: string | null;
}
```

**Important difference from agents:** `user_input` is **required** here.
`variables` is not accepted — variable substitution only happens on turn 1.
On subsequent turns, `ide_state` only injects `selected_text` and `diagnostics`
(the per-turn ephemeral fields). Stable fields like `git`, `workspace`, and
`active_file` were set as variables on turn 1 and do not need re-sending.

---

## POST /ai/conversations/{conversation_id}/tool_results

Return client-executed tool results to unblock the suspended AI loop.

A `tool_delegated` event HARD-SUSPENDS the AI loop (the stream ends) and the
turn is persisted as `paused`; the loop resumes durably when the client POSTs
results and opens `/resume`. There is NO short answer deadline — the user may
take seconds, minutes, hours, or weeks. The only timeout is a far-future
server-side abandonment backstop on `cx_tool_call.expires_at`
(`DELEGATED_CALL_ABANDON_AFTER_SECONDS`, default 30 days; per-tool override via
`tools.max_client_wait_seconds`), and even an expired row is superseded by a
late genuine answer. Call this endpoint to provide the result.

### Request body — `ToolResultsRequest`

```typescript
interface ToolResultsRequest {
  results: ClientToolResult[];
}

interface ClientToolResult {
  call_id: string;       // Must match call_id from the tool_delegated event
  tool_name: string;     // Must match tool_name from the tool_delegated event
  output?: unknown;      // The tool's return value (any JSON-serializable value)
  is_error?: boolean;    // Default: false
  error_message?: string; // Set when is_error=true
}
```

### Response

```json
{ "resolved": ["call_id_1", "call_id_2"], "count": 2 }
```

If any `call_id` is not found (already timed out or unknown), the endpoint
returns HTTP 404 with:
```json
{
  "message": "Some call_ids were not found (already timed out or unknown).",
  "not_found": ["call_id_that_expired"],
  "resolved": ["call_id_1"]
}
```

---

## `user_input`

Either a plain string or a mixed array of typed content blocks:

```json
// Plain string
{ "user_input": "Summarize this document" }

// Mixed array
{
  "user_input": [
    { "type": "text", "text": "What does this image show?" },
    { "type": "image", "url": "https://example.com/photo.png" },
    { "type": "input_webpage", "urls": ["https://docs.example.com"] }
  ]
}
```

All content block types are documented in `CONTENT_TYPES.md`.

---

## `config_overrides` — `LLMParams`

Override any model parameter for this request only. All fields are optional and
do not affect the agent's stored configuration.

```typescript
interface LLMParams {
  model?: string;               // e.g. "gpt-4o", "claude-opus-4-5", "gemini-2.0-flash"

  // Sampling
  max_output_tokens?: number;
  temperature?: number;
  top_p?: number;
  top_k?: number;

  // Tool calling
  tool_choice?: "none" | "auto" | "required";
  parallel_tool_calls?: boolean;

  // OpenAI thinking / reasoning
  reasoning_effort?: "auto" | "none" | "minimal" | "low" | "medium" | "high" | "xhigh";
  reasoning_summary?: "concise" | "detailed" | "never" | "auto" | "always";

  // Google Gemini thinking
  thinking_level?: "minimal" | "low" | "medium" | "high";
  include_thoughts?: boolean;

  // Anthropic + legacy Gemini thinking
  thinking_budget?: number;

  // Cerebras-specific
  clear_thinking?: boolean;      // Strip <thinking> blocks from the final response
  disable_reasoning?: boolean;   // Suppress reasoning entirely

  // Output format / control
  response_format?: Record<string, unknown>;
  stop_sequences?: string[];
  stream?: boolean;
  store?: boolean;
  verbosity?: string;

  // Provider-native features
  internal_web_search?: boolean;    // Model-native web search (OpenAI / Google)
  internal_url_context?: boolean;   // Model reads URLs in context natively (Google)

  // Image generation
  size?: string;
  quality?: string;
  count?: number;

  // Audio / TTS
  tts_voice?: string | Array<
    | { name: string; voice: string }       // Google multi-speaker mapping
    | { text: string; voice_id: string }    // ElevenLabs dialogue turn
  >;
  audio_format?: string;

  // Video generation
  seconds?: string;
  fps?: number;
  steps?: number;
  seed?: number;
  guidance_scale?: number;
  output_quality?: number;
  negative_prompt?: string;
  output_format?: string;
  width?: number;
  height?: number;
  frame_images?: unknown[];
  reference_images?: unknown[];
  image_loras?: unknown[];
  disable_safety_checker?: boolean;
}
```

---

## `client_tools`

A list of tool names the **client** will execute instead of the server.

### How it works

1. Client sends `client_tools: ["open_file", "run_command"]` in the request.
2. When the model decides to call one of those tools, the AI executor does NOT
   run it server-side. Instead it suspends and emits a **`tool_delegated`** stream event.
3. The client receives the event, executes the tool locally, then POSTs the
   result to `POST /ai/conversations/{conversation_id}/tool_results`.
4. The executor receives the result (via `/tool_results` + `/resume`) and
   continues the AI loop normally.
5. There is no short response deadline — the suspended loop waits durably (the
   user may answer in seconds or weeks). Only a far-future abandonment backstop
   (default 30 days) ever times a delegated call out, and a late genuine answer
   still supersedes it and resumes.

### `tool_delegated` event shape

```json
{
  "event": "tool_delegated",
  "data": {
    "tool_name": "open_file",
    "call_id": "tc_abc123",
    "arguments": { "path": "src/App.tsx" }
  }
}
```

Multiple tools may be delegated in a single turn if the model calls more than one.

### Rules

- Tool names must match registered tools on the server. Unknown names are silently ignored.
- The client **must** POST results even for errors — use `is_error: true` and `error_message`.
  The model needs to know a tool failed so it can respond appropriately.
- Multiple tool results can be batched into a single `tool_results` POST.

---

## `ide_state`

Structured snapshot of the user's editor. When provided:

- Fields expand into `vsc_*` variables that slot into `{{variable_name}}`
  placeholders in the agent's system prompt.
- An agent must declare the variable in its `variable_defaults` to use it.
  Agents that don't declare them ignore the state entirely — no side effects.
- The `vsc_get_state` tool is auto-injected so the model can request state on demand.
- On **turn 1 (agents endpoint)**: all fields are expanded as session variables.
- On **turn 2+ (conversations endpoint)**: only `selected_text` and `diagnostics`
  are re-injected (they change per-turn). `git`, `workspace`, `active_file` were
  set on turn 1 and do not need re-sending.

```typescript
interface IdeState {
  active_file?: {
    path: string;
    content: string;
    language: string;
  };
  selected_text?: string;
  diagnostics?: Array<{
    message: string;
    severity: string;   // "error" | "warning" | "information" | "hint"
    range: Record<string, unknown>;
    source?: string;
  }>;
  workspace?: {
    name: string;
    folders: string[];
  };
  git?: {
    branch: string;
    status: string;
  };
}
```

**Variable reference:**

| Variable | Expands from |
|----------|-------------|
| `vsc_active_file_path` | `active_file.path` |
| `vsc_active_file_content` | `active_file.content` |
| `vsc_active_file_language` | `active_file.language` |
| `vsc_selected_text` | `selected_text` |
| `vsc_diagnostics` | `diagnostics` (formatted string) |
| `vsc_workspace_name` | `workspace.name` |
| `vsc_workspace_folders` | `workspace.folders` (newline-joined) |
| `vsc_git_branch` | `git.branch` |
| `vsc_git_status` | `git.status` |
| `vsc_active_file_all` | path + language + content block |
| `vsc_editor` | selected_text + diagnostics block |
| `vsc_workspace_all` | workspace name + folders block |
| `vsc_git_all` | branch + status block |
| `vsc_all` | everything above combined |

---

## `context` — Deferred Context Objects

The most powerful field on the request. Passes **runtime data** to the agent
without embedding it in the agent's system prompt. This is how you inject
large content (files, profiles, records, etc.) dynamically.

### Two-tier injection model

```
Tier 1 → System prompt (condensed, agent-awareness only)
Tier 2 → User message (full manifest, always fresh, never persisted to DB)
```

**Tier 1 — system prompt awareness block**

A short XML block is injected into the system prompt listing available context
keys and their types. This gives the model awareness that data is available
without consuming tokens. Example:

```xml
<deferred_context_available>
  Keys: active_file (text), user_profile (json), style_guide (file_url)
  Use context(action="get", key, mode) to retrieve. Full manifest is in the user message.
</deferred_context_available>
```

The agent developer can opt out of this block by setting `include_context_block: false`
on their agent's system instruction configuration.

**Tier 2 — user message manifest**

The full manifest is appended to the current user message as ephemeral text
(bracketed by internal sentinels). It lists key, type, size, and description
for every context object. It is **never written to the database** — it is always
injected fresh from the current request so it never accumulates across turns.
Example:

```
## Available Context

The following additional context is available on demand.
Use the `context` tool (action="get") to retrieve any item when you need it.

| Key | Type | Size | Description |
|-----|------|------|-------------|
| active_file | text | 12,400 chars | The file currently open in the editor |
| user_profile | json | 3 fields | (no description) |
| style_guide | file_url | remote file | (no description) |
```

### Value types

Content can be any JSON-serializable value. The type is inferred when the
agent has no Mandate for the key:

| Value shape | Inferred type |
|-------------|--------------|
| Plain string | `text` |
| String starting with `http://` or `https://` | `file_url` |
| Object or array | `json` |

If the agent defines a **slot** for the key (stored in the DB under `context_policies`),
that slot's `type`, `label`, `description`, and `max_inline_chars` take precedence.

### Agent-defined context policies (configured per-agent in the DB)

Agents can pre-declare the context keys they expect. Slots are stored in the
`prompts.context_policies` / `prompt_builtins.context_policies` JSONB column and are
loaded as part of the agent definition on every request. They are now carried
on the `Agent` object itself (`.context_policies`), so sub-agents and internal
tools that call `Agent.from_id()` or `Agent.from_prompt()` automatically have
access to the slot metadata without a second DB lookup.

```typescript
interface ContextPolicy {
  key: string;
  type: ContextObjectType;
  label?: string;
  description?: string;
  /**
   * When set, content larger than this many characters is truncated in the
   * Tier 2 manifest. The model calls context(action="get", key, mode="page")
   * to read the rest. Use for large files / documents.
   */
  max_inline_chars?: number;
  /**
   * When set, context(action="get", key, mode="summary") runs this sub-agent
   * (receives full content in {{content}}). mode=summary ALWAYS succeeds even
   * when unset: source-backed objects return descriptor.summary; otherwise the
   * first page with fell_back_from="summary".
   */
  summary_agent_id?: string;
}
```

**How slots interact with the request `context` dict:**

| Scenario | What happens |
|----------|-------------|
| Key in `context` matches a slot | Slot's `type`, `label`, `description`, `max_inline_chars` are used |
| Key in `context` has no matching slot | Ad-hoc object; type inferred (string → `text`, URL → `file_url`, object/array → `json`) |
| Slot defined but no content sent for it | Silently skipped — not an error |

Slot-defined keys and ad-hoc keys can be freely mixed in a single request.

### `ContextObjectType`

```typescript
type ContextObjectType =
  | "text"       // Plain text string
  | "file_url"   // Remote file URL — model receives the URL, not the content
  | "json"       // Object or array
  | "db_ref"     // Database rows (result of a DataRef query)
  | "user"       // User profile object
  | "org"        // Organization object
  | "workspace"  // Workspace object
  | "project"    // Project object
  | "task";      // Task object
```

### `context` tool (action="get") — how the model retrieves content

The `context` tool is **automatically injected** into the model's tool list
whenever `context` is non-empty. The model calls it (with `action="get"`) like
any other tool.

**Arguments:**

| Argument | Type | Required | Notes |
|----------|------|----------|-------|
| `key` | string | yes | Key from the context manifest |
| `mode` | string | yes | `"full"` \| `"page"` \| `"summary"` |
| `offset` | number | `page` only | Starting character position. Default 0 |
| `chars` | number | `page` only | Characters to return. Default 4000 |

**`mode: "full"` response:**

```json
{
  "key": "active_file",
  "type": "text",
  "label": "Active File",
  "content": "// full content here...",
  "total_chars": 12400
}
```

**`mode: "page"` response:**

```json
{
  "key": "active_file",
  "type": "text",
  "label": "Active File",
  "content": "// first 4000 chars...",
  "offset": 0,
  "chars_returned": 4000,
  "total_chars": 12400,
  "has_more": true,
  "next_offset": 4000
}
```

The model can paginate by calling `context(action="get", key, mode="page", offset=4000, chars=4000)` etc.
`has_more: false` when `offset + chars >= total_chars`.

**`mode: "summary"` response:**

Always succeeds. Resolution order: AI summary when `summary_agent_id` is set
(`summary_kind=agent`); source-backed objects return the precomputed
descriptor (`summary_kind=descriptor`); otherwise the first page with
`fell_back_from="summary"`.

```json
{
  "key": "active_file",
  "type": "text",
  "label": "Active File",
  "mode": "summary",
  "summary_kind": "agent",
  "summary": "This file implements a React component that...",
  "total_chars": 12400
}
```

**Error response** (all modes):

```json
{
  "success": false,
  "error": {
    "error_type": "not_found",
    "message": "Context object 'bad_key' not found.",
    "suggested_action": "Available keys: [active_file, user_profile]"
  }
}
```

### Full context example

```json
{
  "user_input": "Review my code and suggest improvements",
  "context": {
    "active_file": "import React from 'react';\n// full TypeScript source...",
    "user_profile": {
      "name": "Alice",
      "role": "Senior Engineer",
      "preferences": { "style": "concise" }
    },
    "style_guide": "https://example.com/style-guide.pdf",
    "recent_errors": [
      { "file": "App.tsx", "line": 42, "message": "Type error..." }
    ]
  }
}
```

---

## Scope fields — `organization_id`, `workspace_id`, `project_id`, `task_id`

These four optional fields set the **organizational scope** for the entire request.
They are stored on `AppContext` and propagate automatically through the entire
call stack — to tools, sub-agents, memory operations, filesystem sandboxes, and
any other server-side system that needs to know "whose context is this?"

### What sets them

| Source | Who sets it | When |
|--------|-------------|------|
| `Authorization` header (JWT) | Middleware — automatic | Every request |
| Request body (agents/conversations) | Client — explicit | When scope needs to differ from JWT defaults |

Auth (user identity) always comes from the JWT. Scope fields must be explicitly sent
in the request body when you want them — they are **not** in the JWT payload.

### What uses them automatically

Once set on the request, every tool the model calls has access to the scope via
`ToolContext.project_id` and `ToolContext.organization_id`, with **no extra arguments
needed from the model**. The model never needs to pass them manually.

| Tool / system | How scope is used |
|---------------|-------------------|
| **Memory tools** (`memory_store`, `memory_recall`, `memory_search`) | Scopes memories to `project` or `organization` namespace. A memory stored with `scope="project"` is only retrievable within the same `project_id`. |
| **Filesystem tools** (`fs_read`, `fs_write`, `fs_list`, etc.) | Sandboxes file paths under `/workspaces/{user_id}/{project_id}/`. Without `project_id`, the sandbox is user-level only. |
| **Code execution** (`code_run`) | Same project-scoped workspace isolation as filesystem. |
| **Shell tool** | Same project-scoped workspace isolation. |
| **`context` tool** (action="get") | Stamps `active_scope` in `AppContext.metadata` so the context manifest knows which scope is active. |
| **Sub-agents** (`fork_for_child_agent`) | Scope fields are inherited automatically — child agents run in the same scope as the parent. |

### Memory scoping in detail

The memory tools support three scope values:

| `scope` arg | Scope ID used | What it means |
|-------------|--------------|---------------|
| `"user"` | `user_id` (always available) | Personal memory — only visible to this user |
| `"project"` | `project_id` from AppContext | Shared across all users in the project |
| `"organization"` | `organization_id` from AppContext | Shared across the entire organization |

If `project_id` is not set and the model calls `memory_store(scope="project")`,
the `scope_id` is `null` — the memory is stored without project isolation and
may collide with other users' project-scoped memories. **Always send `project_id`
if the agent uses memory tools.**

### Filesystem sandbox in detail

File paths are resolved relative to:
```
/workspaces/{user_id}/{project_id}/{relative_path}
```

Without `project_id`:
```
/workspaces/{user_id}/{relative_path}
```

Path traversal attacks are blocked — any path that would escape the sandbox
raises a `PermissionError` immediately.

### `active_scope` in context metadata

When `context` is non-empty, `apply_context_objects()` stamps the current scope
into `AppContext.metadata["active_scope"]` so tools that need it can read it:

```json
{
  "active_scope": {
    "organization_id": "org-uuid",
    "workspace_id": "ws-uuid",
    "project_id": "proj-uuid",
    "task_id": null
  }
}
```

### Example request with full scope

```json
{
  "user_input": "Find my recent notes about the Q4 plan and store a summary",
  "organization_id": "a1b2c3d4-...",
  "workspace_id": "ws-5678-...",
  "project_id": "proj-9abc-...",
  "task_id": "task-def0-..."
}
```

---

## Stream events

All events: `{ "event": "<type>", "data": { ... } }`

| Event | When emitted | `data` shape |
|-------|-------------|-------------|
| `message` | Initial connection / status | `{ "content": string }` |
| `chunk` | Each token from the model | `{ "content": string }` |
| `data` | Structured data payload (tool results, etc.) | Varies by source |
| `tool_delegated` | Model called a client tool | See below |
| `structured_input_warning` | A structured input block partially/fully failed | See below |
| `end` | Stream finished successfully | `{}` |
| `error` | Fatal error — stream closes | `{ "message": string }` |

### `tool_delegated`

The model called a tool listed in `client_tools`. The AI loop is now **suspended**
waiting for the client to POST results to `tool_results`.

```json
{
  "event": "tool_delegated",
  "data": {
    "tool_name": "open_file",
    "call_id": "tc_abc123",
    "arguments": { "path": "src/App.tsx" }
  }
}
```

Multiple `tool_delegated` events can be emitted in a single turn if the model
requests parallel tool calls. Batch all results into a single `tool_results` POST.
**No short answer deadline** — the loop suspends durably and resumes via
`/tool_results` + `/resume` whenever the client answers (seconds → weeks); only a
far-future abandonment backstop (default 30 days) ever expires a delegated call,
and a late genuine answer still supersedes it.

### `structured_input_warning`

One or more structured input blocks encountered scrape/fetch failures. The model's
context already includes a failure notice for each URL/item so it can inform the user.

```json
{
  "event": "structured_input_warning",
  "data": {
    "type": "input_webpage",
    "failures": [
      { "url": "https://example.com/page", "reason": "timeout" }
    ]
  }
}
```

---

## Differences between endpoints

| | `/ai/agents/{id}` | `/ai/conversations/{id}` |
|-|--------------------|--------------------------|
| `user_input` | Optional | Required |
| `variables` | ✅ Accepted | ❌ Not accepted (variables are session-level, set on turn 1) |
| `ide_state` | All fields → `vsc_*` variables | Only `selected_text` + `diagnostics` (per-turn ephemeral) |
| `context` | ✅ Accepted, Mandates applied | ✅ Accepted, no Mandate (all ad-hoc) |
| Generates `conversation_id` | Yes (new UUID) | No (uses the one in the URL) |

---

## Caching / warm endpoints

Both agents and conversations support a warm endpoint for pre-loading:

```
POST /ai/agents/{agent_id}/warm
POST /ai/conversations/{conversation_id}/warm
```

Optional `source` body for agents:
```json
{ "source": "prompt_version" }
```

Response: `{ "status": "cached" | "already_cached" | "error", ... }`

These are fire-and-forget from the client's perspective. Call them in the
background after the user opens a conversation view to reduce first-message latency.

---

## Complete request example

```json
POST /ai/agents/coding-assistant

{
  "user_input": [
    { "type": "text", "text": "Review my code and check this page for relevant docs" },
    {
      "type": "input_webpage",
      "urls": ["https://react.dev/reference/react/useEffect"],
      "keep_fresh": true
    }
  ],
  "variables": {
    "language": "TypeScript",
    "project_name": "Matrx Frontend"
  },
  "config_overrides": {
    "model": "claude-opus-4-5",
    "temperature": 0.3,
    "max_output_tokens": 4096
  },
  "client_tools": ["open_file", "run_terminal_command"],
  "ide_state": {
    "active_file": {
      "path": "src/components/Chat.tsx",
      "content": "// ... file content ...",
      "language": "typescript"
    },
    "git": {
      "branch": "feature/chat-improvements",
      "status": "M src/components/Chat.tsx"
    }
  },
  "context": {
    "coding_standards": "Always use strict TypeScript. Prefer functional components.",
    "user_profile": { "name": "Alice", "role": "Senior Engineer" }
  }
}
```
