# Content Types Reference

All content types that can appear in `user_input`.

Send the **exact `type` name** listed below — no aliases.  
`user_input` can also be a plain string, which is treated as `{ "type": "text", "text": "..." }`.

See `AGENTS_API.md` for the full request structure.  
See `ai-api-types.ts` for ready-to-use TypeScript types.

---

## How to use

Every API that accepts messages (`/ai/chat`, `/ai/conversations/{id}`, `/ai/agents/{id}`) accepts
`user_input` as either a plain string or a mixed list of typed blocks:

```json
{
  "user_input": [
    { "type": "text",  "text": "What is in this image?" },
    { "type": "image", "url": "https://example.com/photo.png" }
  ]
}
```

---

## Live types — fully implemented

### `text`
Plain text. The default type when `user_input` is a string.

| Field | Type | Required | Notes |
|---|---|---|---|
| `text` | string | yes | |

**Provider support:** All

---

### `image`
An image passed by URL or base64.

| Field | Type | Required | Notes |
|---|---|---|---|
| `url` | string | one of | Public URL |
| `base64_data` | string | one of | Include `mime_type` |
| `mime_type` | string | no | Auto-detected from URL/data if omitted |
| `media_resolution` | string | no | Google only: `"low"` / `"medium"` / `"high"` |

**Provider support:** OpenAI, Anthropic, Google

---

### `audio`
An audio file. Routing depends on `auto_transcribe`.

| Field | Type | Required | Notes |
|---|---|---|---|
| `url` | string | one of | Public URL |
| `base64_data` | string | one of | |
| `mime_type` | string | no | Auto-detected if omitted |
| `auto_transcribe` | bool | no | Default `false`. If `true`, transcribes via Groq Whisper first, then sends text to any provider |
| `transcription_model` | string | no | Default `"whisper-large-v3-turbo"`. Used only when `auto_transcribe: true` |
| `transcription_language` | string | no | ISO-639-1 e.g. `"en"`, `"es"`. Auto-detected if omitted |

**Provider support:**
- `auto_transcribe: false` → Google only (native)
- `auto_transcribe: true` → All providers (transcribed to text first)

---

### `video`
A video file.

| Field | Type | Required | Notes |
|---|---|---|---|
| `url` | string | one of | Public URL |
| `base64_data` | string | one of | |
| `file_uri` | string | one of | Google File API URI (if pre-uploaded) |
| `mime_type` | string | no | Auto-detected if omitted |

**Provider support:** Google only. Silently skipped by OpenAI and Anthropic.

---

### `youtube_video`
A YouTube URL passed natively to Google.

| Field | Type | Required | Notes |
|---|---|---|---|
| `url` | string | yes | `youtube.com/watch?v=...` or `youtu.be/...` |

**Provider support:** Google only. Silently skipped (with warning) by OpenAI and Anthropic.

---

### `document`
A file/document (PDF, DOCX, etc.).

| Field | Type | Required | Notes |
|---|---|---|---|
| `url` | string | one of | Public URL |
| `base64_data` | string | one of | Include `mime_type` |
| `mime_type` | string | no | Auto-detected if omitted |

**Provider support:** OpenAI (`input_file`), Anthropic (`document`), Google (inline/fileData)

---

## System / internal types — not for clients to send

These appear in stored conversation history and may be received when loading prior messages.
Do not send them in new requests.

| Type | Purpose |
|---|---|
| `thinking` | AI chain-of-thought reasoning output. Contains provider-specific signature required for multi-turn round-trips. |
| `tool_call` | AI-requested tool invocation. Backend-managed. |
| `tool_result` | Result of a tool execution. Backend-managed. |
| `code_execution` | Google code interpreter execution block. Backend-managed. |
| `code_execution_result` | Output from Google code interpreter. Backend-managed. |
| `web_search_call` | Internal OpenAI web search event. Backend-managed. |

---

## Structured input types

These types are resolved server-side before the message reaches the model. The
resolved content (clean XML) is injected into the current turn. All types are
fully parsed, stored, and reconstructed across conversation turns.

**Status overview:**

| Type | Status |
|---|---|
| `input_webpage` | ✅ Live |
| `input_notes` | ✅ Live |
| `input_task` | ✅ Live |
| `input_table` | ✅ Live |
| `input_list` | ✅ Live |
| `input_data` | ✅ Live |
| `input_context` | ❌ Schema TBD |

### Common control fields

All structured input types share these four fields:

| Field | Type | Default | Notes |
|---|---|---|---|
| `convert_to_text` | bool | `true` | Resolved content is injected as plain text. Currently assumed `true` for all types. |
| `optional_context` | bool | `false` | When `true`, a fetch/resolve failure silently drops the block instead of aborting the request. |
| `keep_fresh` | bool | `false` | When `true`, the resolved content is injected into the current turn and sent to the model, but is **stripped from the message before it is saved to the database**. The block's structural definition (type, IDs, `keep_fresh: true`) is always saved so the next turn re-fetches fresh content automatically. Use for data that changes frequently. |
| `editable` | bool | `false` | When `true`, the CRUD/patch tools for this content type are automatically injected into the agent's active tool list. The model can then read, update, and patch the item without the agent needing to pre-configure those tools. Pairs naturally with `keep_fresh` — edits made by the model are immediately visible on the next turn. |

#### `editable` tool sets

| Type | Tools injected when `editable: true` |
|---|---|
| `input_notes` | `note` (action-dispatched) |
| `input_task` | `task` (action-dispatched) |
| `input_table` | `dataset` (action-dispatched) |
| `input_list` | `picklist` (action-dispatched) |
| `input_workbook` | `workbook` (action-dispatched) |
| `input_document` | `document` (action-dispatched) |
| `input_webpage` | _(none — external content is read-only)_ |
| `input_data` | _(none — raw DB refs are read-only by default)_ |
| `input_context` | _(TBD — schema not yet finalised)_ |

Tools are deduped before injection and only added once per request even if multiple blocks of the same type declare `editable: true`.  The injection happens on every iteration of the tool loop (idempotent), so tools remain available across the full agentic turn.

---

### `input_webpage`
Scrape one or more public URLs and inject their content as `<web_context>` XML.
Each URL is scraped concurrently. Clean AI-optimised content is extracted.

Failed scrapes produce a visible `<webpage status="failed">` notice in the XML
so the model knows the page was unavailable and can tell the user. Failures are
also emitted as a `structured_input_warning` stream event.

**Error policy:** `optional_context=false` raises only if ALL URLs fail.
Partial failures are always included as failure notices regardless.

| Field | Type | Required | Notes |
|---|---|---|---|
| `urls` | string[] | yes | One or more public URLs to scrape |
| `convert_to_text` | bool | no | Default `true` |
| `optional_context` | bool | no | Default `false` |
| `keep_fresh` | bool | no | Default `false`. Recommended `true` for live/frequently-updated pages |
| `editable` | bool | no | Default `false`. No-op for `input_webpage` — external content has no write tools |

```json
{
  "type": "input_webpage",
  "urls": [
    "https://example.com/blog/post-1",
    "https://example.com/docs/api"
  ],
  "keep_fresh": true
}
```

**Client-side warning event** (emitted when one or more scrapes fail):
```json
{
  "event": "structured_input_warning",
  "type": "input_webpage",
  "failures": [
    { "url": "https://example.com/blog/post-1", "reason": "timeout" }
  ]
}
```

---

### `input_notes`
Fetch one or more user notes by ID and inject them as LLM-friendly XML.

| Field | Type | Required | Notes |
|---|---|---|---|
| `note_ids` | string[] | yes | One or more note IDs to fetch |
| `template` | string | no | XML template: `"full"` (default) \| `"compact"` \| `"minimal"` |
| `convert_to_text` | bool | no | Default `true` |
| `optional_context` | bool | no | Default `false`. Per-ID failures are logged; block is dropped if all fail |
| `keep_fresh` | bool | no | Default `false` |
| `editable` | bool | no | Default `false`. When `true` injects: `note_get`, `note_list`, `note_create`, `note_update`, `note_patch`, `note_delete` |

```json
{ "type": "input_notes", "note_ids": ["abc123", "def456"] }
```

---

### `input_task`
Fetch one or more task objects by ID and inject them as LLM-friendly XML.

| Field | Type | Required | Notes |
|---|---|---|---|
| `task_ids` | string[] | yes | One or more task IDs to fetch |
| `template` | string | no | XML template: `"full"` (default) \| `"compact"` |
| `convert_to_text` | bool | no | Default `true` |
| `optional_context` | bool | no | Default `false`. Per-ID failures are logged; block is dropped if all fail |
| `keep_fresh` | bool | no | Default `false` |
| `editable` | bool | no | Default `false`. When `true` injects: `task_get`, `task_list`, `task_create`, `task_update`, `task_delete` |

```json
{ "type": "input_task", "task_ids": ["task-xyz"], "keep_fresh": true }
```

---

### `input_table`
Fetch one or more user table bookmarks and inject them as XML. Each bookmark
is the exact JSON object the UI copies to the clipboard — pass them through as-is.

| Field | Type | Required | Notes |
|---|---|---|---|
| `bookmarks` | object[] | yes | One or more table bookmark objects (see shapes below) |
| `convert_to_text` | bool | no | Default `true` |
| `optional_context` | bool | no | Default `false`. Per-bookmark failures are logged; block is dropped if all fail |
| `keep_fresh` | bool | no | Default `false` |
| `editable` | bool | no | Default `false`. When `true` injects the action-dispatched `dataset` tool |

**Bookmark shapes:**

| `type` | Required fields | What you get |
|---|---|---|
| `full_table` | `table_id` | All rows |
| `table_column` | `table_id`, `column_name` | All rows, one column |
| `table_row` | `table_id`, `row_id` | One row |
| `table_cell` | `table_id`, `row_id`, `column_name` | Single value |

All bookmark shapes accept an optional `table_name` field (cosmetic, for XML labelling).

```json
{
  "type": "input_table",
  "bookmarks": [
    { "type": "table_row", "table_id": "3b81bb2e-...", "row_id": "cfc3841f-...", "table_name": "AI Model Benchmarks" }
  ],
  "keep_fresh": true
}
```

---

### `input_list`
Fetch one or more user list bookmarks and inject them as XML.

| Field | Type | Required | Notes |
|---|---|---|---|
| `bookmarks` | object[] | yes | One or more list bookmark objects (see shapes below) |
| `convert_to_text` | bool | no | Default `true` |
| `optional_context` | bool | no | Default `false`. Per-bookmark failures are logged; block is dropped if all fail |
| `keep_fresh` | bool | no | Default `false` |
| `editable` | bool | no | Default `false`. When `true` injects the action-dispatched `picklist` tool |

**Bookmark shapes:**

| `type` | Required fields | What you get |
|---|---|---|
| `full_list` | `list_id` | All items |
| `list_group` | `list_id`, `group_name` | Items in one group |
| `list_item` | `list_id`, `item_id` | Single item |

All bookmark shapes accept an optional `list_name` field (cosmetic).

```json
{
  "type": "input_list",
  "bookmarks": [
    { "type": "full_list", "list_id": "a1b2c3d4-...", "list_name": "Resume Checklist" }
  ]
}
```

---

### `input_data`
Fetch data from system tables using typed DataRef descriptors. Each ref targets
a table in the server-side allowlist. Results are wrapped in `<data_context>` XML.

**Allowed tables:** `notes`, `tasks`, `projects`, `workspaces`, `organizations`, `contacts`, `contact_groups`

| Field | Type | Required | Notes |
|---|---|---|---|
| `refs` | object[] | yes | One or more DataRef objects (see shapes below) |
| `convert_to_text` | bool | no | Default `true` |
| `optional_context` | bool | no | Default `false`. Per-ref failures are logged; block is dropped if all fail |
| `keep_fresh` | bool | no | Default `false` |

**DataRef shapes:**

**`db_record`** — fetch one full row by primary key

| Field | Type | Required | Notes |
|---|---|---|---|
| `ref_type` | `"db_record"` | yes | |
| `table` | string | yes | Must be in allowlist |
| `id` | string | yes | Primary key of the row |
| `label` | string | no | Display label for XML |
| `fields` | string[] | no | Project specific fields. Omit for all allowed fields |
| `optional_context` | bool | no | Per-ref override |

**`db_query`** — fetch multiple rows with optional filters

| Field | Type | Required | Notes |
|---|---|---|---|
| `ref_type` | `"db_query"` | yes | |
| `table` | string | yes | Must be in allowlist |
| `label` | string | no | Display label for XML |
| `filter` | object | no | Equality filters: `{ "field": value, ... }` |
| `fields` | string[] | no | Project specific fields. Omit for all allowed fields |
| `sort` | object | no | `{ "field": "col", "direction": "asc" \| "desc" }` |
| `limit` | number | no | Default 50 |
| `optional_context` | bool | no | Per-ref override |

**`db_field`** — fetch a single field value from one row

| Field | Type | Required | Notes |
|---|---|---|---|
| `ref_type` | `"db_field"` | yes | |
| `table` | string | yes | Must be in allowlist |
| `id` | string | yes | Primary key of the row |
| `field_name` | string | yes | Field to retrieve. Must be in the allowlist for this table |
| `label` | string | no | Display label for XML |
| `optional_context` | bool | no | Per-ref override |

```json
{
  "type": "input_data",
  "refs": [
    { "ref_type": "db_record", "table": "tasks", "id": "abc-123" },
    { "ref_type": "db_query",  "table": "tasks", "filter": { "status": "incomplete" }, "limit": 10 },
    { "ref_type": "db_field",  "table": "notes", "id": "note-456", "field_name": "content" }
  ],
  "keep_fresh": true
}
```

---

### `input_context`
Flexible named context injection. Schema to be ironed out — do not use yet.

---

## Source files

| File | Contents |
|---|---|
| `unified_content.py` | `TextContent`, `ThinkingContent`, `UnifiedContent` union, `reconstruct_content()` |
| `media_config.py` | `ImageContent`, `AudioContent`, `VideoContent`, `YouTubeVideoContent`, `DocumentContent` |
| `extra_config.py` | `CodeExecutionContent`, `CodeExecutionResultContent`, `WebSearchCallContent` |
| `tools_config.py` | `ToolCallContent`, `ToolResultContent` |
| `structured_input_config.py` | All 7 structured input types + `reconstruct_structured_input()` |
| `message_config.py` | `UnifiedMessage.parse_content()` — entry point for all inbound content parsing |
| `AGENTS_API.md` | Full agents endpoint reference (request fields, context, ide_state, stream events) |
| `ai-api-types.ts` | TypeScript types for everything in this document |
