# Media Variables — Backend Implementation + FE/Translator Handoff

> Companion to the agent template + variable substitution work.
>
> **What just shipped on the backend:** variable substitution into media content parts. An agent template can declare `{{my_start_image}}` inside any `image` / `audio` / `video` / `document` / `youtube_video` block; when the agent runs with that variable filled in, Python substitutes, classifies the value (UUID / cloud URI / URL), routes it to the correct MediaRef field, and drops the block entirely if the user left an optional slot empty.
>
> **What this doc covers:**
> 1. Reference example — the agent template that drove this work
> 2. Verified safety — metadata preservation through the pipeline (no overwrite bugs)
> 3. Backend changes (already shipped)
> 4. What the FE team needs to know about the wire format
> 5. What the per-provider translator authors need to do to honor `metadata.role`
> 6. Edge cases + open considerations

---

## 1. Reference example — the agent template

This is the user's "Basic Image Generator" agent, abbreviated to the message + variables:

```jsonc
{
  "messages": [
    {"role": "system", "content": [{"type": "text", "text": ""}]},
    {
      "role": "user",
      "content": [
        {"type": "text",
         "text": "Create a podcast episode cover image:\n{{image_description}}\nDo not add text not explicitly mentioned."},

        {"type": "image", "url": "{{my_start_image}}",
         "metadata": {"role": "start_image"}},

        {"type": "image", "url": "{{my_end_image}}",
         "metadata": {"role": "end_image"}},

        {"type": "audio", "url": "{{my_audio_file}}"},
        {"type": "video", "url": "{{my_video_file}}"},
        {"type": "youtube_video", "url": "https://youtube.com/watch?v={{youtube_id}}"},
        {"type": "document", "url": "{{my_document_file}}"}
      ]
    }
  ],
  "variableDefinitions": [
    {"name": "image_description", "required": true, "defaultValue": "...", "helpText": "..."},
    {"name": "my_start_image", "required": true, "customComponent": {"type": "image"}, "defaultValue": ""},
    {"name": "my_end_image", "customComponent": {"type": "image"}, "defaultValue": ""},
    {"name": "my_audio_file", "customComponent": {"type": "audio"}, "defaultValue": ""},
    {"name": "my_video_file", "customComponent": {"type": "video"}, "defaultValue": ""},
    {"name": "youtube_id", "customComponent": {"type": "youtube"}, "defaultValue": ""},
    {"name": "my_document_file", "customComponent": {"type": "document"}, "defaultValue": ""}
  ]
}
```

The FE renders one input per `variableDefinitions` entry, picking the widget by `customComponent.type`. When the user submits, the FE sends a flat `{variable_name: value}` dict alongside the agent invocation. Python substitutes those values into the message template before calling any provider.

---

## 2. Verified safety — metadata preservation

A subagent traced `ImageContent.metadata` (and the analogous fields on `AudioContent`, `VideoContent`, `DocumentContent`, `YouTubeVideoContent`) through the pipeline. **No code path overwrites or wipes the metadata dict.** All current modifications are additive (`metadata["alt"] = ...`, `metadata.update({...})`, or reading sub-keys via `.get()`).

Audited locations:

| Location | What it does | Safe? |
|---|---|---|
| `message_config.py::parse_content()` (line ~95) | Parses incoming dicts into content classes via `_filter_kwargs` (whitelists declared fields) | ✓ — metadata flows through |
| `media_config.py::ImageContent.__post_init__` (line ~48) | Reads metadata for mime/file_id hints; sets `metadata["alt"]` from the alt field | ✓ — additive |
| Same for `AudioContent.__post_init__`, `DocumentContent.__post_init__` | Auto-detect mime; no metadata wipe | ✓ |
| `to_storage_dict()` on every media class | Creates a copy via `{**self.metadata}` for storage; conditional merge | ✓ — round-trips |
| `to_google()` / `to_openai()` / `to_anthropic()` on every media class | Build new provider-shaped dicts; do NOT include metadata; do NOT mutate the source | ✓ — immutable |
| `unified_config.py::_resolve_message_patterns` (line ~241) | Resolves `<<MATRX>>` patterns ONLY on TextContent.text | ✓ — media untouched |
| `message_config.py::reconstruct_content` for DB reload | Re-hydrates metadata directly from storage | ✓ — round-trips |

**The one fragility to watch out for in future code review:** never write `image.metadata = {...}` (replaces the whole dict). Always `image.metadata["key"] = value` or `image.metadata.update({...})`. We don't have a guard against this; just convention.

---

## 3. Backend changes (shipped)

### 3a. `coerce_to_media_ref` helper

`packages/matrx-files/matrx_files/cloud_sync/media_ref.py` — appended:

```python
def classify_media_string(value: str) -> tuple[str, str] | None:
    """Returns (field_name, value) or None for empty input."""

def coerce_to_media_ref(value, *, mime_type_hint=None) -> MediaRef | None:
    """Convert string / dict / MediaRef into a canonical MediaRef.

    Detection rules (matching FileManager's resolver expectations):
      - 36-char UUID → file_id
      - s3:// / gs:// / azure:// → file_uri
      - everything else → url

    Returns None for None / empty input — the caller drops the surrounding
    content block.
    """
```

This is the single source of truth for "user gave me a string; how do I shape it as a MediaRef." Other callers (the boundary normaliser, future media-input handlers) should use it too.

### 3b. `replace_variables` on every media content class

`packages/matrx-ai/matrx_ai/config/media_config.py`:

- New module-level helper `_substitute_and_coerce_media_fields(content, variables) -> bool`
  - Substitutes `{{var}}` patterns in `url` / `file_id` / `file_uri` / `base64_data` fields
  - Detects "this field carried a template AND the user supplied empty" → returns `True` (drop the block)
  - Re-routes the post-substitution `url` field through `coerce_to_media_ref` — so a UUID lands on `file_id`, an `s3://` URI lands on `file_uri`, an external URL stays on `url` (matches MediaRef's exactly-one-identifier rule)
  - `hasattr` guard prevents the reroute from corrupting classes (like `YouTubeVideoContent`) that only declare `url`

- `ImageContent.replace_variables(variables) -> bool` — new method
- `AudioContent.replace_variables(variables) -> bool` — new method
- `VideoContent.replace_variables(variables) -> bool` — new method
- `DocumentContent.replace_variables(variables) -> bool` — new method
- `YouTubeVideoContent.replace_variables(variables) -> bool` — new method

### 3c. `UnifiedMessage.replace_variables` dispatch

`packages/matrx-ai/matrx_ai/config/message_config.py` — was TextContent-only; now dispatches to media classes too and DROPS blocks whose `replace_variables` returns `True`. Other content types without `replace_variables` pass through untouched. The drop behavior preserves provider correctness — we never send an `image` block with an empty URL.

### 3d. Smoke test (passing)

`/tmp/smoke_var_subst.py` — exercises the user's exact agent template with 4 cases:

1. All variables filled (UUID + share-link URL + s3 URI + external URL) → 7 blocks, all routed to correct fields.
2. Optional `end_image` empty → end_image block dropped, the rest survive.
3. Mixed UUID + external URL → both image blocks survive, `metadata.role` preserved on each.
4. All optional media empty → only text and youtube survive.

Full matrx-ai test suite still: 13 fail / 323 pass / 2 errors — **identical baseline**, no new regressions from this work.

---

## 4. FE handoff — wire format

When the FE submits an agent run, send:

```jsonc
{
  "agent_id": "fc20fe69-...",
  "variables": {
    "image_description": "cells dividing",
    "my_start_image": "<value>",
    "my_end_image": "<value>",
    "my_audio_file": "<value>",
    "my_video_file": "<value>",
    "youtube_id": "<value>",
    "my_document_file": "<value>"
  }
}
```

### What value shapes Python accepts for each variable

For TEXT variables (`image_description`, `youtube_id`):
- A string. Any string. Empty string is allowed — it just becomes empty in the substituted text.

For MEDIA variables (`my_start_image`, `my_end_image`, `my_audio_file`, `my_video_file`, `my_document_file`):
- **`""`** (empty string) → the surrounding content block is DROPPED. Use this when the user opts out of an optional slot.
- **A bare UUID** (36-char) → routed to `file_id`. The boundary normaliser later resolves it to bytes via cld_files.
- **An s3:// / gs:// / azure:// URI** → routed to `file_uri`.
- **One of our share-link URLs** (`https://share.matrx.com/r/...`, `/files/{id}/url`, `/files/{id}/download`, signed S3 URL) → routed to `url`. The boundary normaliser recognises our patterns and resolves to bytes.
- **An external URL** (`https://example.com/foo.png`) → routed to `url`. Provider fetches it directly, or the resolver downloads as a fallback.
- **Whitespace-only** → treated as empty (block dropped).
- **`null` / `undefined`** → also treated as empty (block dropped).

The FE doesn't need to classify the value before sending. **Python classifies via `coerce_to_media_ref` automatically.**

### What the FE should NOT do

- ❌ Don't try to wrap the value in a `{file_id, url, ...}` object. Just send the bare string.
- ❌ Don't send `null` for a value the user opted out of unless that's clearer than `""` — both are handled the same. (Sending an explicit empty string is slightly clearer.)
- ❌ Don't construct `MediaRef` shapes on the FE. That's Python's job.
- ❌ Don't strip or modify our share-link URLs. Send them verbatim. Our resolver knows them.

### What the FE SHOULD ensure

- ✓ When the user uploads via the resource picker, the picker's MediaRef-shaped output (a `{file_id, url, ...}` object) needs to be flattened to a single string for the wire. **Choose in this priority: `file_id` first (most efficient), then `url`, then `file_uri`.** That single string is what becomes the `variables[var_name]` value.
- ✓ Empty / unfilled optional variables: send `""` so Python knows to drop the corresponding block.
- ✓ Required text variables (per `variableDefinitions[].required: true`) should be validated before submit — Python doesn't enforce required-ness, the FE form should.

### Variable type detection (per Python's request)

> "When the variables reach python, they won't have a 'type' with them since we only get a list of names and values. Matching them to things other than text will need to come from a check we do to identify variables in those parts."

Python figures out the type **from where the variable is used in the message template**, not from the variable value or any tag. The flow:

1. The agent template declares e.g. `{"type": "image", "url": "{{my_start_image}}"}`.
2. When `UnifiedMessage.replace_variables(variables)` runs, it walks each content block.
3. For image / audio / video / document / youtube blocks, it calls the block's `replace_variables(variables)` method.
4. That method substitutes `{{my_start_image}}` into the block's `url` field.
5. The helper detects "this field carried a template" and runs `coerce_to_media_ref` on the result.
6. The MediaRef shape is set on the block.

So the variable's "type" comes from the block type that contains it. **The FE doesn't need to send a type per variable.** A single variable can also be referenced from multiple block types if needed (rare, but technically supported).

---

## 5. Per-translator handoff — honoring `metadata.role`

Translators that want to use the start/end/reference-frame information on user-supplied images need to read the `metadata.role` field on each `ImageContent` block.

> **Status update (just shipped):** the role-pattern is now wired across **every** image / video translator the package ships — see §5b for the canonical helper API and §5c for the per-translator audit. The "future work" section below has been retained for context but is **complete**.
>
> **The same pattern was extended to `negative_prompt`.** Putting `negative_prompt` in agent settings makes it engineer-locked — fine for some templates, wrong for most. The fix: the FE attaches it as a TextContent part with `metadata.role = "negative_prompt"`, exactly the same way image roles work. See §5d.

### Convention

When the FE attaches an image to a user message, the agent template (or a future inline-attach UI) sets `metadata.role` to one of:

- `"start_image"` — use as first frame for image-to-video, or primary input for image-to-image / image-edit
- `"end_image"` — use as last frame for first-to-last interpolation (also accepted: `"last_frame_image"`)
- `"reference"` — use as a reference image for character/scene continuity
- `"mask"` — use as inpainting mask
- `"input"` (or no role tag at all) — generic image input. Treated as `start_image` if no explicit `start_image` tag is present.

When the FE attaches text with `metadata.role = "negative_prompt"`, that text becomes the negative prompt for any provider that supports one — without occupying a settings slot.

### How a translator reads it

#### 5b. Canonical helper API (shipped)

Three helpers live in `packages/matrx-ai/matrx_ai/config/message_config.py`. Use these — don't reimplement the walk:

```python
from matrx_ai.config.message_config import (
    pick_text_by_role,    # role=None returns the FIRST UN-TAGGED user text (the main prompt)
    pick_image_by_role,   # most recent user message's first ImageContent matching role
    iter_images_by_role,  # generator yielding all matching ImageContent across user messages
)

# In any provider's image/video/edit translator:
prompt = pick_text_by_role(config.messages, None) or ""           # main prompt
neg    = pick_text_by_role(config.messages, "negative_prompt")    # None if absent
start  = pick_image_by_role(config.messages, "start_image")
end    = pick_image_by_role(config.messages, "end_image") \
      or pick_image_by_role(config.messages, "last_frame_image")  # accept either tag
mask   = pick_image_by_role(config.messages, "mask")
refs   = list(iter_images_by_role(config.messages, "reference"))
```

All three walk in reverse order (most recent user message first) and return as soon as they find a match — so role-tagged content from the latest turn always wins over older history.

#### 5c. Per-provider mapping (shipped)

Each translator falls back to its corresponding settings field if no role-tagged content is present in the user message — so casual usage and engineer-fixed presets both keep working.

```python
# Google Veo 3.1 — packages/matrx-ai/matrx_ai/providers/google/translator.py::to_google_video
neg   = pick_text_by_role(config.messages, "negative_prompt") or config.negative_prompt
start = pick_image_by_role(config.messages, "start_image") \
     or pick_image_by_role(config.messages, None) \
     or config.image_input or (config.frame_images or [None])[0]
end   = pick_image_by_role(config.messages, "end_image") \
     or pick_image_by_role(config.messages, "last_frame_image") \
     or config.last_frame_image
refs  = list(iter_images_by_role(config.messages, "reference")) or (config.reference_images or [])
```

The same pattern is used for OpenAI (gpt-image-2 edit + Sora video), xAI (image edit + video), Together (image + video), and every Replicate descriptor (FLUX 2, gpt-image-2 on Replicate, nano-banana, ideogram-v3, Veo 3.1, Kling v3, Seedance 2.0, Luma Ray 3, Hailuo 2.3, Wan 2.7, Runway Gen 4.5).

### Fallback rule

If a provider needs e.g. `start_image` but no image in the user message carries `metadata.role == "start_image"`, the translator should fall back to: **the first un-tagged image is treated as start_image.** This keeps casual usage working (a chat user just pastes one image and asks "animate this") without breaking the explicit-tag path for agent recipes.

### 5d. Negative prompt — text role pattern (shipped)

`negative_prompt` follows the exact same convention as image roles. The FE attaches it as a TextContent part on the user message:

```jsonc
{
  "role": "user",
  "content": [
    {"type": "text", "text": "{{image_description}}"},
    {"type": "text",
     "text": "{{negative_prompt}}",
     "metadata": {"role": "negative_prompt"}}
  ]
}
```

When the user leaves the negative prompt field empty:
- `TextContent.replace_variables` returns `True` for the role-tagged block (template was set, post-substitution text is empty/whitespace).
- `UnifiedMessage.replace_variables` honors that drop signal and removes the block entirely.
- The translator sees zero negative-prompt parts and falls back to whatever `config.negative_prompt` carries (engineer preset, if any).

When the user provides text:
- `pick_text_by_role(messages, "negative_prompt")` returns the resolved string.
- The translator wires it into the provider-specific kwargs (`negative_prompt=` for Google Veo, Together image+video, Ideogram v3, Replicate Veo 3.1).
- The main `pick_text_by_role(messages, None)` call ignores it — role-tagged TextContent never bleeds into the main prompt.

Providers that accept a negative prompt today: Google Veo 3.1, Together image (FLUX 2 dev only — others reject it), Together video (Veo 3.0 family + Kling + Seedance + Wan 2.7), Ideogram v3 on Replicate, Veo 3.1 on Replicate. Providers that ignore it: OpenAI (no negative_prompt support), xAI image+video (no negative_prompt support), most other Replicate models.

### Translators that need updating

~~This is FUTURE work~~ — **DONE** as of this commit. Every translator listed below now reads `metadata.role` first, settings second. Verified by `/tmp/smoke_role_pattern.py` which builds a UnifiedConfig containing all four roles + a negative-prompt text part and asserts each provider's exact kwargs dict.

| Translator | Roles read | Status |
|---|---|---|
| `GoogleTranslator.to_google_video` (Veo 3.1) | `start_image`, `end_image`, `reference`, `negative_prompt` | ✅ shipped |
| `OpenAITranslator.to_openai_image_edit` (gpt-image-2 family) | `start_image`, `reference`, `mask` | ✅ shipped |
| `OpenAITranslator.to_openai_video_generate` (Sora 2) | `start_image` | ✅ shipped |
| `XAIImageGeneration._build_kwargs` (grok-imagine-image) | `start_image`, `reference` | ✅ shipped |
| `XAIVideoGeneration._build_kwargs` (grok-imagine-video) | `start_image`, `reference` | ✅ shipped |
| `TogetherImageGeneration._build_kwargs` (FLUX/Imagen/Nano-Banana/Seedream/Ideogram/Qwen) | `start_image`, `reference`, `negative_prompt` | ✅ shipped |
| `TogetherVideoGeneration._build_kwargs` (Veo/Kling/Sora/Seedance/Wan) | `start_image`, `end_image`, `reference`, `negative_prompt` | ✅ shipped |
| Replicate descriptors (FLUX 2, gpt-image-2, Imagen 4, Nano-Banana, Ideogram v3, Recraft v4, Seedream 4.5, Veo 3.1, Runway Gen 4.5, Seedance 2.0, Kling v3, Wan 2.7, Luma Ray 3, Hailuo 2.3) | per-model: see `_start_image_url` / `_end_image_url` / `_reference_urls` / `_negative_prompt` helpers in `model_descriptors.py` | ✅ shipped |

The agent recipe is now the source of truth (via `metadata.role` on its template) instead of an agent-settings field that doesn't make sense to fix per-recipe.

---

## 6. Edge cases + open considerations

### 6a. Same variable, multiple references

If `{{my_image}}` appears in two image blocks (e.g. used as both start AND a reference), the substitution runs independently per block. Both end up with the same value — no conflict. But the `metadata.role` on each block is what the translator reads, so the same image can be used in two roles. Useful for "use this image as both the first frame AND a style reference" cases.

### 6b. The MIME type chain

Python's `coerce_to_media_ref` does NOT set `mime_type` from the bare string (it can't — the string is just a UUID or URL). The MIME type is filled in by:
- The boundary normaliser (`aidream/services/media_resolvers/request_normalizer.py`, entry point `normalize_request_body`) which calls `FileManager.resolve_media_async` and pulls the canonical MIME from `cld_files` for owned files.
- Otherwise, content-type inference at the provider call (each provider has different rules).

If the FE wants to pass a MIME hint (e.g. when the user uploaded a known file type), the wire format would need to grow from a bare string to a `{value, mime_type}` shape per variable. **Not implemented today** — propose it if needed.

### 6c. Required-but-empty handling

The substitution helper drops blocks when an optional template variable is empty. But what if the variable is REQUIRED (per `variableDefinitions[].required: true`)?

Today: Python doesn't know which variables were marked required. It just drops empty blocks.

Recommended: the FE should reject submission if a required text or required media variable is empty. Pass that responsibility to the form layer; backend doesn't need to know.

If you DO want backend enforcement, send the `variableDefinitions` along with the variables. Add a validator in `UnifiedConfig` or at the agent invocation entry point. Out of scope for this PR.

### 6d. Substitution into `system_instruction`

`UnifiedConfig.replace_variables` (line ~589 of unified_config.py) already substitutes `{{var}}` patterns into `system_instruction.base_instruction`. That continues to work — it's text-only, so no media coercion needed.

### 6e. Substitution into structured fields

The current substitution only walks message content. If an agent recipe wants `{{model_id}}` in `config.model` or `{{aspect_ratio}}` in `config.aspect_ratio`, those WOULDN'T be substituted today. Likely uncommon — settings are engineer-fixed, not user-overridden via templates. If needed later, extend `UnifiedConfig.replace_variables` to walk specific scalar fields.

### 6f. Setting overrides

The user mentioned: "For the settings overrides, we already have something in place. It's not ideal but it will work for now until I find a better solution." This PR does NOT touch settings overrides — the existing path is unchanged. The architectural recommendation in the previous handoff (parallel `user_inputs` schema with allowlist + per-key descriptors) stands as a future direction when ready.

### 6g. Persistence behavior

When a substituted message is persisted to `cx_message`, the metadata.role survives the round-trip (per the audit in §2). So if you reload an old conversation that had image-with-role parts, the role is preserved on read.

### 6h. No alternate ingress path

The substitution + coercion path uses ONLY existing infrastructure:
- Message parsing — existing `parse_content` + `_filter_kwargs`
- Media normalization — existing `MediaRef` + `FileManager.resolve_media_async`
- File system — canonical files service (Python-managed S3 bytes, Postgres metadata)

There is no second media path. Anything the user sends via a `{{var}}` flows through the same pipeline as a manually-attached media block.

---

## 7. Quick reference checklist

For the FE team:
- [ ] When sending agent variables, send media values as bare strings (UUID or URL).
- [ ] Send `""` for unfilled optional media variables.
- [ ] Validate `required: true` variables before submit.
- [ ] Read `customComponent.type` to render the right input widget per variable.
- [ ] Existing resource picker → flatten its output to a single string (`file_id` first, else `url`).

For per-translator authors (now ✅ done across the board, but the rules apply to any future translator):
- [x] Read `metadata.role` from user message ImageContent / TextContent blocks.
- [x] Use `pick_text_by_role`, `pick_image_by_role`, `iter_images_by_role` from `matrx_ai.config.message_config`.
- [x] Fall back: untagged image = `start_image`; tagged content always wins over settings; settings = engineer preset, never per-run input.
- [x] Stop reading `config.image_input` / `config.last_frame_image` / `config.negative_prompt` directly — read role first, fall back to settings only when the role is absent.

For the platform team (optional follow-ups):
- [ ] Decide if backend should enforce required-variable validation (vs FE-only).
- [ ] Decide if MIME-hint per variable is worth the wire shape change.
- [ ] Build the user-overridable settings (`user_inputs`) schema when ready.
- [ ] Build mask-drawing UI for in-app inpaint mask creation.
