# Podcast Generation Pipeline — Flow & Gate Contract

**This is the authoritative spec for `podcast_generator.py`.** Every run obeys it.
The core law: **`Content → Script → Audio`**, with optional acquisition steps in
front. Each arrow is a **gate** — you do not cross it until the prior step
produced output that passes a hard validation. A stage that fails its gate
**stops the run there** (or, for soft/optional stages, is skipped) — it NEVER
passes unvalidated output downstream. The bug this kills: a script agent's raw
thinking text reaching the TTS model because nothing checked that a real script
was produced.

---

## 1. The spine (assume every step, skip what isn't needed)

```
  ACQUIRE ──► RESEARCH ──► [CONTENT gate] ──► [SCRIPT gate] ──► [AUDIO gate] ──► episode
 (optional)  (optional)      ≥ N chars      valid <dialogue>    audio URL
                                            + speaker turns      present
```

Think of it as a fixed conveyor belt. Every job is placed on the belt at the
station matching its starting point, and rides to the end. Stations before its
entry point are skipped; stations after are mandatory.

- **Metadata + images + videos** run concurrently off `CONTENT`/metadata — they
  are best-effort and NEVER block or gate the `Content → Script → Audio` spine.
  A failed image is a soft failure (retry alternate model, then a notice).

---

## 2. Starting points → where they enter the belt

The frontend has 8 source tiles; they collapse to 4 backend `input_data_type`
values. The required acquisition differs per source:

| Source (UI) | `input_data_type` | Acquisition before CONTENT | Notes |
|---|---|---|---|
| Topic | `topic` | **Research agent** (web research → content) | The only path that researches. |
| Rough notes | `partial_content` | **none — pass through** | Already text. NO extractor, NO research. |
| Full script | `full_content` | **none — pass through** | If it's already a valid `<podcast_dialogue>` it SKIPS script generation (→ straight to audio). Otherwise it's scripted. |
| File (PDF/slides) | `file_url` | **Content-extractor agent** (multimodal file read) | The ONLY path that uses the extractor. |
| Website URL | `partial_content` | FE scrapes → text; backend passes through | FE must confirm scrape ≥ 2000 chars before sending. |
| Note | `full_content` | **none — pass through** → script | A note is content, not a finished script. |
| YouTube | `partial_content` | FE runs the YouTube extractor → transcript; pass through | Dedicated FE agent, not the file extractor. |
| Audio file | `partial_content` | FE transcribes → text; pass through | |

**Routing rule (backend `_prepare_content`):**
- `topic` → research agent → content.
- `file_url` → content-extractor agent → content. (multimodal: reads file bytes)
- `partial_content` → **pass through unchanged** as content. (rough notes,
  scraped/transcribed text are already content — the file extractor must never
  touch them; that is the routing bug being fixed.)
- `full_content` → **pass through**, then the SCRIPT stage decides: if it already
  contains a valid script block, skip generation; else generate.

---

## 3. The three gates (hard, non-negotiable)

### GATE 1 — CONTENT (`_prepare_content` result)
Output must be **≥ `MIN_CONTENT_CHARS` (1000)** of usable text.
- Below threshold → prep stage **FAILS**, run stops with an actionable error
  ("research/scrape/extraction returned too little to script from").
- `full_content` is exempt from the floor ONLY when it is already a valid
  script (caught at GATE 2) — a user's finished short script is legitimate.
- URL scrapes are additionally gated at the FE at **≥ 2000 chars** before they're
  even sent (a thin scrape is a failed scrape, not thin content).

### GATE 2 — SCRIPT (`_create_script` result, before audio launches)
**The contract is EXACT: what the UI promised, the script delivers, and that is
what reaches TTS.** No "close enough", no downstream name/count repair. The
output must pass ALL of:
1. Contains a usable `<podcast_dialogue>…</podcast_dialogue>` block (the LONGEST
   one wins, so a model that "thinks out loud" about the tag first is fine).
2. The block is **non-empty** after trimming.
3. Has **≥ 1** valid `Name: text` turn (the `_dialogue_speaker_lines` parser).
4. The number of **distinct speakers** is **exactly `host_count`** — 10 means 10,
   not 9; 2 means 2, not 1 or 3. A drift is rejected, not patched.
5. If the request named speakers, every named speaker appears as a label
   (case-sensitive).
6. If the agent appended `<speaker_settings>` (§4.1), its declared names match
   the dialogue's actual labels **exactly** — an agent that lies about its own
   script is rejected, never trusted.

- Any failure → script stage **FAILS** (`success=False`), **audio is never
  launched**, the run stops with a clear, resumable error. The raw output is
  logged for debugging but NEVER forwarded.
- `_extract_dialogue` is the enforcement point: it returns the validated block
  or RAISES — it **never** falls back to returning the raw script.
- **Reliability note:** an exact-count gate means a large cast (say 10) that the
  agent under/over-fills will FAIL rather than degrade. That is intentional —
  but it makes the agent prompt's "produce EXACTLY N speakers" instruction
  load-bearing, and `<speaker_settings>` (§4.1) the recommended robustness aid.

### GATE 3 — AUDIO (`_audio_stage_result`, already enforced)
A "successful" audio stage with **no audio URL** is rewritten to FAILED
(prevents a completed-empty checkpoint from poisoning resume). Keep as-is.

---

## 4. Script-agent selection (the matrix that scales)

The script writer is chosen from three dimensions: **host_count × format ×
language**. The principle: **use the most specific custom agent that matches all
constraints; otherwise fall back to a correct generic agent.** As more custom
agents are built, more combinations get a specialized route; the generic path is
the safety net that must always work.

**Selection order (first match wins):**

1. **Legacy proven custom (2-host only):** `host_count == 2`, no custom
   speakers/theme, format ∈ {educational, news}, language English (or the
   Persian path). These are the hand-tuned 2-speaker agents — best quality.
   - educational → `_EducationalScriptAgent`
   - news → `_NewsScriptAgent`
   - persian → `_PersianScriptAgent`
   (Hardcoded for exactly 2 speakers — never used for 1 or 3+.)

2. **Host-count-aware generics** (everything else):
   - `host_count == 1` → `podcast_solo_script` (monologue/narration).
   - `host_count 2–4` → `podcast_multihost_script` (format/language/speaker aware).
   - `host_count 5–10` → `podcast_roundtable_script` (moderated large cast).

3. **Future custom format/flexible agents** slot in ABOVE the generics: e.g. a
   future `podcast_entertainment_script` that handles entertainment across 2–6
   speakers would be selected for `format=entertainment, host_count∈2..6` before
   falling to the generic multihost. Registration is a single table entry (see
   `SCRIPT_AGENT_REGISTRY` once added) — keyed by `(format, language?, host_min,
   host_max)`, picked by specificity. The generic remains the final fallback.

**Every script agent — custom or generic — MUST emit the
`<podcast_dialogue>Name: line</podcast_dialogue>` contract** AND be told to
produce **exactly `num_speakers` distinct speakers** (GATE 2 enforces the exact
count). A misbehaving custom agent fails loudly instead of poisoning audio.

**Language:** handled by the agent (`language` variable on the generics) or by a
language-specific custom agent (e.g. Persian news). No language ever silently
falls back to English script text — if no agent can write the language, GATE 2's
output will be wrong-language but still validated structurally; quality issues
surface in review, not as a crash.

### 4.1 `<speaker_settings>` — the agent-declared cast (recommended)

The most robust way to keep UI ↔ script ↔ TTS in lock-step is to have the script
agent **declare its cast**, so the pipeline never guesses. After the dialogue
block, the agent MAY append (the pipeline reads it if present, ignores it if not):

```
<speaker_settings>{"speakers":[{"name":"Alex","voice":"orus"},{"name":"Sarah","voice":"kore"}]}</speaker_settings>
```

- `name` is required; `voice` is optional. The string form `["Alex","Sarah"]`
  also works.
- **The pipeline parses it (`_parse_speaker_settings`) and prefers it for the
  voice config.** GATE 2 cross-checks the declared names against the dialogue's
  actual labels — a mismatch fails the script (the agent must not lie about its
  own script).
- **To let the agent assign voices**, pass it a voice list in its prompt (the
  curated ElevenLabs voices for 3+, the Gemini palette for ≤2) and ask it to fill
  `voice`. Without it, the pipeline assigns from the default palette in order.
- OPTIONAL at the pipeline layer (no `<speaker_settings>` → the cast derives
  from the dialogue's own labels, which already works) — but **all three generic
  script agents (solo / multihost / roundtable) REQUIRE it in their prompts as
  of 2026-08-08** (name + gender, never voice; the server owns voice selection),
  and their prompts carry a roster-first + post-write count-check protocol so
  large casts (7–20) hit the exact `host_count`. The declaration doubles as the
  agent's own final self-check against GATE 2's exact-count rule.

**To add it to a script agent (their job):** append to the agent's instructions —
*"After the `</podcast_dialogue>` close tag, output a single `<speaker_settings>`
block: a JSON object `{"speakers":[…]}` listing every speaker by the EXACT name
used in the dialogue, in first-appearance order. [If a voice list is provided:]
assign each speaker one `voice` from this list: …"*

---

## 5. Audio-provider selection (host_count → provider)

| host_count | Provider | Agent | Voice config |
|---|---|---|---|
| 1 | Google Gemini (single voice) | `_AudioEnglishAgent`/`_AudioPersianAgent` | `tts_voice: "<voice>"`; labels stripped (one voice shouldn't read "Alex:") |
| 2 | Google Gemini (multi-speaker) | same | `tts_voice: [{name,voice}]`; Google caps at 2 |
| 3–10 | ElevenLabs `text_to_dialogue` | `_AudioDialogueAgent` (`eleven_v3`) | `tts_voice: [{text,voice_id}]` built from the script turns. **This provider's max-10-DISTINCT-voices-per-request limit (`max_voices_exceeded`, verified live 2026-08-08 at 14 and 20 hosts) IS why `host_count` stops at 10** — every speaker gets their own voice, so the largest renderable cast equals the cap. |

**The cast is resolved ONCE, at our layer, from what the script actually says**
(`_effective_speakers`): name priority `request.speakers` → `<speaker_settings>`
→ the dialogue's own labels; voice priority request → declared → default palette.
The config we hand TTS therefore carries **only names the transcript contains** —
the provider/translator never receives a mismatched name and never has to "fix"
anything. This is the fix for the production `Speaker name mismatch: ['Alex',
'Sarah'] not found` error: the legacy 2-host path used to fall back to the audio
agent's *pinned* `Alex/Sarah` names; now the 2-host Google config is **always**
built from the script's real speakers.

- **Speaker→voice mapping** (`_dialogue_to_elevenlabs_turns`) raises BEFORE the
  paid call if any script label has no voice — a missing voice is a loud failure,
  never a silent drop.
- The Google translator's `align_speaker_labels` / `validate_speaker_names` are
  now a redundant safety net, not load-bearing: by the time the config reaches
  them the names already match (we never rely on the API to align them).

---

## 6. What every request setting does

| Field | Effect |
|---|---|
| `input_data_type` | Picks the acquisition station (§2). |
| `input_data` / `file_urls` | The raw input for that station. |
| `host_count` (1–10) | Picks the script generic band AND the audio provider (§4, §5). **10 is a hard ceiling** — see §5. |
| `format` | Picks/shapes the script agent; passed as a variable to generics. |
| `language` | Script language; Persian also selects the Persian custom path. |
| `theme` | Optional stance/framing passed to the generics; forces off the legacy path. |
| `speakers[]` (name, voice) | Optional. Names feed the script agent + are enforced by GATE 2; voices feed TTS. Empty → defaults from the palette. |
| `podcast_type` | Legacy discriminator (educational/news/persian); derives format/language when those are empty. |
| `post_prep_option` | Optional transform between content and script: `translation` (into the run's `language`) / `summarization` / `expansion` / `fact_checking`, each a DB-managed `podcast.post_prep_*` Mandate run via `_apply_post_prep`. SOFT stage (§7): any failure — including a degenerate too-short output — keeps the original content. |
| `audio_style` | TTS delivery style passed to the Gemini audio agents. |
| `max_images` / `max_videos` | Per-run media caps (0 = skip) — cost control, never gates the spine. |
| `truncate_audio_for_testing` | Trims the validated dialogue to one turn/speaker before TTS — cheap testing only. |

---

## 7. Failure semantics (fatal vs soft)

- **Fatal (stop the run):** CONTENT gate fail, SCRIPT gate fail, AUDIO produced
  no URL. These are the spine — without them there is no episode. All are
  **resumable** (the checkpoint re-runs only the failed stage).
- **Soft (carry on, notice):** any single image/video (content-policy reject →
  retry alternate model → small notice), post-prep, official-video compose,
  episode-row persistence. Never block the spine.

---

## 8. Invariants (the discipline, enforced in code — not by hope)

1. **No audio without a GATE-2-valid script.** Enforced in `_create_script`
   (validates before returning success) AND `generate_podcast` (only launches
   audio on `script_result.success`).
2. **No script from sub-threshold content.** Enforced by GATE 1.
3. **`partial_content` never touches the file extractor.**
4. **`_extract_dialogue` never returns the raw script** — block or failure.
5. **Every script agent emits the `<podcast_dialogue>` contract**; a custom
   agent that doesn't fails GATE 2 loudly.
6. **Most-specific custom agent wins; the generic is the always-works fallback.**
7. **Every generated asset is BORN PUBLIC.** Image, video, and feature-image
   generation run inside `matrx_ai.media.public_media_scope()`, so each asset is
   persisted with a permanent CDN URL — never an expiring signed S3 URL that a
   stage checkpoint, an `ASSET` stream event, or the client's `pc_studio_runs`
   row would then store and serve to anonymous readers. Audio is born public via
   `BORN_PUBLIC_FEATURES`; the composed official video passes
   `visibility="public"`. Locked by
   `tests/test_media_visibility_policy.py::test_podcast_*_slots_persist_inside_the_public_scope`.
   **Persisting private and flipping public afterwards is NOT equivalent** — the
   flip moves the S3 object, so any URL already emitted is dead at once.

---

*Change log:*
- 2026-08-09 — **`host_count` capped at 10** (`_MAX_SPEAKER_COUNT`). Live
  10/14/20-host runs proved the 10-distinct-voice-per-request ElevenLabs limit
  is a hard wall; Arman's ruling was to cap rather than build multi-request
  render + stitch. The interim voice-SHARING workaround was deleted with it —
  every speaker gets a distinct voice again, unconditionally
  (`_enforce_distinct_voices`), and the roundtable agent + slot advertise 5–10.
- 2026-08-11 — **born-public media** (invariant 7): podcast covers/videos were
  persisted `personal`, so the signed URL reached the client and
  `podcast.pc_studio_runs.image_urls`/`video_urls`; the end-of-run flip to public
  then killed those stored URLs outright. Four runs were found holding dead URLs.
  Fixed at the source with the new `public_media_scope()` primitive. Backstops +
  the healer: `aidream/services/podcast/FEATURE.md`.
- 2026-08-08 (later) — **Canonical script persistence:** GATE 2 pass now emits
  `_canonical_script(output)` (dialogue block + optional `<speaker_settings>`,
  never the agent's raw output), so `pc_episodes.script` and every downstream
  consumer can't inherit reasoning prose; existing rows backfilled. **Audience
  adaptation** stage added (`target_audience` on the request → soft
  `audience_adaptation` stage via the `podcast.audience_adapter` slot, before
  `post_prep_option`). **Suggested default cast:** a request naming no speakers
  now hands the script agent a fresh rotated draw from the default cast instead
  of `[]` (kills LLM name mode-collapse on API-path runs). **10-voice audio
  cap** (see §5) — verified 10-host e2e pass; 14/20 rejected by ElevenLabs.
- 2026-08-08 — post-prep went live: all four `post_prep_option` transforms now
  run dedicated mandated agents in `_apply_post_prep` (translation targets the
  run's derived language; every failure is soft and keeps the original
  content, with a scaled minimum-output guard against degenerate transforms).
- 2026-06-12 — created after a run leaked script-agent thinking text into Google
  TTS (no script gate existed). Documents the gate contract being implemented in
  `podcast_generator.py`.
- 2026-08-08 — large-cast hardening: the three generic script agents were
  updated (and their `podcast.*_script` Mandates rebound) to REQUIRE the
  `<speaker_settings>` declaration and follow a cast-count verification
  protocol; the roundtable user message's "Output only the dialogue block"
  line (which contradicted and suppressed the declaration) was removed. E2E
  matrix gained 10/14/20-host scenarios.
- 2026-06-16 — official-video compose stage now surfaces its failure/skip reason
  to the FE via `official_video_error` (on `PodcastGenerationResult` and the
  router's `PodcastCompleteEvent`) and logs loudly (green on success, red on
  fail, yellow on skip) instead of failing silently.
