# AI Matrx Streaming Constitution

### The Universal Rules for Python-to-React Block Streaming

*Version 2.0 — This document governs how every AI-generated block is parsed, emitted, and consumed.*

---

## Scope

This constitution governs **structured blocks** — timelines, quizzes, flashcards, diagrams, etc.

It does **NOT** govern plain text. Text streams character-by-character as tokens arrive from the LLM. That is the industry standard. Users expect it. We do not buffer text.

The only exception: when accumulating text looks like it *might* become a block (e.g., the buffer ends with `<timeli`), the processor holds emission briefly until it can determine whether a block tag is forming. See **Part III — The Detection Buffer**.

---

## Part I — First Principles

### Principle 1: The Renderable Unit

Every block type has a **Renderable Unit** — the smallest piece of that block that can be displayed without looking broken.

| Block type | Renderable unit |
|---|---|
| timeline | One complete item (title + date + category) |
| flashcards | One card face (front alone is OK; back can show loader) |
| quiz | One complete question (text + all options + answer + explanation) |
| presentation | One complete slide |
| diagram / orgchart | One complete node OR one complete edge |
| math_problem | The problem statement; then each solution step |
| research | One complete `##` section |
| resources | One complete resource link |
| progress_tracker | One complete task item |
| troubleshooting | One complete issue section (symptom + causes + solutions) |
| comparison | One complete criterion row |
| cooking_recipe | Title + ingredients OR title + instructions (whichever comes first) |
| table | One complete row |
| questionnaire | One complete question (with all its options) |
| transcript | One complete speaker turn |
| tasks | One complete task item |

The parser **MUST NOT** emit a partial unit. A quiz question with 2 of 4 options, a diagram node with no label, an item with a half-written title — these are worse than nothing. They cause React to render broken UI.

**The Loader Exception:** If a unit has two halves and the first half is useful on its own, emit the first half immediately. The second half renders as a loading state — NOT an absence. React must handle `{ front: "What is X?", back: null }` with a spinner on the back.

### Principle 2: Forward Triggering

The parser almost never knows that item N is complete when it finishes parsing item N. It knows item N is complete when:

- **Item N+1 begins** (the next delimiter arrived)
- **The block's closing tag arrives** (last item in the block)
- **An unrecoverable error occurs** (flush whatever is buffered)
- **The stream ends** (finalize flushes all pending items)

This means the buffer always holds **one item in progress**. When the trigger fires, the buffer is flushed as a completed unit and the new item takes its place.

### Principle 3: Growing Snapshot

Each emission for a structured block is a **complete, valid data object** — not a delta, not a patch. When the timeline parser emits after detecting a new item, it sends the entire `TimelineBlockData` with all items seen so far. React replaces its state wholesale.

This means:
- React never needs to accumulate deltas or track ordering
- Reconnection is trivial — resend the latest snapshot
- No bugs from out-of-order updates
- Each emission is independently renderable

The data just **grows richer** with each emission. Same shape, more content.

### Principle 4: Errors are Seals, Not Voids

If anything goes wrong — a timeout, malformed LLM output, a Python exception — the parser **MUST** emit a final event with `status: "error"` and whatever partial data it has. React must never be left with an open, unsealed block. An open block is a spinner that never resolves.

### Principle 5: React Owns Display, Python Owns Semantics

Python decides **what** to send and **when**. React decides **how** to display it.

Python emits structured data at semantic boundaries. React renders that data. Python never sends raw markdown fragments as the payload for structured blocks. If the block type has defined structure, emit that structure.

---

## Part II — The Two Channels

All LLM output flows through one of two channels. The processor determines which channel applies on every token.

### Text Channel

**Behavior:** Stream tokens directly as they arrive. Every `process_token()` call emits a `content_block` event with the growing `content` string.

**Applies to:** `text`, `thinking`, `reasoning`, `info`, `task`, `database`, `private`, `plan`, `event`, `tool`, `structured_info`, `code`

**Why:** These are free-form content. Users expect to see characters appear in real time. Buffering text is perceived as lag.

### Block Channel

**Behavior:** Buffer tokens. Run the semantic parser on each token. Emit a `content_block` event **only** when a new renderable unit is sealed.

**Applies to:** Everything else — `timeline`, `flashcards`, `quiz`, `presentation`, `diagram`, `math_problem`, `research`, `resources`, `progress_tracker`, `troubleshooting`, `comparison`, `cooking_recipe`, `table`, `questionnaire`, `transcript`, `tasks`

**Why:** Sending `"### Proj"` then `"### Project T"` then `"### Project Ti"` for a timeline header is noise. The UI can't render any of those fragments. Wait until the header line is complete, then emit.

### The Decision Point

When the first tokens of a new block arrive, they are **always text** — until the processor detects a block opening (XML tag, code fence with special language, JSON root key). At that moment, the block switches from Text Channel to Block Channel and all subsequent tokens for that block are governed by Block Channel rules.

---

## Part III — The Detection Buffer

### The Problem

LLM tokens arrive a few characters at a time. When the buffer ends with `<timeli`, we don't yet know if this will become `<timeline>` (a block) or `<timelier approach>` (plain text). If we emit it as text immediately, React renders it as text. Then when the full tag arrives, we'd have to replace that text block with a timeline block — causing a visible flicker.

### The Solution: Tag Speculation

When the buffer ends with what looks like the start of an XML tag — specifically, `<` followed by one or more word characters with no closing `>`, no spaces, and no newline — the processor **suppresses emission** and waits for more tokens.

**Detection rule:**
```
Buffer tail matches: <[a-zA-Z_]\w*$   (no > yet, no space, no newline)
```

**Resolution:**
- The tag closes with `>` → check if it's a known block tag. If yes, switch to Block Channel. If no, it's just text — resume Text Channel emission.
- A space or newline appears before `>` → not a self-contained tag. Emit as text.
- The stream ends → emit as text.
- The 1-second silence rule fires (Part VI) → emit as text.

**Maximum hold:** The detection buffer holds at most **one tag's worth of characters** (typically < 30 characters). This is not a general-purpose buffering mechanism.

### Pipe Speculation (tables)

Tables get the same treatment as tags. A trailing line that starts with `|` and has no newline yet cannot be classified — with one pipe it's text, with two it's a table row, and the detector flips between those on every token. Emitting during that window created a speculative text block per table row (`"| One"`) that the next re-split absorbed into the table, leaving a phantom block the client had already been told about. The processor suppresses emission while the buffer tail (after the last newline) starts with `|`, and resumes on the row's newline.

Two companion invariants in the reconciler back this up (2026-07-16):
- **Retraction, not truncation:** any tracked block absent from a later re-split is emitted one final time as an empty `complete` event before being dropped — the client store replaces blocks wholesale on upsert, so silent truncation left phantoms on screen forever.
- **The re-split is the truth:** a still-streaming block whose detected type disagrees with the type it was created under re-types in place (stable blockId), dropping parsed state. Content must never end up frozen under a stale type (a markdown table living inside a `code` block).

### What This Does NOT Apply To

- Code fences (` ``` `) — already handled by the code block detector.
- JSON blocks — detected via code fence language, not raw `<` tags.
- Partial content inside an already-detected block — once we're inside a `<timeline>`, all content is already on the Block Channel.

---

## Part IV — The Emission Model

### Event Format

Every emission is a `content_block` event. Same envelope, always:

```json
{
  "event": "content_block",
  "data": {
    "blockId": "blk_0",
    "blockIndex": 0,
    "type": "timeline",
    "status": "streaming",
    "content": "raw markdown (for text-like blocks)",
    "data": { "growing structured snapshot (for block-channel blocks)" },
    "metadata": {}
  }
}
```

### When to Emit (Block Channel)

A `content_block` event is emitted when ANY of these occur:

1. **Block opens** — The opening tag/delimiter is detected. Emit with `status: "streaming"`, `data: null` (or minimal shell). React mounts the component.

2. **Header completes** — Title and/or description are parsed. Emit with growing snapshot containing the header.

3. **Renderable unit seals** — A complete item/question/slide/node/card is confirmed (via forward triggering). Emit with the growing snapshot containing all items so far.

4. **Group/phase opens** — A new container within the block starts (e.g., `**Phase 2:**`). Emit so React can render the empty group.

5. **Block closes** — Closing tag detected. Flush any buffered item. Emit with `status: "complete"` and final snapshot.

6. **Error occurs** — Flush buffer. Emit with `status: "error"` and whatever data exists.

### When NOT to Emit

- Token arrived but no new renderable unit was sealed → no emission
- Buffer grew but we're mid-item → no emission
- Partial tag is forming (detection buffer active) → no emission

### The Growing Snapshot Pattern

For blocks using semantic streaming, every emission sends the **full current state** via the `data` field:

```
Emission 1: { title: "Timeline", periods: [] }
Emission 2: { title: "Timeline", periods: [{ period: "Phase 1", events: [] }] }
Emission 3: { title: "Timeline", periods: [{ period: "Phase 1", events: [item1] }] }
Emission 4: { title: "Timeline", periods: [{ period: "Phase 1", events: [item1, item2] }] }
Emission 5: { title: "Timeline", periods: [{ period: "Phase 1", events: [item1, item2] }, { period: "Phase 2", events: [] }] }
```

React always receives a valid, self-contained object. Replace state, re-render. No delta accumulation.

---

## Part V — Streaming Behavior Categories

Every block type falls into exactly one category. The category determines how the processor handles it.

### Category 1: INCREMENTAL (Text Channel)

**Types:** `text`, `thinking`, `reasoning`, `info`, `task`, `database`, `private`, `plan`, `event`, `tool`, `structured_info`

**Behavior:** `content` field grows with every token. Emitted on every `process_token()` call that changes content.

**Parser:** None. Content is the payload.

### Category 2: SEMANTIC_STREAM (Block Channel — Growing Snapshot)

**Types:** `timeline`

**Behavior:** A dedicated stateful parser (e.g., `TimelineStreamParser`) tracks semantic boundaries. Emits a full growing snapshot only when a renderable unit seals. Between emissions, tokens are silently buffered.

**Parser:** Stateful stream parser with `feed(content)` → snapshot or None, `finalize()` → final snapshot.

**This is the gold standard.** Every block type that benefits from progressive rendering should eventually have a semantic stream parser.

### Category 3: PARTIAL_UPDATES (Block Channel — Parser on Every Change)

**Types:** `flashcards`, `cooking_recipe`, `questionnaire`, `table`, `tasks`, `transcript`

**Behavior:** The parser runs on every content change. It returns whatever complete items it can extract from the partial content. Items that are incomplete are either omitted or flagged as partial.

**Parser:** Stateless parser that takes full raw content → structured data with `isComplete` flags.

**Trade-off:** Simpler than a stateful stream parser but runs the full parse on every token. Acceptable for blocks where the parser is fast (flashcards, tables).

### Category 4: MARKDOWN_COMPLETE (Block Channel — Parse on Close)

**Types:** `research`, `resources`, `progress_tracker`, `troubleshooting`

**Behavior:** Raw `content` is streamed (so React can show a raw markdown preview) but `data` is only populated when the closing tag arrives. The parser needs the full content to produce useful output.

**Parser:** Stateless parser that takes complete content → structured data.

**When to upgrade:** If a MARKDOWN_COMPLETE block would benefit from progressive rendering (e.g., research sections appearing one by one), build a semantic stream parser and move it to SEMANTIC_STREAM.

### Category 5: COMPLETE_ONLY (Block Channel — Parse on Close)

**Types:** `quiz`, `presentation`, `decision_tree`, `comparison_table`, `diagram`, `math_problem`

**Behavior:** JSON-based blocks detected via code fence. Raw content is streamed as a preview. Parser runs only when JSON is complete and valid. On success, `data` contains the parsed structure and `content` is replaced with the canonical JSON.

**Parser:** Stateless parser that takes complete JSON string → structured data.

**When to upgrade:** If a COMPLETE_ONLY block would benefit from progressive rendering (e.g., quiz questions appearing one by one), build a semantic stream parser for it.

### Category 6: CODE

**Types:** `code`

**Behavior:** Content streams incrementally. `data` field includes `language`, `code`, `isDiff` once the language identifier is stable (i.e., past the opening fence line).

### Category 7: MEDIA

**Types:** `image`, `video`

**Behavior:** Detected on a single line. `data` contains `src` and `alt`. Emitted immediately.

---

## Part VI — Timing Rules

### The One-Second Rule

If a block is open (`status: "streaming"`) and the processor has not emitted any event for that block in **1 second**, the infrastructure layer should trigger a keepalive or progress signal. This prevents the user from staring at a frozen UI.

**Implementation note:** The `StreamBlockProcessor` is synchronous and has no concept of time. The 1-second rule is enforced by the async streaming infrastructure that wraps it (e.g., the heartbeat mechanism in `StreamEmitter`). The processor's job is to emit when semantically appropriate; the infrastructure's job is to ensure the stream doesn't go silent.

### No Artificial Delays

Never add sleep/delay to "batch" emissions. If two renderable units seal on the same token (e.g., a timeline item is flushed and a new phase opens simultaneously), emit both events immediately. Let the network and React handle the pacing.

---

## Part VII — Error & Flush Protocol

### The Flush Invariant

> At any point where parsing is interrupted, **the content in the buffer is worth more than silence**.

When a flush is triggered:

1. **Emit the buffer** — if the semantic parser has pending data, finalize it and emit the growing snapshot.
2. **Emit a seal** — `status: "error"` with `metadata.reason` describing what happened.
3. **Do not throw** — the flush must complete even during exception handling.

### Flush Triggers

| Trigger | Action |
|---|---|
| `</closing_tag>` detected | Normal completion. Flush pending item. Emit with `status: "complete"`. |
| Python exception in parser | Flush pending data. Emit with `status: "error"`, `metadata.parseError: true`. |
| LLM stream ends unexpectedly | `finalize()` closes all open blocks. Flush pending items. Mark complete. |
| Malformed structure | Parser returns `None`. Record `metadata.parseFailReason`. Emit with raw content preserved. |

### Never Leave a Block Open

Every block that receives an emission with `status: "streaming"` **MUST** eventually receive an emission with `status: "complete"` or `status: "error"`. The `finalize()` method on `StreamBlockProcessor` guarantees this by closing all open blocks.

---

## Part VIII — Block Registry: Per-Type Specifications

For each block type, this section defines the renderable unit, the delimiter that triggers emission, whether it has groups, and an example of the growing snapshot.

---

### `timeline`

**Category:** SEMANTIC_STREAM
**Renderable unit:** One complete item (title + date + category)
**Delimiter:** Next `- ` bullet, next `**Phase**` header, or `</timeline>`
**Has groups:** Yes (phases)
**Parser:** `TimelineStreamParser` (stateful)

Growing snapshot progression:
```
1. { title: "Project Timeline", description: "Overview", periods: [] }
2. { title: "...", periods: [{ period: "Phase 1", events: [] }] }
3. { title: "...", periods: [{ period: "Phase 1", events: [{ title: "Research", date: "Jan-Feb", category: "Planning" }] }] }
4. { title: "...", periods: [{ period: "Phase 1", events: [item1, item2] }] }
5. { title: "...", periods: [phase1_complete, { period: "Phase 2", events: [] }] }
...
```

---

### `flashcards`

**Category:** PARTIAL_UPDATES
**Renderable unit:** One card face (front or back)
**Delimiter:** `---` separator, `Back:` marker, next `Front:` marker, or `</flashcards>`
**Has groups:** No
**Loader Exception:** Yes — emit front when back starts streaming. React shows spinner on back.

---

### `quiz`

**Category:** COMPLETE_ONLY (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One complete question (question + all options + correct answer + explanation)
**Delimiter:** Next question object `{"id":` or closing `]`
**Has groups:** No
**Notes:** A question is NOT emittable until ALL options are present.

---

### `presentation`

**Category:** COMPLETE_ONLY (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One complete slide
**Delimiter:** Next slide object or closing `]`
**Has groups:** No

---

### `diagram` / `orgchart`

**Category:** COMPLETE_ONLY (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One node or one edge
**Delimiter:** Next node/edge object or closing `}`
**Has groups:** No
**Critical rule:** ALL nodes must be emitted before ANY edges. An edge referencing an unknown node breaks the render.

---

### `math_problem`

**Category:** COMPLETE_ONLY (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** Problem statement (first); then each solution step
**Delimiter:** Detection of step content; next step object
**Has groups:** Yes (solution approaches)

---

### `research`

**Category:** MARKDOWN_COMPLETE (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One complete `##` section
**Delimiter:** Next `## ` heading or `</research>`
**Has groups:** Yes (top-level sections)
**Notes:** Do NOT stream line-by-line. Each section is a meaningful unit.

---

### `resources`

**Category:** MARKDOWN_COMPLETE (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One complete resource link
**Delimiter:** Next `- [` item, next `**Category**` heading, or `</resources>`
**Has groups:** Yes (categories: Documentation, Tools, Videos, Articles)

---

### `progress_tracker`

**Category:** MARKDOWN_COMPLETE (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One complete task item
**Delimiter:** Next `- [` marker, next `**Category**` header, or `</progress_tracker>`
**Has groups:** Yes (categories)

---

### `troubleshooting`

**Category:** MARKDOWN_COMPLETE (upgrade candidate for SEMANTIC_STREAM)
**Renderable unit:** One complete issue section (symptom + causes + solutions)
**Delimiter:** Next `### ` heading or `</troubleshooting>`
**Has groups:** No

---

### `comparison`

**Category:** COMPLETE_ONLY
**Renderable unit:** One criterion row
**Delimiter:** Next criterion object or closing `}`
**Has groups:** No

---

### `cooking_recipe`

**Category:** PARTIAL_UPDATES
**Renderable unit:** Title + ingredients block OR title + instructions block
**Delimiter:** Section markers within the recipe
**Has groups:** No

---

### `questionnaire`

**Category:** PARTIAL_UPDATES
**Renderable unit:** One complete question with all its options
**Delimiter:** `---` separator or `</questionnaire>`
**Has groups:** No

---

### `table`

**Category:** PARTIAL_UPDATES
**Renderable unit:** One complete row
**Delimiter:** Newline (each line is a row)
**Has groups:** No

---

### `transcript`

**Category:** PARTIAL_UPDATES
**Renderable unit:** One complete speaker turn
**Delimiter:** Next speaker marker or end of content
**Has groups:** No

---

### `tasks`

**Category:** PARTIAL_UPDATES
**Renderable unit:** One complete task item
**Delimiter:** Next task marker or end of content
**Has groups:** No

---

### `thinking` / `reasoning`

**Category:** INCREMENTAL (Text Channel)
**Notes:** Despite being inside XML tags (`<thinking>`, `<reasoning>`), the content is free-form prose. Stream it like text — token by token. The XML tags are just markers for detection, not structural delimiters within the content.

---

## Part IX — Adding New Block Types

When a developer introduces a new block type, they MUST complete this checklist **before** writing parser code:

```
Block type name:                    _______________
Opening tag/delimiter:              _______________
Closing tag/delimiter:              _______________
Streaming category:                 INCREMENTAL / SEMANTIC_STREAM / PARTIAL_UPDATES /
                                    MARKDOWN_COMPLETE / COMPLETE_ONLY
Renderable unit:                    _______________
  (The smallest piece that makes visual sense alone)
Unit delimiter:                     _______________
  (What confirms unit N is complete)
Has groups?                         Yes / No
  If yes — group delimiter:         _______________
Loader Exception applies?           Yes / No
  If yes — part A (emit immediately):   _______________
  If yes — part B (can show loader):    _______________
Data snapshot shape:
  { ... }
```

This definition must be reviewed and registered in `BlockRegistry` before the parser is merged. The type must also be added to this document's Part VIII.

---

## Part X — React Contract

### What Python Guarantees

1. **Every emission is a valid, self-contained snapshot.** React can render any single `content_block` event without needing prior events for context (except the initial `open` event to mount the component).

2. **`data` is always a dict or null, never a string.** The `content` field holds raw text. The `data` field holds parsed structure.

3. **`status` transitions are monotonic:** `streaming` → `complete` or `streaming` → `error`. A block never goes from `complete` back to `streaming`.

4. **Every block that opens will close.** `finalize()` guarantees it.

5. **Atoms within `data` are complete.** A quiz question in `data.questions` has all options. A timeline event in `data.periods[n].events` has a title. React does not need to validate completeness of items within the `data` payload.

### What React Must Handle

1. **Null `data` during streaming.** A block may emit several events with `data: null` before the first renderable unit seals. React should show a loading skeleton.

2. **Growing `data` snapshots.** The `data` object grows over time. React should diff against its current state and render additions. Do not re-mount the entire component on each update.

3. **`status: "error"` with partial `data`.** Render what exists. Show a subtle error indicator. Do not discard everything.

4. **The Loader Exception.** For flashcard backs and similar part-B patterns, render a loading placeholder when part-B is null — not an error, not empty space.

5. **Blocks with no groups.** Not every block has phases/sections/categories. Render items flat when there are no groups.

---

## Appendix A — Anti-Patterns

**Character streaming for structured blocks.**
Sending `"t"`, `"ti"`, `"tim"` for a timeline title. The UI can't render any of those fragments. Use semantic boundaries.

**Emitting a partial renderable unit.**
A quiz question with 2 of 4 options. A diagram node with no label. A timeline item with a half-written title. These are worse than nothing.

**Waiting for the entire block before emitting anything.**
The whole point of this architecture is that we don't do this. A timeline with 30 items should stream ~35 events, not 1.

**Emitting edges before all nodes (diagrams).**
React will try to render connections to nodes that don't exist yet.

**Leaving a block unsealed on error.**
Always seal. Always. `finalize()` does this automatically; manual parser code must do the same.

**Running expensive parsers on every single token.**
If the parser for a block type is slow, it should be SEMANTIC_STREAM (stateful, emits at boundaries) not PARTIAL_UPDATES (runs full parse every time).

**Mixing content and data fields.**
`content` holds raw markdown/text for preview rendering. `data` holds parsed structure for component rendering. Don't put markdown in `data`. Don't put structured JSON in `content` (except as a serialization of `data` for COMPLETE_ONLY blocks after parsing succeeds).

**Buffering text.**
Plain text goes through the Text Channel. Stream it immediately. Don't hold text tokens waiting for "a good time" to emit. The only text hold is the detection buffer for potential XML tags.

---

## Appendix B — Quick Reference

```
TEXT:
  → Stream every token. No buffering. No parsing.
  → Exception: hold briefly if buffer ends with potential XML tag opening.

STRUCTURED BLOCK:
  1. Detect opening tag          → emit content_block (status: streaming, data: null)
  2. Parse title/description     → emit content_block (status: streaming, data: {header})
  3. Detect group delimiter      → emit content_block (data grows: new empty group)
  4. Buffer one renderable unit
  5. Detect next delimiter       → emit content_block (data grows: buffered unit added)
  6. Repeat 3–5
  7. Detect closing tag          → emit content_block (status: complete, data: final)

ON ERROR AT ANY STEP:
  → flush buffer → emit content_block (status: error, data: whatever we have)
```

---

*This document is a living specification. All block type additions must go through the checklist in Part IX. No parser ships without a registered block definition and an entry in this document.*
