# Block Processing — Open Tasks

> The one list of outstanding work in `ai/processing/blocks/`.
>
> The system performs an automatic integrity check at startup (see `stream_processor.py` →
> `_run_integrity_check()`). Any gaps listed here will also surface as **yellow/red warnings in the terminal**
> the moment the server starts, so nothing silently falls through.

---

## OPEN TASKS

---

## TASK-000 — Extend semantic chunking to other block types

**Status:** `OPEN`
**Severity:** Enhancement — not a bug, but the correct long-term pattern

### What is "semantic chunking"?

Timeline now uses a dedicated `TimelineStreamParser` (`ai/processing/blocks/parsers/timeline_stream_parser.py`)
registered under `_SEMANTIC_STREAM_TYPES` in `stream_processor.py`.

Instead of running a full parser on every token (PARTIAL_UPDATES, which always returns nothing until a unit seals),
or waiting until the block is completely received (MARKDOWN_COMPLETE), the semantic parser watches the *structure*
of the stream and emits a data update every time a **renderable unit is sealed**.

### How to add semantic chunking to a new block type

1. Create `ai/processing/blocks/parsers/{type}_stream_parser.py` with a class `{Type}StreamParser`
   - `feed(content: str) -> ModelType | None` — called on every content update; returns data when boundary crossed
   - `finalize() -> ModelType` — final flush; always returns complete data
2. Add the type key to `_SEMANTIC_STREAM_TYPES` in `stream_processor.py`
3. Remove the type from its current category (`_MARKDOWN_COMPLETE_TYPES` etc.)
4. `_apply_semantic_stream_block` handles the rest — no other wiring needed

### Block types that would benefit most

| Block type | Why semantic chunking helps |
|---|---|
| `troubleshooting` | Issues appear one at a time as the LLM writes each problem/solution pair |
| `research` | Sections appear progressively; findings within a section seal as the next section starts |
| `progress_tracker` | Categories and items appear as the LLM writes each group |
| `resources` | Each resource item seals when the next one or the close tag is seen |

---

## TASK-003 — Activate `consolidated_reasoning` block type

**Status:** `OPEN`
**Severity:** Low — type exists in `BlockType` enum and `BlockRegistry` but is unreachable

**Current state:**
- Declared in `BlockType.CONSOLIDATED_REASONING`
- Registered in `block_registry.py` with `StreamingBehavior.COMPLETE_ONLY`
- **NOT** in `XML_TAG_BLOCKS` in `block_detector.py` → detector can never produce it
- **NOT** in any processing type set in `stream_processor.py`
- No parser

**Note:** React's `EnhancedChatMarkdown` handles `consolidated_reasoning` client-side by consolidating
consecutive `reasoning` blocks after streaming completes. The Python side does not need to emit this
type unless we want server-side consolidation in the future.

**What to decide / build (if needed):**
1. Decide the detection mechanism: XML tag (e.g. `<consolidated_reasoning>`) or something else
2. Add the tag to `XML_TAG_BLOCKS` in `block_detector.py`
3. Add to `_INCREMENTAL_TYPES` in `stream_processor.py` (it's just joined reasoning text)
4. Add the XML tag to prompt templates that should produce this block type

---

## TASK-004 — Audit and complete `block_registry.py` parser assignments

**Status:** `OPEN`
**Severity:** Low — `BlockRegistry` has no `parser` set on any `BlockTypeDefinition` even though parsers exist

In `block_registry.py`, all `BlockTypeDefinition` entries are registered without a `parser=` field.
The parsers are wired separately in `stream_processor._build_parser_map()`.
This means `registry.get_parser(type_key)` always returns `None`.

**Options:**
- Wire parsers into `BlockTypeDefinition` so `registry.get_parser()` works
- Or document that parsers live only in `_build_parser_map()` and remove `get_parser()` from the registry API

---

## TASK-005 — Document `MODULE_README.md` Architecture section

**Status:** `OPEN`
**Severity:** Low — cosmetic / developer experience
**Location:** `ai/processing/MODULE_README.md` → `## Architecture` section

Fill in:
- Layer table (entry points, parsers, models, detector, stream_processor)
- Call flow diagram for a streaming request

---

## TASK-006 — `_parser_map` global cache is not thread-safe

**Status:** `OPEN`
**Severity:** Low — in practice only one server process, but technically a data race

`_parser_map` in `stream_processor.py` is a module-level global assigned once via a double-check
idiom without a lock. In a multi-threaded ASGI server (if ever used without a worker per process
model), two concurrent requests could both call `_build_parser_map()` on first startup.

**Fix:** Wrap with `threading.Lock` or convert to `functools.lru_cache(maxsize=1)`.

---

## TASK-007 — `generate_ts_types.py` drift from actual Python models

**Status:** `OPEN`
**Severity:** Medium — the generated `content-blocks.ts` in `matrx-admin` lags behind Python model changes

`ai/processing/blocks/generate_ts_types.py` generates `types/python-generated/content-blocks.ts`
in the React codebase. Recent Python model changes are NOT automatically reflected:

- `FlashcardsBlockData`: removed `partial_card` field; `FlashcardItem.back` now `str | None`
  → React `content-blocks.ts` was manually patched but future changes will drift again
- `ProgressTrackerBlockData`: field names differ between Python model and React `PROCESSING_TASKS.md` spec
  (Python uses `completion_percent`, React parser expects `completionPercentage`)

**Fix:** Add `generate_ts_types.py` to the CI pipeline, or run it as part of the deployment checklist.
Document the command: `python ai/processing/blocks/generate_ts_types.py`

---

## TASK-008 — `flashcards` block in `BlockRenderer.tsx` bypasses `serverData` path

**Status:** `OPEN` (React team task)
**Severity:** Medium — flashcards always re-parse on client even when Python sends structured data

In `BlockRenderer.tsx`, the `flashcards` case passes `block.content` to `FlashcardsBlock` (client-side
parser), ignoring `block.serverData`. All other block types have a `if (block.serverData)` fast path.

**Fix:** Add serverData path to `flashcards` case in `BlockRenderer.tsx`:
```tsx
case "flashcards":
    if (block.serverData) {
        return <BlockComponents.FlashcardsBlock
            key={index}
            serverData={block.serverData as FlashcardsBlockData}
            taskId={taskId}
        />;
    }
    return <BlockComponents.FlashcardsBlock key={index} content={block.content} taskId={taskId} />;
```
`FlashcardsBlock.tsx` also needs updating to accept `serverData` prop and skip `parseFlashcards()` when present.

---

## TASK-009 — `transcript` and `tasks` blocks missing `serverData` path in `BlockRenderer.tsx`

**Status:** `OPEN` (React team task)
**Severity:** Low — these use `content` only, never check `serverData`, but Python now sends structured data

`BlockRenderer.tsx` lines 189–192:
```tsx
case "transcript":
    return <BlockComponents.TranscriptBlock key={index} content={block.content} />;
case "tasks":
    return <BlockComponents.TasksBlock key={index} content={block.content} />;
```
Both always re-parse on client even when Python has already sent structured data in `block.serverData`.

---

## TASK-010 — `isNewProtocol` missing from `stream-events.ts` but referenced by `types.ts`

**Status:** `OPEN` (React team task — see also note below)
**Severity:** Medium — causes TypeScript compile error if `types.ts` export is used

`components/mardown-display/chat-markdown/types.ts` exports `isNewProtocol` from
`@/types/python-generated/stream-events`, but that function does not exist in `stream-events.ts`.

**Fix:** Either:
- Remove `isNewProtocol` from the `types.ts` export list (it is not used in `StreamAwareChatMarkdown.tsx`)
- Or add a simple helper to `stream-events.ts`:
  ```ts
  export function isNewProtocol(events: StreamEvent[]): boolean {
      return events.some(e => e.event === 'content_block');
  }
  ```

---

## COMPLETED TASKS (do not reopen)

| Task | Resolution |
|------|------------|
| TASK-001: Write `presentation` parser | DONE — `parsers/presentation_parser.py`, `models/presentation.py` |
| TASK-002: Write `math_problem` parser | DONE — `parsers/math_problem_parser.py`, `models/math_problem.py` |
| GAP-01: `code` block Two-Channel violation | DONE — moved to `_INCREMENTAL_TYPES`; `data` only on `is_final` |
| GAP-02: `MARKDOWN_COMPLETE` double-fire | DONE — parse gated on `is_final` only |
| GAP-03: `PARTIAL_UPDATES` emitting partial units | DONE — all parsers accept `is_final`, withhold last in-progress item |
| GAP-04: `StreamingBehavior` enum missing values | DONE — added `SEMANTIC_STREAM`, `MARKDOWN_COMPLETE` to enum and registry |
| GAP-05: `timeline_stream_parser.py` operator precedence bug | DONE — fixed event-bullet detection |
| GAP-08: `flashcard_parser` split `cards + partial_card` | DONE — single `cards` list with `back: None` loader pattern |
| GAP-09: `_stream_parsers` typed too narrowly | DONE — loosened to `dict[str, Any]` |
| GAP-10/11: `_CODE_TYPES` dead code + integrity set | DONE — removed `_CODE_TYPES`, `code` in incremental classified set |
| React 6-block shape issues (presentation, timeline, progress_tracker, math_problem, research, troubleshooting) | DONE — see resolved section below |
| Snake_case keys in events | DONE — `alias_generator=to_camel` + `model_dump(by_alias=True)` |
| Table blocks emitting as `type: "text"` | DONE — type promotion in `_update_existing_block` |
| Quiz blocks typed as `"code"` + wrong shape | DONE — JSON type promotion + renamed fields |
| `content_block` event type missing from `EventType` registry | DONE — added to `events.py`, `ContentBlockPayload` model added, `stream-events.ts` regenerated |
| `matrx-admin/types/python-generated/stream-events.ts` stale | DONE — regenerated and synced |
| `FlashcardsBlockData.partialCard` stale in `content-blocks.ts` | DONE — removed `partialCard`, `back` typed `string \| null` |
| `lib/api/types.ts` missing `ContentBlockPayload`, `ContentBlockEvent`, `isContentBlockEvent` | DONE — added to re-exports |

---

## React Team: Data Shape Reference

All file paths are relative to `matrx-admin/`.

| Block type | Python behavior | React parser file | React renderer file |
|---|---|---|---|
| `presentation` | COMPLETE_ONLY — JSON | inline `JSON.parse` in `BlockRenderer.tsx` L278-308 | `blocks/presentations/Slideshow.tsx` |
| `timeline` | SEMANTIC_STREAM — stateful parser | `blocks/timeline/parseTimelineMarkdown.ts` (fallback only) | `blocks/timeline/TimelineBlock.tsx` |
| `progress_tracker` | MARKDOWN_COMPLETE | `blocks/progress/parseProgressMarkdown.ts` (fallback only) | `blocks/progress/ProgressTrackerBlock.tsx` |
| `math_problem` | COMPLETE_ONLY — JSON | inline `safeJsonParse` in `BlockRenderer.tsx` L526-530 | `blocks/math/MathProblemBlock.tsx` |
| `research` | MARKDOWN_COMPLETE | `blocks/research/parseResearchMarkdown.ts` (fallback only) | `blocks/research/ResearchBlock.tsx` |
| `troubleshooting` | MARKDOWN_COMPLETE | `blocks/troubleshooting/parseTroubleshootingMarkdown.ts` (fallback only) | `blocks/troubleshooting/TroubleshootingBlock.tsx` |
| `flashcards` | PARTIAL_UPDATES | `blocks/flashcards/flashcard-parser.ts` (always, no serverData path yet) | `blocks/flashcards/FlashcardsBlock.tsx` |
| `quiz` | COMPLETE_ONLY — JSON | `safeJsonParse` fallback in `BlockRenderer.tsx` L260-275 | `MultipleChoiceQuiz` component |
| `table` | PARTIAL_UPDATES | handled in `StreamingTableRenderer` directly | `StreamingTableRenderer` |
| `transcript` | PARTIAL_UPDATES | always re-parses from `content`, no serverData path | `TranscriptBlock` |
| `tasks` | PARTIAL_UPDATES | always re-parses from `content`, no serverData path | `TasksBlock` |
