# matrx_ai.processing.blocks — Module Overview

> Cross-repo system-of-record for the planned unified content pipeline (this detector's TS twin, the golden round-trip corpus, and generated detection bootstraps that will replace the hand-mirrored `SPECIAL_CODE_LANGUAGES`/`XML_TAG_BLOCKS` lists): `/Users/armanisadeghi/code/common-docs/projects/unified-content-pipeline/FEATURE.md` — read it before touching block detection in ANY repo.


## IMPORTANT: Do not edit auto-generated sections.

> This document is partially auto-generated. Sections tagged `<!-- AUTO:id -->` are refreshed by the generator.
> Everything else is yours to edit freely and will never be overwritten.

<!-- AUTO:meta -->
## About This Document

This file is **partially auto-generated**. Sections wrapped in `<!-- AUTO:id -->` tags
are overwritten each time the generator runs. Everything else is yours to edit freely.

| Field | Value |
|-------|-------|
| Module | `matrx_ai/processing/blocks` |
| Last generated | 2026-04-07 11:38 |
| Output file | `matrx_ai/processing/blocks/MODULE_README.md` |
| Signature mode | `signatures` |


**Child READMEs detected** (signatures collapsed — see links for detail):

| README | |
|--------|---|
| [`matrx_ai/processing/blocks/models/MODULE_README.md`](matrx_ai/processing/blocks/models/MODULE_README.md) | last generated 2026-04-07 11:38 |
| [`matrx_ai/processing/blocks/parsers/MODULE_README.md`](matrx_ai/processing/blocks/parsers/MODULE_README.md) | last generated 2026-04-07 11:38 |
| [`matrx_ai/processing/blocks/tests/MODULE_README.md`](matrx_ai/processing/blocks/tests/MODULE_README.md) | last generated 2026-04-07 11:38 |
**To refresh auto-sections:**
```bash
python utils/code_context/generate_module_readme.py matrx_ai/processing/blocks --mode signatures
```

**To add permanent notes:** Write anywhere outside the `<!-- AUTO:... -->` blocks.
<!-- /AUTO:meta -->

<!-- HUMAN-EDITABLE: This section is yours. Agents & Humans can edit this section freely — it will not be overwritten. -->

## Architecture

> **Fill this in.** Describe the execution flow and layer map for this module.
> See `utils/code_context/MODULE_README_SPEC.md` for the recommended format.
>
> Suggested structure:
>
> ### Layers
> | File | Role |
> |------|------|
> | `entry.py` | Public entry point — receives requests, returns results |
> | `engine.py` | Core dispatch logic |
> | `models.py` | Shared data types |
>
> ### Call Flow (happy path)
> ```
> entry_function() → engine.dispatch() → implementation()
> ```


<!-- AUTO:tree -->
## Directory Tree

> Auto-generated. 67 files across 6 directories.

```
matrx_ai/processing/blocks/
├── MODULE_README.md
├── __init__.py
├── block_detector.py
├── block_registry.py
├── generate_ts_types.py
├── models/
│   ├── MODULE_README.md
│   ├── __init__.py
│   ├── artifact.py
│   ├── base.py
│   ├── comparison.py
│   ├── decision.py
│   ├── decision_tree.py
│   ├── diagram.py
│   ├── flashcards.py
│   ├── math_problem.py
│   ├── media.py
│   ├── presentation.py
│   ├── progress.py
│   ├── questionnaire.py
│   ├── quiz.py
│   ├── recipe.py
│   ├── research.py
│   ├── resources.py
│   ├── table.py
│   ├── tasks.py
│   ├── text.py
│   ├── thinking.py
│   ├── timeline.py
│   ├── transcript.py
│   ├── troubleshooting.py
├── parsers/
│   ├── MODULE_README.md
│   ├── __init__.py
│   ├── artifact_parser.py
│   ├── comparison_parser.py
│   ├── decision_parser.py
│   ├── decision_tree_parser.py
│   ├── diagram_parser.py
│   ├── diff_parser.py
│   ├── flashcard_parser.py
│   ├── math_problem_parser.py
│   ├── presentation_parser.py
│   ├── progress_parser.py
│   ├── questionnaire_parser.py
│   ├── quiz_parser.py
│   ├── recipe_parser.py
│   ├── research_parser.py
│   ├── resources_parser.py
│   ├── table_parser.py
│   ├── task_parser.py
│   ├── timeline_parser.py
│   ├── timeline_stream_parser.py
│   ├── transcript_parser.py
│   ├── troubleshooting_parser.py
├── stream_processor.py
├── tag_detection_buffer.py
├── tests/
│   ├── MODULE_README.md
│   ├── __init__.py
│   ├── block_tester.py
│   ├── stream_debug.py
│   ├── test_block_detector.py
│   ├── test_parsers/
│   │   ├── __init__.py
│   │   ├── test_flashcard_parser.py
│   │   ├── test_various_parsers.py
│   ├── test_results/
│   ├── test_stream_processor.py
│   ├── ts_bridge/
│   │   ├── parse_blocks.ts
# excluded: 6 .md, 1 .txt, 1 .json
```
<!-- /AUTO:tree -->

<!-- AUTO:signatures -->
## API Signatures

> Auto-generated via `output_mode="{mode}"`. ~5-10% token cost vs full source.
> For full source, open the individual files directly.
> Submodules with their own `MODULE_README.md` are collapsed to a single stub line.

```
---
Filepath: matrx_ai/processing/blocks/tag_detection_buffer.py  [python]

  class TagDetectionBuffer:
      def should_suppress(self, buffer: str) -> bool
  def _build_tag_sets() -> tuple[set[str], set[str]]
  def _ensure_tag_sets() -> None



---
Filepath: matrx_ai/processing/blocks/__init__.py  [python]




---
Filepath: matrx_ai/processing/blocks/generate_ts_types.py  [python]

  PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
  def _py_type_to_ts(annotation: Any) -> str
  def _model_to_interface(model_class: type, seen: set[str]) -> list[str]
  def _collect_deps(annotation: Any, seen: set[str], deps: list[str], lines: list[str]) -> None
  def _to_camel_case(snake: str) -> str
  def generate_content_blocks_ts() -> str
  def main() -> None



---
Filepath: matrx_ai/processing/blocks/stream_processor.py  [python]

  class StreamBlockProcessor:
      def __init__(self) -> None
      def process_token(self, token: str) -> list[ContentBlockEvent]
      def process_chunk(self, chunk: str) -> list[ContentBlockEvent]
      def finalize(self) -> list[ContentBlockEvent]
      def get_blocks(self) -> list[ContentBlock]
      def get_buffer(self) -> str
      def _next_block_id(self) -> str
      def _detect_and_emit(self) -> list[ContentBlockEvent]
      def _reconcile_blocks(self, detected: list[DetectedBlock], *, is_final: bool) -> list[ContentBlockEvent]
      def _create_block(self, detected: DetectedBlock, index: int, is_final: bool) -> ContentBlock
      def _update_existing_block(self, existing: ContentBlock, detected: DetectedBlock, is_final: bool) -> bool
      def _apply_block_content(self, block: ContentBlock, detected: DetectedBlock, block_type: str, is_final: bool) -> None
      def _apply_semantic_stream_block(self, block: ContentBlock, detected: DetectedBlock, is_final: bool) -> None
      def _apply_code_data(self, block: ContentBlock, detected: DetectedBlock, is_final: bool) -> None
      def _apply_media_block(self, block: ContentBlock, detected: DetectedBlock) -> None
      def _try_promote_code_block(self, block: ContentBlock, detected: DetectedBlock) -> str | None
      def _try_promote_json_block(self, block: ContentBlock, detected: DetectedBlock) -> bool
      def _run_parser(self, block: ContentBlock, *, is_final: bool = False, metadata: dict[str, Any] | None = None) -> None
      def _consolidate_reasoning(self) -> ContentBlockEvent | None
  def _build_parser_map() -> dict[str, Callable[[str], Any]]
  def _model_to_dict(model: Any) -> dict[str, Any] | None
  def _get_parser_map() -> dict[str, Callable[[str], Any]]
  def _is_incremental(block_type: str) -> bool
  def _is_partial_update(block_type: str) -> bool
  def _is_semantic_stream(block_type: str) -> bool
  def _is_markdown_complete(block_type: str) -> bool
  def _is_complete_only(block_type: str) -> bool
  def _run_integrity_check() -> None
  def process_complete_content(content: str) -> list[ContentBlockEvent]
  def process_complete_to_blocks(content: str) -> list[dict[str, Any]]
  def _parse_artifact_with_metadata(c: str, metadata: dict | None = None, **_kw: Any) -> Any
  def _parse_decision_with_metadata(c: str, metadata: dict | None = None, **_kw: Any) -> Any



---
Filepath: matrx_ai/processing/blocks/block_detector.py  [python]

  MATRX_FIELD_PATTERNS = {8 keys}
  SPECIAL_CODE_LANGUAGES = {'transcript', 'tasks', 'structured_info', 'questionnaire', 'flashcards', 'cooking_recipe'}
  class DetectedBlock:
  class ExtractionResult:
  def _extract_first_json_key(content: str) -> str | None
  def detect_json_block_type(content: str) -> str | None
  def _contains_placeholder_text(content: str) -> bool
  def validate_json_block(content: str, block_type: str) -> dict[str, Any]
  def detect_xml_block_type(line: str) -> tuple[str, str] | None
  def extract_xml_block(block_type: str, matched_tag: str, start_index: int, lines: list[str]) -> ExtractionResult
  def detect_attribute_xml_block(line: str) -> dict[str, Any] | None
  def detect_mid_line_attribute_xml(line: str) -> dict[str, Any] | None
  def _parse_xml_attributes(opening_tag: str) -> dict[str, str]
  def extract_attribute_xml_block(detection: dict[str, Any], start_index: int, lines: list[str], raw_xml_override: str | None = None) -> ExtractionResult
  def _validate_streaming_xml_block(block_type: str, content: str, found_closing: bool) -> dict[str, Any]
  def _validate_questionnaire_streaming(content: str, found_closing: bool) -> dict[str, Any]
  def _validate_flashcard_streaming(content: str, found_closing: bool) -> dict[str, Any]
  def _validate_recipe_streaming(content: str, found_closing: bool) -> dict[str, Any]
  def detect_code_block(line: str) -> tuple[bool, str | None]
  def extract_code_block(start_index: int, lines: list[str]) -> ExtractionResult
  def detect_table_row(line: str) -> bool
  def _is_table_separator(line: str) -> bool
  def extract_table(start_index: int, lines: list[str]) -> ExtractionResult
  def detect_image(line: str) -> tuple[bool, str | None, str | None]
  def detect_video(line: str) -> tuple[bool, str | None, str | None]
  def split_content_into_blocks(md_content: str) -> list[DetectedBlock]
  def flush_text() -> None



---
Filepath: matrx_ai/processing/blocks/block_registry.py  [python]

  class DetectionMethod(str, Enum):
  class DetectionRule:
      def matches_xml_tag(self, tag: str) -> bool
      def matches_json_key(self, key: str) -> bool
      def matches_code_lang(self, lang: str) -> bool
  class BlockTypeDefinition:
      def __post_init__(self) -> None
  class BlockRegistry:
      def __init__(self, *, auto_register_builtins: bool = True) -> None
      def register(self, definition: BlockTypeDefinition) -> None
      def get(self, type_key: str) -> BlockTypeDefinition | None
      def has(self, type_key: str) -> bool
      def all_types(self) -> dict[str, BlockTypeDefinition]
      def type_keys(self) -> list[str]
      def get_parser(self, type_key: str) -> Callable[[str], Any] | None
      def get_streaming_behavior(self, type_key: str) -> StreamingBehavior
      def detect_xml_type(self, tag: str) -> str | None
      def detect_json_type(self, root_key: str) -> str | None
      def detect_code_lang_type(self, language: str) -> str | None
      async def load_from_db(self, supabase_client: Any) -> int
      def _register_builtins(self) -> None



---
Submodule: matrx_ai/processing/blocks/parsers/  [23 files — full detail in matrx_ai/processing/blocks/parsers/MODULE_README.md]

---
Submodule: matrx_ai/processing/blocks/models/  [25 files — full detail in matrx_ai/processing/blocks/models/MODULE_README.md]

```
<!-- /AUTO:signatures -->

<!-- AUTO:dependencies -->
## Dependencies

**External packages:** matrx_utils, pydantic, pytest
<!-- /AUTO:dependencies -->

<!-- AUTO:config -->
## Generation Config

> Auto-managed. Contains the exact parameters used to generate this README.
> Used by parent modules to auto-refresh this file when it is stale.
> Do not edit manually — changes will be overwritten on the next run.

```json
{
  "subdirectory": "matrx_ai/processing/blocks",
  "mode": "signatures",
  "scope": null,
  "project_noise": null,
  "include_call_graph": false,
  "entry_points": null,
  "call_graph_exclude": [
    "tests"
  ],
  "signatures_exclude": [
    "tests",
    "db",
    "local_data",
    "docs"
  ]
}
```
<!-- /AUTO:config -->

## Python ↔ TypeScript parity gate

`block_detector.py` is a port of matrx-frontend's `content-splitter-v2.ts`, and
the kinds program makes it the canonical resolver of agent output into kinds for
the workflow system — so a detection difference between the two runtimes means
silently different results per surface.

**The gate is `packages/matrx-ai/tests/parity/`.** Committed adversarial markdown
fixtures run through BOTH production splitters (the TS side via
`tests/ts_bridge/parse_blocks.ts`) with a field-by-field diff.

```bash
uv run pytest packages/matrx-ai/tests/parity          # CI face
uv run python packages/matrx-ai/tests/parity/harness.py   # human report
```

Rules that make it worth trusting:

* **The frontend is the richer, currently-live implementation** — a difference is
  a Python defect until a recorded finding says otherwise. Never weaken either
  parser to make them match; reconcile UP.
* **Metadata exclusions are declared, scoped to one block type, and carry a
  reason** (`FE_ONLY_METADATA_KEYS` / `PY_ONLY_METADATA_KEYS`) — never a silent skip.
* **A known divergence is an ASSERTED divergence** (`EXPECTED_DIVERGENT_FIXTURES`):
  the gate fails if it disappears, so a finding gets re-decided rather than
  quietly resolving itself.
* **An unverifiable gate skips LOUDLY** (no sibling checkout / no node) — it never
  reads as passing.

Adding a block type? Add it to the splitter, to `SPLITTER_ONLY_BLOCK_TYPES` (if
it is not already named by one of the detection tables), classify it in
`envelope.py` and in a `stream_processor.py` processing set, and add a fixture
here that exercises it.
