# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2.9.12] - 2026-07-18

**BedrockProcessing: source the output schema from the embedded prompt ruleset, so the separate
`validation_schema` input channel is no longer required.**

The prompt-ruleset producers now embed the output schema in the `{ruleset, rules}` prompts.json
(`ruleset.output_schema`). `bedrock_processing.py` now prefers that embedded schema when resolving
the validation schema, so the schema travels WITH the prompt and a dedicated schema channel is
unnecessary (a step toward simplifying the Bedrock step inputs from 3 to 2).

### Changed

- `bedrock_processing.py` — `_adapt_ruleset_templates` surfaces `ruleset.output_schema` as
  `templates["_output_schema"]`; `_load_schema_with_fallback` prefers it (embedded-in-ruleset →
  upstream channel → env var → empty). Additive; channel/env-var paths remain as fallbacks.

## [2.9.11] - 2026-07-18

**BedrockPromptTemplateGeneration: refactored to a declarative meta-prompt assembler emitting one
`prompts.json` prompt ruleset.**

The 1,599-line prose-generation script is replaced by a ~470-line assembler: validate the four
config blanks (system prompt, category rules, output schema, instructions) → assemble → emit ONE
`prompts.json` in the `{ruleset, rules}` shape — the same contract the rule-driven routing step
emits, so both prompt-ruleset producers share one downstream contract. The output schema is
embedded IN the prompt (the `{OUTPUT_CONSTRAINTS}` block) and carried as `ruleset.output_schema`,
enum-locked to the rule names.

### Changed

- `bedrock_prompt_template_generation.py` — rewritten as a meta-prompt assembler. Output is now
  `{ruleset, rules}` (was the flat `{system_prompt, user_prompt_template, input_placeholders}`);
  `rules[]` mapped to the shape the consumer's ruleset adapter reads. Removed the tone-register
  table, placeholder example-fabricator, and self-output prose validator — the fill-in-the-blank
  contract is enforced by the step config's Pydantic models at authoring time. Step config +
  interface I/O contract unchanged.
- Test suite rewritten to exercise the assembler functions + the `{ruleset, rules}` contract.

## [2.9.10] - 2026-07-18

**Bedrock steps: `validation_schema` can no longer mis-bind to a data/prompt output.**

`BedrockProcessing` and `BedrockBatchProcessing` declare a `validation_schema` input whose only
legitimate producer is `BedrockPromptTemplateGeneration`. Both interfaces also listed
`SlipboxKnowledgeRouting` as a compatible source, but SKR emits no schema output (only
`prompt_ruleset` + `routed_records`; the schema is built in-step). Because all three inputs were
`type: processing_output` / `S3Uri`, the resolver could bind the schema channel to SKR's
`routed_records` — the same 3-way collision class as the 2.9.8 SlipboxKnowledgeRouting redesign,
on the consumer side.

### Fixed

- `bedrock_processing.step.yaml` + `bedrock_batch_processing.step.yaml` — removed
  `SlipboxKnowledgeRouting` from `validation_schema.compatible_sources` and retyped the
  `validation_schema` dependency `processing_output → custom_property` (type-compatible only with
  `custom_property`, so any `processing_output`/data producer scores `0.0` and is excluded). The
  `compatible_sources` drop alone was insufficient — it is a ±0.1 penalty, not a hard gate.
- `bedrock_prompt_template_generation.step.yaml` — retyped its `validation_schema` output to
  `custom_property` so the legitimate template-generation → Bedrock schema edge still binds
  (verified `1.000`). `property_path`/`data_type` unchanged.

## [2.9.9] - 2026-07-18

**BedrockProcessing: no longer crashes with `KeyError` on an empty (0-row) input.**

A Processing run whose input parquet carries a full column schema but zero data rows (e.g. an
incremental delta window with no new records to score) made `process_batch_concurrent` /
`process_batch_sequential` return a column-less `pd.DataFrame([])`. `main()` then accessed
`result_df[f"{output_column_prefix}status"]` unconditionally and raised `KeyError` on that status
column — even though an empty input is a legitimate state, not a failure.

### Fixed

- `bedrock_processing.py` — at both `process_batch_concurrent` and `process_batch_sequential`
  return sites, restore the invariant these methods promise (the returned frame always carries the
  status column) when the results frame is empty: return a schema-preserving 0-row frame
  (`df.iloc[0:0]` + the `{output_column_prefix}status` column). All downstream count / filter /
  write logic then runs cleanly on 0 rows, and a valid empty output is written. Non-empty runs are
  unchanged.

## [2.9.8] - 2026-07-17

**SlipboxKnowledgeRouting step-interface redesign: bundled `knowledge_corpus` + `embedding_model`
no longer mis-resolve to the TabularPreprocessing output.**

The step declares three inputs — `records`, `knowledge_corpus`, `embedding_model` — but the corpus
and encoder ship *inside* the step's `source_dir` (the bundled knowledge closure), so they are not
upstream data. The interface had declared all three as `processing_output` dependencies with
overlapping `compatible_sources`, so the dependency resolver could bind **all three to the same
`TabularPreprocessing` output**.

### Changed

- `slipbox_knowledge_routing.step.yaml`: `knowledge_corpus` + `embedding_model` moved under
  **`contract.skip_inputs`** (declared-but-not-mounted, the same pattern `PercentileModelCalibration`
  uses for its bundled `calibration_config`); their `contract.inputs` paths re-pointed to the
  source_dir (`/opt/ml/code/…`); `patterns.direct_input_keys` narrowed to `[records]`. Dependency
  `type` changed to **`custom_property`** (corpus) and **`model_artifacts`** (encoder) — neither
  type-compatible with a `processing_output` — and `compatible_sources` narrowed to dedicated
  producers, excluding `TabularPreprocessing`/`ProcessingStep`/`CradleDataLoading`.
- `SlipboxKnowledgeRoutingConfig` gains **`use_bundled_corpus: bool = True`**, documenting/defaulting
  the bundled mode (mirrors the interface `skip_inputs`).

Verified with the real dependency resolver on a `TabularPreprocessing → SlipboxKnowledgeRouting`
graph: only `records` resolves; the bundled inputs are correctly left unbound.

## [2.9.7] - 2026-07-17

Two general fixes to the `SlipboxKnowledgeRouting` and `StratifiedSampling` processing steps.

### Fixed

- **`stratified_sampling` no longer crashes on parquet with a >2 GiB string column.** Reading a
  split with `pandas.read_parquet()` materializes each column as one contiguous Arrow array; a
  large-text feature whose bytes sum past **2^31 (2,147,483,646)** across the file overflows Arrow's
  int32 string-offset limit (`ArrowCapacityError: array cannot contain more than 2147483646 bytes`)
  inside `read_table()`. `stratified_sampling.py` now takes a **two-pass, memory-bounded streaming
  path** for a parquet split with no row-level filter: Pass 1 reads only the tiny `(id, strata)` key
  columns and runs the existing allocation to choose rows; Pass 2 streams the file row-group-by-
  row-group and writes only the selected rows — **by id** when a unique id column is present, else
  **by position** — with a single `ParquetWriter`. The `test` split is copied byte-for-byte. The
  full file is never materialized. CSV/TSV and the filter / `external_proportional` cases keep the
  in-memory path.
- **`SlipboxKnowledgeRoutingConfig` now emits its env vars to the container.** It had no config-owned
  `get_environment_variables()`, so the universal builder fell back to the generic `NAME → self.<name>`
  resolver, which only emits vars a `.step.yaml` declares — dropping `USE_SECURE_PYPI` (needed so the
  step can pull its container-start dependencies from a secure PyPI index in a restricted-network VPC).
  Added a bespoke `get_environment_variables()` (mirrors the pytorch model-eval config) emitting
  `USE_SECURE_PYPI` + the routing knobs.

### Added

- `StratifiedSamplingConfig.id_column` (default `order_id`) + `ID_COLUMN` in
  `stratified_sampling.step.yaml`, making the streaming path's id-selection column configurable.
  Absent/non-unique id ⇒ automatic fallback to positional selection.

## [2.9.4] - 2026-07-14

**Repository hygiene (internal-identifier scrub) + generic `BedrockProcessing` routed-rule injection.**

### Security / hygiene

- **Scrubbed Amazon-internal infrastructure identifiers repo-wide** (shipped `src/`, `projects/`, notebook output, config JSON, tests). Real AWS account IDs, IAM/KMS/Step-Functions ARNs, internal S3 buckets, team IDs, subnets/SGs, and internal wiki URLs were replaced with placeholders. **No functional change**: the CodeArtifact/SecurePyPI install bootstrap is now driven by env vars (`SECURE_PYPI_ROLE_ACCOUNT`, `SECURE_PYPI_DOMAIN`, `SECURE_PYPI_DOMAIN_OWNER`, `SECURE_PYPI_REPOSITORY`) with a `123456789012` placeholder fallback; that path is opt-in (`USE_SECURE_PYPI=false` default → public PyPI), so the placeholder is never used unless an operator sets the env vars. Load-bearing tokens (step types, imports, data-source enums) were intentionally left unchanged.
- **Added `Agent.md`** at the repo root: a transfer-hygiene checklist reminding any agent to scrub internal identifiers (sweeping the *whole* repo, not just `src/`) before pushing, with the env-var-indirection pattern and a list of what must never be renamed.

### Added

- **`scripts/bedrock_processing.py`** — generic per-record routed-rule injection: `_adapt_ruleset_templates` (detects an upstream `{ruleset, rules}` shape, adds a `{routed_rules}` slot), `_render_routed_rules`, `_parse_output_column_map` + `canonicalize_output_columns` (config-driven output-column rename). `{routed_rules}` is threaded per record via the config's `routed_rules_column`. No domain field names hardcoded.
- **`configs/config_bedrock_processing_step.py`** — `bedrock_output_column_map` (`BEDROCK_OUTPUT_COLUMN_MAP`) + `bedrock_routed_rules_column` (`BEDROCK_ROUTED_RULES_COLUMN`), enumerated in `get_public_init_fields()`.
- **`interfaces/bedrock_processing.step.yaml`** — the 2 new optional env vars.
- **`shared_dags/bedrock/bedrock_slipbox_routing_pytorch_incremental.dag.json`** — two-source incremental training catalog pipeline (`SlipboxKnowledgeRouting` → Bedrock → EDX anti-join + write-back).

### Notes

- Purely additive for the Bedrock change (empty map ⇒ no rename; `{routed_rules}` injected only when the template carries the slot). Mirrors the internal 2.9.4 revision. All edited `.py` compile; all edited `.json`/`.ipynb`/`.yaml` parse.

## [2.9.2] - 2026-07-13

**`SlipboxKnowledgeRouting` — script↔interface↔config alignment pass.** A three-way alignment audit of the step surfaced several gaps, fixed here: `knowledge_corpus` is now an optional input with a guaranteed in-container bundled default and a sanctioned S3 override path, and the full runtime config (including the corpus-default + index paths) flows through declared env vars.

### Changed

- **`knowledge_corpus` input → optional** — `required: true` → **`false`** in `contract.inputs` and `spec.dependencies`. The script reads it via `input_paths.get(...)` and falls back to the bundled DKS corpus, so the channel is an override; only `records` stays required.
- **`knowledge_corpus.compatible_sources` += `DummyDataLoading`** — now `[DummyDataLoading, CradleDataLoading, ProcessingStep]`. `DummyDataLoading` is the sanctioned way to bring an external S3 corpus into a pipeline, so it must be a compatible source for the optional override (`DummyDataLoading.DATA → knowledge_corpus` resolves at 0.717).
- **Config-backed env surface** — added `DEFAULT_CORPUS_VAULT`, `DEFAULT_PROMPTS_DIR`, `ROUTING_INDEX_PATH` to `contract.env_vars.optional` and backed each with a config field (`default_corpus_vault`/`default_prompts_dir` default `""` ⇒ the script self-locates the bundled corpus from `__file__`; `routing_index_path` → scratch pkl). All config resolves via `get_environment_variables()` (`NAME ← self.name`), so the script reads every knob from `main()`'s `environ_vars` (testability). `get_public_init_fields()` now enumerates the new fields + `framework_version`/`py_version`.

### Notes

- Env alignment verified 7-way (script-reads == interface-declares == config-backs); interface validates with 0 errors.

## [2.9.1] - 2026-07-13

**`SlipboxKnowledgeRouting` → Processing pattern 2 (source-dir / PyTorch FrameworkProcessor).** The step ships a **source directory** (its entry-point script plus a co-located package + DKS corpus it imports at runtime), so its interface now adopts the same source-dir pattern as the model-eval / model-inference steps. Corrective follow-up to 2.9.0 (which declared the step as a self-contained `sklearn` `ScriptProcessor` — `ScriptProcessor.run` has no `source_dir`, so it could not ship the sibling package).

### Changed

- **`interfaces/slipbox_knowledge_routing.step.yaml`** — `compute.kind: sklearn` → **`framework`** with `sdk_class: PyTorch` + `py_version_field: py_version` (the PyTorch base image already carries torch, so the container only layers on `sentence-transformers`, vs. installing the whole torch stack onto a sklearn image); `patterns.step_assembly: step_args`; `contract.source_dir: true`.
- **`configs/config_slipbox_knowledge_routing_step.py`** — added `framework_version` (default `2.1.2`) + `py_version` (default `py310`), mirroring the pytorch model-eval config; `processing_source_dir` (inherited) points at the step's script directory at pipeline-wiring time.

### Notes

- No dependency/output changes — the 2.9.0 `compatible_sources` wiring is unchanged; the delta DAG still resolves.

## [2.9.0] - 2026-07-13

New pipeline step **`SlipboxKnowledgeRouting`** (Processing) — hosts a DKS knowledge + ruleset corpus and runs *compile → index → route* inside one container, emitting a compiled prompt ruleset (+ tool schema) and per-record routed rules to a downstream Bedrock inference step. Reusable for any DKS-driven LLM-labeling pipeline.

### Added

- **`SlipboxKnowledgeRouting` step** — three artifacts under `src/cursus/steps/`:
  - `interfaces/slipbox_knowledge_routing.step.yaml` — self-registering interface (registry-by-construction). Dependencies: `records` (required; `compatible_sources` includes `TabularPreprocessing`), `knowledge_corpus` (required — the `rule_*/pattern_*/behavior_*` DKS corpus), `embedding_model` (optional — offline `all-MiniLM-L6-v2`). Outputs: `prompt_ruleset` (compiled `prompts.json` + tool schema) and `routed_records`. `sagemaker_step_type: Processing`, `compute.kind: sklearn`.
  - `configs/config_slipbox_knowledge_routing_step.py` — `SlipboxKnowledgeRoutingConfig(ProcessingStepConfigBase)` with routing knobs (`routing_scoring_mode`/`routing_threshold`/`routing_top_k`/`embedding_model_name`).
  - `scripts/slipbox_knowledge_routing.py` — Cursus contract entry point (`main(input_paths, output_paths, environ_vars, job_args)` + `--job_type`); compile → embed-index → route scaffold.

### Changed

- **`BedrockProcessing` + `BedrockBatchProcessing` interfaces** — added `SlipboxKnowledgeRouting` to `compatible_sources` on `input_data`, `prompt_templates`, and `validation_schema`.
- **`LabelRulesetExecution` interface** — `validated_ruleset.compatible_sources` now accepts both `LabelRulesetGeneration` and `SlipboxKnowledgeRouting`.

## [2.8.5] - 2026-07-06

Robustness: keep `stdout` clean **at import time** so the stdio MCP server can't be corrupted by import-time logging, without any external wrapper.

### Fixed

- **Import-time stdout guard** (`cursus/__init__.py`). Raising the `sagemaker.config` logger to WARNING before the core import (2.8.3) is enough in most environments, but on a host where a `StreamHandler` bound to `stdout` is already attached before that line runs, `sagemaker.config INFO` lines still reach `stdout` during `import cursus` and corrupt a stdio JSON-RPC stream. The sagemaker-pulling `core.compiler` import now additionally runs inside a `redirect_stdout(sys.stderr)` guard, so **any** import-time byte (a pre-attached stdout handler, a warning, a stray print) lands on `stderr` and can never corrupt a stdout protocol stream. Set `CURSUS_KEEP_IMPORT_STDOUT=1` to opt out; graceful-degradation on a missing dependency is preserved (a real `ImportError` still propagates).

## [2.8.4] - 2026-07-06

Make the MCP server production-ready for **public** use. A public-readiness review found the
2.8.3 server connected fine but tools failed the moment a model invoked them, and the surface
was unsafe by default under an LLM. This release fixes both.

### Fixed

- **Tool calls failed from every host (blocker).** All ~70 tool names used a dotted namespace
  (`catalog.list_steps`). MCP hosts relay tool names to the Anthropic/OpenAI tool-calling APIs,
  whose name pattern (`^[a-zA-Z0-9_-]{1,64}$`) rejects the `.`, so calls 400'd. Tools are now
  exposed on the wire with `__` in place of the dot (`catalog__list_steps`) and round-trip back
  to the dotted name; human-facing help still shows the dotted names. Both forms are callable.
- **`mcp` version floor.** The server calls `Server(name, version=…)`, a kwarg added in mcp
  1.2.0 (1.0/1.1 raised `TypeError`). The extra is now `mcp>=1.2.0,<2`.
- **Import-time stdout leak.** The server now builds with stdout redirected to stderr and sets
  a confinement root, closing a window where an import-time log line could corrupt JSON-RPC.
- Corrected the `cursus.mcp` package docstring example (it used an argument the schema rejects).

### Added

- **Safe by default.** The public server is **read-only** unless the operator opts in:
  `CURSUS_MCP_ENABLE_DESTRUCTIVE=1` enables filesystem writes + AWS upserts (`compile.dag`,
  `project.init`, `dag.serialize`); `CURSUS_MCP_ALLOW_SCRIPT_EXEC=1` enables running step
  scripts (`validate.run_scripts`). Gated tools are neither listed nor callable otherwise.
- **`project.init` write confinement.** `name` may not contain path separators, `target_dir`
  may not contain `..`, and under a server (`CURSUS_MCP_PROJECT_ROOT`, defaulting to the
  server's working dir) writes cannot escape that root — closing a path-traversal write.
- **MCP tool annotations** (`readOnlyHint` / `destructiveHint` / `openWorldHint`) on every
  tool, so hosts can auto-approve read-only calls and confirm mutating ones.
- **`isError` signaling.** Failed tool calls now set the protocol `isError` flag instead of
  returning a success envelope with the error buried in text.
- **`src/cursus/mcp/README.md`** — onboarding for Claude Desktop / Cursor / Kiro with
  paste-ready configs (absolute launcher path, AWS env passthrough, the safety matrix), plus
  refreshed `installation.md` / README MCP sections.

## [2.8.3] - 2026-07-06

Public MCP server: installable extra + two stdio-server fixes.

### Added

- **`pip install "cursus[mcp]"` and a `cursus-mcp` console script.** The MCP server was already documented (`cursus mcp serve` / `python -m cursus.mcp.server`), but the `mcp`/`anyio` SDK had no install path and no stable launch command. `cursus[mcp]` now pulls the SDK (kept optional so a plain `pip install cursus` stays lean), `cursus-mcp` launches the stdio server, and `cursus[all]` includes it.

### Fixed

- **stdio MCP protocol corruption.** `sagemaker.config` attaches a `StreamHandler(stdout)` and logs two INFO lines ("Not applying SDK defaults from location: …") the moment sagemaker is imported (cursus pulls it in transitively). Because a stdio MCP server frames JSON-RPC on **stdout**, the client failed with "Failed to parse JSONRPC message". Fixed at the source: `cursus/__init__.py` raises the `sagemaker.config` logger to WARNING *before* the core import, so the noise never reaches stdout; the server adds a defense-in-depth stdout scrub for anything else that logs at runtime.
- **`serverInfo` reported the wrong version.** The MCP server was constructed without a version, so it advertised the **mcp SDK's** version as its own. It now reports `cursus.__version__`.

## [2.8.2] - 2026-07-06

### Docs

- **Documentation site hosting finalized.** Read the Docs is now the canonical docs home (**https://cursus.readthedocs.io/**), with GitHub Pages (https://tianpeiluke.github.io/cursus/) as a mirror; the README and `[project.urls] Documentation` point at Read the Docs. Fixes the Read the Docs `stable` build: the `.readthedocs.yaml` PDF/LaTeX output (which fails on large auto-generated API docs) is dropped, and — because `stable` tracks the newest release tag — that fix only takes effect once it lands in a tag (this release). No package/code changes.

## [2.8.1] - 2026-07-05

### Fixed

- **`from cursus import PipelineDAG` was broken.** The top-level package did `from .api.dag import EnhancedPipelineDAG, PipelineDAG`, but `EnhancedPipelineDAG` had been removed from `cursus.api.dag` — so the whole import raised `ImportError` and the `except` bound a broken **`PipelineDAG` stub** that raised `"DAG functionality not available"` on instantiation. `PipelineDAG` is now imported independently, so `from cursus import PipelineDAG` yields the real class again. The dead `EnhancedPipelineDAG` name is removed from the public surface (`__init__` and `__all__`); use `PipelineDAG`.

### Docs

- Published a comprehensive documentation site (Sphinx + Furo; auto-generated API/CLI/MCP/step-catalog/pipeline-catalog reference + tutorials/concepts/guides), hosted at **https://tianpeiluke.github.io/cursus/**; `[project.urls] Documentation` now points there.

## [2.8.0] - 2026-07-05

Ported from upstream 2.7.0 — plugin step-pack discovery. A consumer can define its own steps (`interfaces/*.step.yaml` + `configs/` + `scripts/`) in a folder OUTSIDE the pip-installed package and have cursus discover them as NATIVE — no fork, no vendored copy, no package edit. Strictly ADDITIVE: the internal steps are ALWAYS available; a pack can only add steps (shadow-with-warning on clash), never remove or replace one. Reuses the caller hook as the discovery anchor. Backward-compatible; with no pack the registry/catalog are byte-identical to package-only.

### Added

- **External config/hyperparam import by file location** (`step_catalog/config_discovery.py`) — a workspace config module not under `package_root` is imported via `spec_from_file_location` under a unique path-hashed module name (previously AST-detected then silently dropped).
- **`refresh_registry(pack_interfaces_dir)`** + `merge_pack_registry` — merge a pack's `.step.yaml` rows ON TOP of the package registry (in-place `.update`, never replace), rebuild derived registries, re-sync the hybrid manager. Collisions surfaced via `get_registry_health()['pack_collisions']`.
- **Ordered interface loader** (`steps/interfaces/__init__.py`) — pack `interfaces/` dirs searched AFTER the package (package always wins) with cache invalidation.
- **`PipelineDAGCompiler(workspace_dirs=...)`** + `_derive_step_pack_dir` from the resolved `project_root`/`anchor_file`; `set_default_workspace_dirs` so bare `StepCatalog()` sees packs.
- Regression tests (`tests/step_catalog/test_plugin_pack_additive_invariant.py`) locking the additive invariant.

## [2.7.0] - 2026-07-05

Ported from upstream 2.6.0 — `anchor_file=__file__` caller hook + a shared `resolve_anchor()` normalizer. A pipeline's entry module can register its project folder self-documentingly (`anchor_file=__file__`) instead of `project_root=Path(__file__).parent`, and the anchor is threaded through every entry point that resolves docker `source_dir`/`processing_source_dir` paths. Backward-compatible; additive only.

### Added

- **`resolve_anchor(anchor)` in `core/utils/hybrid_path_resolution.py`** — the single shared normalizer that collapses a file (`__file__` → its parent), a directory (used as-is), or a nonexistent file-shaped path (→ its parent) into one project-root string. Exported from `core.utils`; `set_project_root` routes through it.
- **`anchor_file` parameter** on `PipelineDAGCompiler` and `SingleNodeCompiler` — `anchor_file=__file__` is equivalent to `project_root=Path(__file__).parent`. Precedence: explicit `project_root` > `anchor_file` > config-anchored inference, with a warning when the two disagree.
- **`project_root` / `anchor_file` on `ExecutionDocumentGenerator`** (pushes the root before configs load; matters for exec-doc-only CLI/MCP flows), plumbed through the `exec-doc generate` CLI (`--project-root` / `--anchor-file`) and the `execdoc.generate` MCP tool.
- **`project_root` / `anchor_file` on `DAGConfigFactory`** — config-only generation anchors correctly.
- **`project.init` scaffold emits `anchor_file=__file__`** in the generated `@MODSTemplate` class, `run_pipeline.py`, and `generate_config.py`.
- Tests: `resolve_anchor`, `set_project_root` file-vs-dir parity, `anchor_file` wiring + precedence.

## [2.6.0] - 2026-07-04

Ported from upstream 2.5.0 — self-documenting MCP toolset. Backward-compatible; additive only.

### Added

- **`tools.help` — the toolset front door.** One call returns a prose overview, the planner/validator/programmer phase taxonomy, and every tool grouped by namespace with its description. Filters: `namespace`, `phase`, `include_schema`. CLI: `cursus mcp help [--namespace --phase --schema --examples --format json]`.
- **Auto-generated `<namespace>.help` tools** (`compile.help`, `catalog.help`, …) — one per non-meta namespace, built from each module's co-located `NAMESPACE` constant.
- **`ToolDef.when` + `ToolDef.examples`** on every tool (135 schema-faithful examples). `render_description()` folds them into the exported description so MCP/OpenAI clients see them too.
- Co-located namespace descriptions via a module-level `NAMESPACE` constant (`registry.get_namespaces()`); `tools.describe_tool` surfaces `when`/`examples`. Tests in `test_registry.py` + `test_info_tools.py`.

## [2.5.10] - 2026-07-04

Ported from upstream 2.4.12.

### Fixed

- **`ModelMetricsComputationConfig.score_field` default `None` → `"prob_class_1"`.** The v2.2.0 model-validator requires "at least one of `score_field`/`score_fields`", but the field had no default — so `set_step_config("ModelMetricsComputation_*", ...)` raised and saved configs omitting it failed validation on reload. `prob_class_1` is the positive-class column upstream `XGBoostModelInference` emits and `model_metrics_computation.py` auto-detects (`prob_class_` prefix). Single-task binary pipelines are now valid without an explicit value; multi-class/multi-task still overrides via `score_field`/`score_fields`.

## [2.5.9] - 2026-07-04

Ported from upstream 2.4.11 — config-resolver + DAGConfigFactory robustness (multi-pipeline validation campaign). Backward-compatible; no API removals.

### Fixed

- **`job_type=None` no longer crashes resolution** — `step_catalog/adapters/config_resolver.py`, 4 sites changed to `(getattr(config, "job_type", "") or "").lower()`. A config whose `job_type` is present-but-`None` previously raised `AttributeError`, masking a genuine DAG↔config mismatch as an opaque "unresolvable node".

### Added

- **Bare-step-name resolution (resolver + factory)** — a suffix-less DAG node resolves to the single config keyed with a suffix (node `PercentileModelCalibration` → config `PercentileModelCalibration_calibration`), guarded to be unambiguous. `DAGConfigFactory.set_step_config` accepts the bare base name + `job_type=` via the new `_resolve_step_name_to_node`, instead of raising "Step not found in DAG".
- **`DAGConfigFactory.validate_dag_config_alignment()`** (called in `generate_all_configs()`) — asserts each configured instance's derived save-key equals its DAG node key, turning step-TYPE drift into a loud build-time error.
- **`DAGConfigFactory.is_dag_step()` + `configure_step_if_present()`** — non-silent replacement for the `if "X" in pending_steps` guard that hides typos/renamed steps.
- Regression tests: `TestJobTypeNoneRobustness`, `TestBareStepNameMatching`, `TestSetStepConfigBareName`, `TestDagConfigAlignmentValidation`.

## [2.5.8] - 2026-07-03

Ported from upstream 2.4.9 + 2.4.10.

### Added

- **`SOPAInstructionTuning` step** (interface + config, synced from the BAMT vendored copy) — SOPA Stage-2 instruction fine-tuning (BLIP2 Q-Former + Phi-3 LLM); makes the `bedrock_batch_sopa` catalog DAG buildable here.
- **`TestSharedDagResolvability`** — every catalogued shared-DAG node must resolve to a routable builder (config-free). `KNOWN_UNROUTABLE` is empty (all DAGs route now that SOPA exists).

### Fixed

- **TSA = "Temporal Self-Attention"** corrected across all TSA step interfaces/configs + the `tsa_complete_e2e` catalog DAG (removed the wrong "Temporal Sequence Analysis/Attention", "Time Series Analysis", "Time-Series Aggregation", "Travel & Spend Analytics" expansions).

## [2.5.7] - 2026-07-03

Ported from upstream 2.4.8: add 2 temporal-split multi-task shared DAGs to the pipeline catalog.

### Added

- **`lightgbmmt_temporal_split_e2e`** (`shared_dags/mtl/temporal_split_e2e.dag.json`) — LightGBMMT MT e2e with `TemporalSplitPreprocessing` + dual testing/calibration eval.
- **`xgboost_mt_temporal_split_e2e`** (`shared_dags/mtl/xgboost_mt_temporal_split_e2e.dag.json`) — XGBoost-MT temporal-split e2e; adds the new `xgboost_mt` framework to the index.

`catalog_index.json`: `total_dags` 42 → 44, `frameworks` + `xgboost_mt`. Both load via `load_shared_dag` (11 nodes each).

## [2.5.6] - 2026-07-03

Ported from upstream 2.4.7: robust node→base step-name resolution + cross-DAG `compatible_sources` gap closure.

### Changed

- **Node base-step-name resolution is registry-driven, not suffix-list-driven.** `naming.resolve_base_step_name` / `split_job_type_suffix` strip trailing `_segment`s and validate the base against the actual step registry via `canonical_key` (replacing the `JOB_TYPE_SUFFIXES` membership check at `get_step_info`, `get_step_interface`, and `config_resolver._semantic_matching`). Any suffix resolves (job_type is open — `munged`/`sampling`/`tagging`/…); `XGBoostModel` is never mis-stripped.

### Fixed

- **9 `compatible_sources` gaps closed across 8 interfaces** — producer→consumer edges used in shipping BAMT DAGs that the consumer didn't list: `TemporalSplitPreprocessing` → MT training/eval; `XgboostMt`/`LightGBMMT` training → Package; `XgboostMtTraining` → Payload; `XgboostMtModelEval` → ModelCalibration; `PyTorchModelInference` → XGBoostModelInference. Re-audit across all BAMT DAGs: 0 gaps.

## [2.5.5] - 2026-07-03

Ported from upstream 2.4.5 + 2.4.6: config-resolver deep-dive Tranche 3 (allow-all job_types).

### Changed

- **`from_yaml` unknown-`job_type` → warn + base-spec fallback** (was: raise). Configs don't restrict `job_type` to the interface's variant set, so a legitimate open value (e.g. `CradleDataLoading`'s `munged`/`normal`) resolves to the base spec instead of crashing the build; real declared variants still tighten. (3a)
- **Closed `job_type` validators relaxed to allow-all (14 config classes)** — batch_transform, tsa_tabular_preprocessing, the 8 xgboost/lightgbm/lightgbmmt/pytorch model_eval & inference configs, and the 4 training configs (`{pretrain,finetune}`, `None`-handling preserved). Each now accepts any lowercase-alphanumeric-with-underscores value, matching the interface layer. (3b)

### Fixed

- **Preview/compile coverage gap** — `validate_dag_compatibility` now loads each node's interface with its resolved `job_type` (the same call the builder makes at compile), so interface-load failures surface in preview, not only at build time. (3c)

Verified: complete `munged_address_pytorch` DAG resolves 15/15 + validates clean; `from_yaml('CradleDataLoading', job_type='munged')` no raise.

## [2.5.4] - 2026-07-03

Ported from upstream 2.4.4: test-suite + offline-collection hardening (test/fixture/conftest only — no `src` behavior change).

### Fixed

- **torch offline-collection stubs completed (`tests/conftest.py`)** — added `torch.optim`, `torch.distributed`, `torch.distributed.fsdp`, `torch.utils.tensorboard` (imported at module level by the pytorch scripts) so those modules collect offline. `transformers`/`lightning` are deliberately NOT stubbed (their tests use `pytest.importorskip`; stubbing turns a clean skip into a hard collection error on the next heavy import).
- **Two tests/fixtures aligned to the 2.5.3 fail-loud contract** — `test_job_type_integration` asserts the new `ValueError` (was: silent base fallback); the `test_generator_integration` fixture corrected a stale nonexistent class name `CradleDataLoadConfig` → `CradleDataLoadingConfig`.

## [2.5.3] - 2026-07-03

Ported from upstream 2.4.3: second tranche of the silent-misresolution deep dive + validation hardening.

### Fixed

- **Assembler raises instead of fabricating a placeholder S3 URI** for a matched edge whose output spec is missing (`pipeline_assembler._instantiate_step`) — the old `s3://pipeline-reference/...` placeholder defeated the required-input guard.
- **`StepInterface.from_yaml` raises on an unknown `job_type`** when the step declares variants (was: silent base-spec fallback that dropped variant tightenings such as a dependency the variant makes required).
- **`compatible_sources` is alias/suffix-aware** (`Payload` ≡ `PayloadStep`, `Package` ≡ `PackagingStep`, + `LEGACY_ALIASES`) with a −0.1 penalty for out-of-list providers; `registration.step.yaml` / `tsa_model_calibration.step.yaml` now list canonical producer step types.
- **`load_configs` / `merge_and_save_configs` no longer silently drop or overwrite configs** — they raise listing unmaterializable declared configs / duplicate step names.
- **Validation module flags misresolutions as errors**: `validate_dag_compatibility` records a `config_errors` entry (→ `is_valid=False`) for a node bound to a config of a different step type, accepts `metadata` so explicit `metadata.config_types` mappings are exempt, and consults `has_builder_provider` so routable SDK-delegation steps (CradleDataLoading/Registration) are no longer false-flagged as unresolvable offline. `compile()` honors an explicit `skip_validation`.
- **`preview_resolution` consumers read the real `node_resolution` shape** (was `candidates[0]` → KeyError → empty preview).

Verified against `munged_address_pytorch/dag_NA.json`: complete config resolves 15/15 and validates clean; a partial 3-config subset raises `ResolutionError` naming all 13 unsatisfiable nodes. The 5 corresponding test files are synced from upstream.

## [2.5.2] - 2026-07-03

Correctness bugfix (ported from upstream 2.4.2): the config resolver no longer silently compiles a wrong pipeline from an incomplete config.

### Fixed

- **`StepConfigResolverAdapter.resolve_config_map` now resolves-or-raises.** It delegates every node to the existing safe `_resolve_single_node` (raises `ResolutionError` on no-match, warns below the 0.7 threshold), accumulates a `failed_nodes` list, and raises a `ResolutionError` naming **every** unsatisfiable node — instead of the old tier-3 catch-all that bound any unmatched node to the first loaded config (`next(iter(available_configs.values()))`) and the `except: return {}` that swallowed errors. Also honors `metadata.config_types` on the compile path and fails fast on duplicate generated step names.
- **`_semantic_matching` is gated by step-type agreement** — a keyword-category overlap is necessary but not sufficient; the config's real step type must equal the node's base step type, so `TabularPreprocessing_*` no longer matches `BedrockProcessingConfig` at the 0.7 threshold.
- **`StepCatalog.get_step_info` strips job_type suffixes** — a suffixed node (`TabularPreprocessing_training`) with `job_type=None` falls back to the base name via the `naming.JOB_TYPE_SUFFIXES`-guarded strip (`XGBoostModel` not mis-stripped), fixing the catalog-tier resolution miss.
- **`DynamicPipelineTemplate._create_config_map` asserts completeness** — raises `ConfigurationError` if any DAG node is unresolved and logs `N/total` rather than "resolved all N".

Verified against `munged_address_pytorch/dag_NA.json` (15 nodes): complete config resolves 15/15 with zero cross-step-type mis-bindings; a partial 3-config subset raises `ResolutionError` naming all 13 unsatisfiable nodes with zero silent wrong-bindings.

## [2.5.1] - 2026-07-03

Completes the 2.5.0 discovery migration to full parity with the internal `upstream` source, and fixes the compound-acronym / snake-case name-resolution drift the migration surfaced.

### Fixed

- **`builder_discovery.py` collapsed to synthesis-only (940 → 420 lines).** The 2.5.0 tranche deleted `steps/builders/` but had left the old AST/file-scan `BuilderAutoDiscovery` in place (it still referenced the deleted directory; harmless only because scanning a missing dir returns empty). It is now the registry-interface synthesizer that `upstream` ships — no dead file-scan code.
- **Robust interface-derived step-name resolution.** `BuilderAutoDiscovery` builds a normalized (`canonical_key`) index of registry keys, and `_resolve_to_canonical_name_for_indexing` gains a `canonical_key` fallback, so a compound-acronym name resolves under any casing: `load_builder_class("XGBoostMTTraining")` and `("XgboostMtTraining")` return the same synthesized class, and a config/script discovered as `tsa_training` / `xgboost_mt_training` / `package` folds into its canonical registry row instead of creating a phantom duplicate index entry (index 67 → 57). The registry keys ARE the authored `.step.yaml` `step_type` values, so this needs no hand-maintained `COMPOUND_ACRONYMS` edit and can't drift.
- **TSA configs standardized.** `config_tsa_model_eval_step.py` / `config_tsa_preprocessing_step.py` / `config_tsa_model_calibration_step.py` drop their module-level `load_step_interface(...)` + `get_script_contract()` override and inherit the Design-B contract-from-interface path from the base config (matching ModelCalibration / XGBoostModelEval / TabularPreprocessing); `TSAPreprocessingConfig.job_type` now defaults to `"training"`.

## [2.5.0] - 2026-07-03

The **interface-first discovery + folder deletion** tranche: step discovery now sources contract and spec data exclusively from the unified `.step.yaml` interfaces (+ configs + scripts + registry), the residual per-step data folders are physically deleted, and a single canonical interface loader replaces the two parallel load paths. This completes the Design-B discovery migration (Cursus Simplification Trail, FZ 31): a step is an interface + config + script, and nothing scans `builder_*.py` / `*_contract.py` / `*_spec.py` files anymore because those files — and now their folders — are gone. Mostly internal (discovery internals, one new facade method); two breaking changes are called out under **Removed**.

### Added

- **`StepCatalog.get_step_interface(step_name, job_type=None) -> Optional[StepInterface]` — the single canonical interface loader.** Wraps the cached `cursus.steps.interfaces.load_interface` with null-safety (never raises → returns `None`) and, in ONE place, the job_type-suffix fallback: a variant-suffixed name with no dedicated YAML (e.g. `ModelCalibration_calibration`) strips a known suffix (`naming.JOB_TYPE_SUFFIXES` — deliberately excludes `"model"` so `XGBoostModel` is never mis-stripped) and retries the base name as that `job_type`. `load_contract_class`/`load_spec_class` collapse to one-line views (`iface.contract` / `iface`) over it, so all ~15 consumers inherit the fallback for free and contract↔spec always come from one `_sync_and_align`-validated object (no double load).

### Changed

- **`contract_discovery.py` / `spec_discovery.py` rewritten interface-first.** All AST / `importlib` / `glob` file-scan machinery deleted; `ContractAutoDiscovery` and `SpecAutoDiscovery` now derive every result from `registry.get_all_step_names()` + `load_interface`. The pure serializers (`serialize_contract`, `serialize_spec`), smart-specification selection (`create_unified_specification`, `validate_logical_names_smart`), and their helpers are preserved verbatim — they already duck-type on `ContractSection`/`StepInterface`. `get_job_type_variants` now returns `StepInterface.variants` keys; `find_specs_by_contract` fans out variants via `load_interface(name, job_type=…)`. Return shapes are unchanged (drop-in for the validation/CLI/MCP consumers).
- **`steps/__init__.py` made resilient.** No longer star-imports the (now-deleted) `builders`/`contracts`/`specs` subpackages; `import cursus.steps` no longer depends on those folders existing.
- **`io_view.describe_step_patterns` is 2-tier and folder-free.** The AST/glob scan of `steps/builders/builder_*_step.py` (`_scan_builder_overrides`) is deleted; per-axis `custom_override` flags now come from inspecting the loaded/synthesized builder class, with a declarative-shell note otherwise. The `note` + per-axis keys the MCP/CLI consumers read are preserved.
- **`registration.step.yaml` no longer declares `MODS_WORKFLOW_EXECUTION_ID` as a required env var.** Registration is a SAIS-SDK sink step whose runtime env is supplied by the execution doc; the var is read only by `edx_uploading.py`, and execution-doc generation never references it, so the declaration was a misdeclaration surfaced (and now fixed) by the interface-first env-vars conformance check.
- **Workspace scaffolding updated to Design B** — `registry/hybrid/setup.py` and `cli/registry_cli.py` no longer create or require `steps/{builders,contracts,specs}` directories; a new workspace step is a `.step.yaml` interface + config + script.

### Removed

- **BREAKING — `src/cursus/steps/{builders,contracts,specs}/` folders deleted.** The `from cursus.steps.builders import <X>StepBuilder` public import surface (a PEP-562 lazy shim) is removed. Builder classes are synthesized from the registry at runtime; obtain one via `StepCatalog().load_builder_class("<StepName>")`. `S3PathHandler` lives in `cursus.steps.utils`; `StepBuilderBase` in `cursus.core.base.builder_base`. The dead `ValidationResult`/`ScriptAnalyzer` re-export from `steps/contracts` is gone (real homes: `core.base.contract_base`, `validation.alignment.analyzer.script_analyzer`).
- **BREAKING — `StepCatalog.get_contract_entry_points()` key semantics changed.** Keys are now the PascalCase canonical step name (e.g. `TabularPreprocessing`) instead of the old file-stem (`tabular_preprocessing_contract`). This aligns with the step-name lookups its CLI/validation consumers already perform and fixes a lookup that previously returned an empty map under the dead file-scan.

## [2.4.0] - 2026-07-03

### Added

- **Seven XGBoost-MT + TSA step types (new) — `steps/interfaces/{xgboost_mt_training, xgboost_mt_model_eval, tsa_preprocessing, tsa_tabular_preprocessing, tsa_training, tsa_model_eval, tsa_model_calibration}.step.yaml` + paired `configs/config_*_step.py`.** Brings previously project-only step types into the catalog as native steps (**56 registry rows**): `XgboostMtTraining` / `XgboostMtModelEval` (multi-task XGBoost) and the TSA family — `TSAPreprocessing`, `TSATabularPreprocessing`, `TSATraining`, `TSAModelEval`, `TSAModelCalibration`. Each is a Design-B step (one `.step.yaml` interface + one `<StepType>Config`; builder synthesized at runtime, registry derived by construction). `XgboostMtTraining`/`TSATraining` are `Training`, the rest `Processing`. No scripts ship in the package — these are model-dependent / project-provided steps whose scripts live in the consuming project's `dockers/`; the `.step.yaml` + config make them discoverable, resolvable, and constructible so a project supplies only the script. Configs use relative imports only, so they resolve unchanged. Verified: `cursus validate step-interface` passes (0 errors) on all seven, all seven auto-derive into `STEP_NAMES` with the correct `config_class` + `sagemaker_step_type`, and the registry golden-snapshot was re-based (delta = exactly the seven new rows).
- **`hyperparameters_xgboost_mt.py` (new) — `steps/hyperparams/`.** The multi-task XGBoost hyperparameter model (a `ModelHyperparameters` subclass) that `XgboostMtTraining` relies on; the TSA hyperparameter models (`hyperparameters_tsa.py`, `hyperparameters_dual_sequence_tsa.py`) already shipped.

## [2.3.0] - 2026-07-01

### Added

- **New `PiperMetricGeneration` step type — `steps/{configs/config_piper_metric_generation_step.py, interfaces/piper_metric_generation.step.yaml, scripts/piper_metric_generation.py}`.** A new user-facing step (bringing the catalog to **47 steps**), a drop-in peer of `ModelMetricsComputation` that pipeline authors can add to a DAG: listed by the step catalog and buildable via a runtime-synthesized `PiperMetricGenerationStepBuilder`. The `.step.yaml` declares `node_type: internal`, `sagemaker_step_type: Processing` (sklearn compute), a required `eval_output` input (compatible sources: the XGBoost / LightGBM(MT) / PyTorch model-eval/inference steps plus `TabularPreprocessing` / `CradleDataLoading` / `RiskTableMapping` / `CurrencyConversion`), and a `metric_output` output. `PiperMetricGenerationConfig` (a `ProcessingStepConfigBase` subclass) adds three required fields — `id_name`, `label_name`, `variant_model_id` — with a validator requiring at least one of `score_field` / `score_fields`, supports single-task / multi-task / comparison (control-series) scoring, and its `get_environment_variables` emits the PIPER contract env vars (`VARIANT_MODEL_ID` / `CONTROL_MODEL_ID` / `DATASET_TYPE` / `METRICS_TO_RENDER` / `PIPELINE_NAME`). The `piper_metric_generation.py` script loads the eval predictions, requires the ground-truth label column (erroring with guidance if absent — e.g. `PyTorchModelInference` drops labels), recomputes ROC/PR curves and data statistics with scikit-learn, and writes PIPER `.metric` JSON files plus paired 2-column data CSVs flat to the output root (with `_SUCCESS` / `_HEALTH` / `_FAILURE` markers).
- **Opt-in `contract.output_path_token` override — `core/base/step_interface.py`, `core/base/builder_templates.py`, `steps/interfaces/io_view.py`, `mcp/tools/author.py`.** Re-introduces a declarable `ContractSection.output_path_token` field (`Optional[str] = None`, previously removed) as an explicit, default-off escape hatch: when set to a non-empty string it is used **verbatim** as the output-destination S3 prefix segment instead of the step-name-derived `canonical_to_snake(step_type)` token — changing only the emitted S3 output folder, not the SageMaker/DAG step name. Honored by the Processing/Training/Transform output-path handlers via `getattr(b.contract, "output_path_token", None)`, surfaced by `describe_step_patterns` (which reports an `output_path_token_source` of `"override: contract.output_path_token"` vs `"derived: canonical_to_snake(step_type)"`), and documented in the `author.rules` packaging guidance. It exists for external consumers keying off a fixed S3 folder — e.g. PIPER scanning `<pipeline>/Model_Metric_Generation_Step/` for `.metric` files (the value `piper_metric_generation.step.yaml` sets). Fully backward-compatible: the default `None` preserves every existing step's derived path.
- **Kiro workflow runtime — run the cursus dynamic workflows on `kiro-cli` — `src/cursus/mcp/workflows/kiro/`.** The shipped workflow scripts (`cursus-author-step.js`, `cursus-configure-pipeline.js`) target the Claude Code `Workflow` runtime: the `agent()` / `parallel()` / `pipeline()` / `phase()` primitives are injected by the Claude Code host, so the scripts are inert under `kiro-cli` (there is no `node <workflow>.js`). This adds a small, dependency-free Node runtime that re-implements those primitives on top of `kiro-cli`, so the **unmodified** workflow scripts run on Kiro — same phases, same gates, same MCP tools; each `agent()` call becomes one `kiro-cli` turn (a fresh sub-agent, matching Claude Code semantics). Files: `kiro-workflow-runtime.js` (the engine — primitives + `args`/`budget`, with schema-forced output emulated by a JSON-schema prompt suffix + parse/validate + bounded re-prompt, returning `null` on terminal failure), `run-workflow.js` (the CLI — loads an unmodified script, binds the primitives as globals, runs it; result to STDOUT, progress to STDERR; the Kiro counterpart to the Claude Code `Workflow` tool), `test-runtime.js` (offline tests via mock `kiro-cli`), and `README.md` (rationale, usage, transport, and a fidelity table vs the Claude Code host). Verified: offline tests pass, both real workflows load and run unmodified, and a live smoke test against `kiro-cli 2.10.0` returned schema-validated output. Fidelity caveat: the runtime *validates and re-prompts* schema output rather than tool-forcing it, so the workflows' own executable oracles (`py_compile`, `yaml.safe_load`, `json.load`, `validate.deps_resolve`) remain the real gates.
- **Schema-output shape coercion for the Kiro runtime.** Since Kiro can't tool-force structured output (the Claude Code host does), a capable model can still return the right JSON in the wrong container — e.g. `[{...}]` (single-element array) when the schema wants an object, staying stuck on that shape across generic re-prompts. The runtime now auto-coerces before validating (unwrap single-element array → object; wrap bare object → array when the schema wants one) without changing field values, and the re-prompt names a container mismatch loudly. `maxSchemaRetries` default raised 2 → 3, with a new `--schema-retries` flag. (This is distinct from a *weak* model returning prose — that needs a stronger model, e.g. `--model <id>`.)
- **Harness-conditional Resolve split for the Kiro runtime (Claude Code path unchanged).** A follow-on SAIS run surfaced a shape failure coercion cannot fix: Opus 4.8 answered `cursus-author-step`'s 15-field `PLAN_SCHEMA` (which discusses producer/NEW/consumer nodes) with a **multi-element** array `[{...},{...},...]` — one plan object *per node* — plus a per-turn timeout on the oversized turn. A genuinely multi-element array is ambiguous to unwrap, and re-prompts didn't move it. Fix is workflow-side and keyed to host capability: `run-workflow.js` now sets `__workflowHost.toolForcesSchemaOutput = false` on the sandbox, and `cursus-author-step` reads it — on the Claude Code `Workflow` host (marker undefined ⇒ tool-forcing) Resolve stays ONE `StructuredOutput`-forced `PLAN_SCHEMA` turn, byte-for-byte unchanged; only on the non-tool-forcing Kiro runtime does it split into three small single-object turns (locate → triage → identity, merged into the same `PLAN_SCHEMA`-shaped plan). Each small turn has one unambiguous object shape, so no per-node array and a far smaller/faster turn. New `kiro/test-author-step-e2e.js` runs the real workflow on the Kiro runtime to a green step and asserts the combined `PLAN_SCHEMA` turn is never emitted (a mock sentinel reproduces the SAIS array trap if it ever were); a separate harness check proves the Claude Code single-turn path is unchanged. Takeaway: a large schema a non-tool-forcing host can't reliably shape should be **decomposed by the workflow for that host**, not forced through coercion.
- **Explicit shape directive + identical-array coercion for the Kiro runtime (runtime-level, covers both workflows).** A further SAIS run showed Opus 4.8 on kiro-cli 2.5.0 wraps its answer in an array even for the *small* split `locate` schema — a general array bias the raw JSON Schema suffix does not override, appearing on the first attempt. Fixed in the runtime (not per-workflow) so it applies to every schema-gated turn in both workflows at once — `cursus-author-step`'s locate/triage/identity and `cursus-configure-pipeline`'s DagCheck + per-node Validate (which showed the same multi-element-array failure) — with no workflow edits and no model change. Three layers: (1) `shapeDirective(schema)` prepends an explicit natural-language instruction plus a `{ "field": ..., ... }` skeleton to the INITIAL prompt ("Return EXACTLY ONE JSON object … do NOT return a list of candidates or alternatives … pick the single best"); (2) `coerceToSchema` now also unwraps a multi-element array whose elements are all IDENTICAL (the model emitted the same answer N times) to the first object, while a multi-element array of DIFFERENT objects stays ambiguous and falls through; (3) the re-prompt names an object-schema-answered-with-an-N-element-array as exactly that ("a JSON ARRAY of N items … pick one, not a list"). Field values are never changed. Tests: 46 runtime (adds `shapeDirective` + multi-element coercion units and two e2e cases) + 12 author-step e2e.
- **Tolerant JSON parse + cursus MCP wiring for the Kiro runtime's post-Resolve phases (RC#3).** After the Resolve split, a SAIS run (Sonnet 4.6) cleared Resolve and authored all 3 artifacts, but the post-Resolve phases (Challenge, AlignEdges, validate, check_script, resolve, preflight, and configure-pipeline's DagCheck + per-node Validate) still failed on schema/JSON-parse. Two runtime-level fixes, covering both workflows with no workflow edits. (A) **Tolerant JSON parse** — new `tolerantParse(text)` tries strict `JSON.parse` FIRST (valid JSON is returned untouched and never mutated), and only on failure applies SAFE context-aware syntax repairs (strip `//` and `/* */` comments, remove trailing commas, convert single-quoted strings, normalize smart/curly quotes, map Python `True`/`False`/`None`/`NaN`/`Infinity`, quote unquoted keys) then re-parses; each repair provably can't corrupt valid JSON since valid JSON already returned. It reports the defects fixed, so the parse-failure re-prompt now names them instead of echoing an opaque `SyntaxError` the model repeated across retries. (B) **cursus MCP wiring** — the post-Resolve phases are *relay-tool-result* prompts ("call `author.*`/`validate.*`/`steps.io`/`catalog.*` and return its JSON"); the ACP client opened `session/new` with `mcpServers:[]` and no server was registered anywhere, so kiro-cli had no tools and fabricated the JSON. The runtime now accepts `mcpServers`/`mcpCursus` config, the ACP client encodes them into `session/new` (env as the ACP `[{name,value}]` array), and `run-workflow.js` adds `--mcp-cursus` (auto-register `cursus mcp serve`), `--mcp-python <cmd>`, and repeatable `--mcp-server '<json>'`; headless warns once if servers are set without `--agent`. Registering the server is the durable fix; the parse/coerce/re-prompt layers remain the safety net. Tests: 67 runtime (adds `tolerantParse` units for every defect class + the strict-first "valid JSON untouched" guarantee, ACP `mcpServers` wire-shape, `mcpCursus` auto-register, and an ACP e2e asserting `--mcp-cursus` reaches `session/new`) + 12 author-step e2e.
- **Guard the kiro-cli 2.5.0 ACP-MCP crash (SAIS Run 8).** The RC#3 MCP-wiring delivered via ACP `session/new` works on 2.10.0 but **crashes** the frozen 2.5.0 ACP server (it exits code 0 the moment it receives a non-empty `mcpServers` payload), and `--mcp-cursus` is a no-op in headless (kiro-cli reads MCP from its `--agent` config, not a per-call flag). Fixed by gating on `allowNewFlags` (the same version-skew pattern as `--effort`): under `--legacy-kiro`, the ACP client DROPS the `session/new` `mcpServers` payload entirely (one-time warning) rather than sending the crash trigger; and the runtime, when it can't deliver MCP (headless without `--agent`, or acp+`--legacy-kiro`), prints the exact `~/.kiro/settings/mcp.json` snippet + the working headless-`--agent` invocation instead of steering to `--transport acp`. Docs corrected: `--mcp-cursus` is kiro-cli ≥ 2.10.0 (+acp) only; on 2.5.0 the sole MCP route is a static `~/.kiro/settings/mcp.json` + `--transport headless --agent`.
- **Tool-forcing structured output — `submit_result` MCP server + SAIS probe (offline-safe half; not yet wired).** The heuristic layers (shape directive, tolerant parse, coercion, re-prompt) *salvage* free-text JSON; they don't *force* it the way the Claude Code host does (a `StructuredOutput` tool call whose parameters ARE the schema). Internal code-search research (4 independent kiro-cli ACP client implementations + the ACP spec) established that tools reach kiro-cli ONLY via MCP servers (no inline ACP tool declaration), and on 2.5.0 that must be a static config + headless `--agent`. This ships the analog: **`submit-result-server.js`**, a dependency-free stdio MCP server exposing one tool `submit_result` whose `inputSchema` IS the phase schema — kiro-cli validates the model's arguments against it before dispatch (killing the array-bias at the source), and the runtime (being the server) receives the validated arguments directly, no stdout scraping. `runtime.buildSubmitResultConfig()` emits the 2.5.0-safe static `mcp.json` + prompt suffix; **`run-toolforce-probe.js`** is a one-command SAIS probe that reports GREEN/RED on whether 2.5.0 headless actually runs the tool loop. Shipped offline-safe and mock-tested (in-process handler + a real stdio round-trip), but deliberately **NOT wired into `agent()` as the primary path** until the probe returns green on SAIS — the same "don't ship an unprobed wire mechanism" discipline the Run-8 crash taught. Tests: 85 runtime + 12 author-step e2e.
- **Drop MCP on the legacy (kiro-cli 2.5.0) path + decompose the post-Resolve phases.** A SAIS run (Run 10) proved kiro-cli 2.5.0 loads NO MCP servers in any mode — headless ignores `~/.kiro/settings/mcp.json` (built-in tools only) and its ACP crashes on registration (Run 8). So MCP tool-forcing is dead on that build; it works only on kiro-cli ≥ 2.10.0 (validated locally). Two harness-conditional changes result (Claude Code path byte-for-byte unchanged). (1) **MCP dropped on the legacy path** — with `--legacy-kiro`, `runtime._mcpServers()` returns `[]` with a one-time warning and the workflow phases use their built-in `cursus` CLI fallback (passing servers would be a no-op/crash on 2.5.0); on ≥ 2.10.0 MCP still works (headless via `--agent` config, ACP via `session/new`). (2) **Post-Resolve schema decomposition** — the AlignEdges and Validate re-resolve phases carry `edges[]` arrays-of-objects that triggered the multi-element-array bias on 2.5.0 (the AlignEdges/resolve failures in Runs 5-6); on the non-tool-forcing host `cursus-author-step` now scores ONE edge per turn (`EDGE_ALIGN_SCHEMA` + a small `ARITY_SCHEMA` turn; `EDGE_RESOLVE_SCHEMA`) and assembles the `ALIGN_SCHEMA`/`RESOLVE_SCHEMA` shape in plain code, so no `edges[]` array is ever requested from the model. Vacuously-ready for a source/sink/singular node, matching the single-turn contract. Under Claude Code both stay single tool-forced turns (gated on `HOST_TOOL_FORCES_SCHEMA`). Tests: 85 runtime + 14 author-step e2e (now asserts the Kiro path runs one per-edge align turn per real edge and never emits the single-turn array prompt; a separate harness check confirms the Claude Code single-turn path is unchanged).
- **Fix a control-flow regression in the per-edge AlignEdges decomposition (SAIS Run 11).** When the per-edge align turns returned non-JSON on 2.5.0 (Sonnet emitted `[step_type=...` / `[OPTIONS]`), the decomposition still built a non-null `align` with `ready:false`, so the workflow short-circuited before Author and produced 0 artifacts — whereas the pre-split single-turn path returned `null` on failure and PROCEEDED to Author (the authoritative edge gate is the post-write re-resolve at Stage 4c + the executable oracles). Fix: Stage 2 now distinguishes INCONCLUSIVE (fewer edges scored than expected → `align=null`, proceed) from DEFINITIVE (all edges scored, some don't resolve → short-circuit); the short-circuit condition tightened from `!align.ready` to `align.ready === false` so a null align never blocks. Stage 4c (the final `green` verdict) stays strict — `both_edges_resolve` is false when edges can't be scored, correctly withholding green. Tests: +3 e2e (17 total) including a Run-11 regression case that returns non-JSON for the per-edge align turns and asserts a step is STILL authored.
- **Reroute the failing relay-tool-result phases through the cursus CLI on kiro-cli 2.5.0 (the path to green WITHOUT MCP).** Run 11's `[OPTIONS]` / `[step_type=...` output was the model ALREADY reaching for the cursus CLI via its built-in shell/execute tool (which works on 2.5.0 — Author writes files, the parse phase runs `py_compile`), just forgetting `--format json` and args. Since MCP is dead on 2.5.0 (Run 10) but the CLI + shell tool are alive, the relay phases are rerouted to prescriptive CLI commands — harness-conditional (Claude Code keeps its MCP-tool prompts byte-for-byte; only the non-tool-forcing Kiro path reroutes). `cliJson(cmd)` returns the mandatory clean-JSON recipe (`PYTHONNOUSERSITE=1 python3 -m cursus.cli <cmd> --format json`, stderr discarded, piped through `sed -n '/^{/,$p'` — the sed filter is REQUIRED because the `sagemaker.config INFO` lines print to stdout and survive stderr-discard, verified live). Per phase: Validate/validate → `validate step-interface` (mapping `results[0]`, since top-level errors/warnings are counts); Challenge → `catalog show` + `steps io`; AlignEdges → `steps io <adjacent>`; Guide → `strategies knobs` + `steps io` + `steps patterns` + `catalog fields` (author.rules/config_constraints have no CLI → reason from the exemplar per HABIT-1); check_script → no CLI → `status:"skipped"` (publish path permits skip, PARSE oracle backstops, re-runs at CR); Validate/resolve → **deleted the phantom `cursus validate deps-resolve` command (it does not exist and hard-errored)** and reason the edge from two `steps io` JSON blobs; Preflight → reason constructibility from the parsed `.step.yaml` + specs. The `both_edges_resolve` + `constructible` green gates are reasoned/relaxed on the frozen host (noted in-run) and re-verified by the real resolver/preflight at CR. Tests: 85 runtime + 20 author-step e2e — adds a LIVE proof that the `cliJson` recipe yields parseable JSON from the real cursus CLI and that a bare stderr-discard does NOT (the sed filter is mandatory); a CC-path check confirms zero CLI-reroute leakage and all MCP calls kept.
- **Fix the missing-required-field failures the CLI reroute exposed (SAIS Run 13).** With the reroute, Run 13 was major progress — `check_script` and `preflight` passed for the first time (`constructible=True`) — and the failure mode SHIFTED from "unparseable JSON" to "required property missing" (`$.holds`/`$.edge`/`$.ok`/`$.axis`): the model now returns valid JSON but drops a required field. Two fixes. (A) **Named missing-field re-prompt** (`kiro-workflow-runtime.js`): when the only errors are missing-required-properties, the re-prompt names the exact dropped keys loudly with each one's expected type/enum ("missing REQUIRED field(s): holds (boolean); …") instead of burying `$.holds: required property missing` in an 8-line list the model ignored. (B) **Inject the edge label the runtime already knows** (`cursus-author-step.js`): the per-edge align/resolve turns kept dropping `edge` — the one field the runtime dictates ("ALIGN EXACTLY ONE EDGE: <edge>") — so `edge` was removed from `EDGE_ALIGN_SCHEMA`/`EDGE_RESOLVE_SCHEMA` `required` and is injected after parse, making a dropped `edge` un-failable (deterministic, not model-dependent; the CC single-turn schemas still require it). Tests: 86 runtime (adds a missing-field e2e that recovers only when the re-prompt names the field) + 20 author-step e2e (the mock now OMITS `edge` and the workflow still reaches green).
- **Two research-backed structured-output levers for the multi-field-synthesis failures (SAIS Run 14).** Run 14 showed the named re-prompt + edge injection did NOT move the 6 remaining phases (Challenge/Guide/validate/gaps/edge-align/edge-resolve) — the failure is whole-object SYNTHESIS (a multi-field object with array members), not a single nameable dropped field. A search (internal SAGE 372052/373135/373070 + code patterns like VelociraptorDocker `_coerce_to_schema`; external fill-in-the-blank / few-shot / json-repair) converged on two runtime-level levers, one edit covering all phases, harness-conditional, and structurally unable to fake a green. (1) **Typed-example skeleton** in `shapeDirective()` — replaces the bare `{ "holds": ... }` with a recursive TYPED skeleton (string→`"..."`, boolean→`true`, number→`0`, array→`[ <elem> ]`, enum→first literal, nested→recursed) plus "keep EVERY key, replace only the values" and an explicit "EVERY one of these keys is REQUIRED: …" line; because every key is pre-written the model can't drop the field set and the pinned braces re-anchor object-not-array (the most-cited cross-model shape lever, SAGE 373135/372052). (2) **`backfillDefaults()` before validation** — injects `[]` for omitted array props and `""` for omitted OPTIONAL string props, but NEVER a required boolean/enum/number/string (those carry the decision and must come from the model, so a dropped verdict still fails validation — cannot fake a green; unit-tested). Plus a small third lever: Guide `axis` injection (runtime-dictated, like `edge`; fixes the 0/6 `$.axis`-missing failures). Tests: 98 runtime (typedSkeleton units; backfill safety units proving required scalars are never fabricated while omitted arrays/optional-strings are filled; an e2e where the model omits an optional array and it backfills to `[]` first-try) + 20 author-step e2e; CC path re-confirmed single-turn + green.
- **`cursus dag resolve` CLI + move validate/resolve off the model into code — breaks the SAIS Run-15 "hard ceiling."** Run 15 was the 3rd consecutive negative (all three prior fixes were prompt-shaping); the two remaining green blockers (`validate.ok`, `resolve.both_edges_resolve`) turned out to be a RELAY (copy a CLI's JSON) and a DETERMINISTIC formula (the 6-component resolution score), not open-ended synthesis — so the fix is to move that burden off the model into runtime code, which no prior run tried. **New `cursus dag resolve <step>... --format json`** (`src/cursus/cli/dag_cli.py`) runs the REAL `UnifiedDependencyResolver` (same weights, 0.5 threshold, and `SemanticMatcher` the compiler/CI use) and emits per-edge `{consumer, dependency, provider, score, resolves}`. In `cursus-author-step.js` (non-tool-forcing / Kiro branch only, harness-conditional — the Claude Code path keeps its MCP `validate.step_interface` + `validate.deps_resolve` unchanged): the `validate` turn now returns `cursus validate step-interface --format json` VERBATIM and the runtime projects `results[0]` in code; the re-resolve turn FETCHES `cursus dag resolve` JSON and `matchResolvedEdge` maps the real per-edge scores onto producer→NEW / NEW→consumer, computing `both_edges_resolve` in code (a new `RELAY_JSON_SCHEMA` makes these turns parse-only). This preserves H4 — both gates run the same real oracles CI runs, no model arithmetic and no drifting JS re-implementation of the resolver. Tests: new `tests/cli/test_dag_cli.py` (5, including a drift-guard asserting the CLI score equals the resolver's score for the same specs); +6 runtime units for `matchResolvedEdge` (name-matching, best-of-ties, sub-threshold `resolves:false`, null-when-unscored) = 104 runtime; the author-step e2e reaches GREEN via the CLI-reroute path = 20.
- **`cursus dag resolve` relay accepts a bare array on frozen kiro-cli.** Follow-on to the Run-15 CLI reroute: `cursus-author-step`'s resolve-relay turn loosened `RELAY_JSON_SCHEMA` from `{ type: 'object' }` to `{}` (accept any valid JSON), and the handler now maps a top-level array (`Array.isArray(raw) ? raw : []`) in addition to `{ edges: [...] }`, so the model relaying `cursus dag resolve` output as a bare array parses on kiro-cli 2.5.0. Kiro-path only; the Claude Code path is unchanged.
- **Version-skew docs corrected for locked-down environments.** Upgrading `kiro-cli` is not always possible — a no-egress environment (e.g. a Red-certified VPC whose install pins an old version) cannot pull a newer release, so the README no longer prescribes "install >= 2.10.0" as the universal fix. In such environments the fix must be version-independent: pin a capable model with `--model <id>`, and/or reduce what one turn demands (trim the prompt, split a large schema into smaller sequential turns); upgrading is framed as a separate-machine benchmark option only.
- **Runtime warns when `--effort` is dropped on old kiro-cli, and the README documents the capability gap.** A frozen `kiro-cli 2.5.0` build has no `--effort` flag, so `--legacy-kiro` drops it — which silently under-powers any step that requested a high reasoning effort (e.g. the `cursus-author-step` **Resolve** phase, which needs a strong model for a 15-field schema off a large prompt; a weak model returns prose instead of JSON and the schema turn fails). The runtime now emits a one-time warning when this happens, and the README's "Version skew" section states plainly that a fair authoring run needs `kiro-cli >= 2.10.0` (simple/low-schema workflows still run on 2.5.0 with `--legacy-kiro`).
- **Kiro workflow runtime handles kiro-cli version skew (2.5.x ↔ 2.10.x).** The runtime was captured on `kiro-cli 2.10.0`, but frozen environments (e.g. the SAIS SageMaker snapshot) run `2.5.0`, which lacks some flags/entry points. Both transports now degrade cleanly: `--legacy-kiro` emits only the 2.5.0-safe flag set (drops `--effort` + granular `--trust-tools`; keeps `--no-interactive`/`--trust-all-tools`/`--agent`/`--model`); `--acp-entry chat-binary` (+ `--kiro-chat-bin`) drives ACP through the separate `kiro-cli-chat` binary that 2.5.0 ships instead of the `kiro-cli acp` subcommand; and the ACP `protocolVersion` is requested (`--acp-protocol`, default 1) but re-negotiated to whatever the server echoes in `initialize`. Offline arg-shape tests assert the 2.5.0-safe shape without needing a live old binary. See the runtime README's "Version skew" section.
- **Two transports for the Kiro workflow runtime — headless and ACP — selectable with `--transport`.** `headless` (default) spawns one `kiro-cli chat --no-interactive` process per `agent()` turn (independent processes → true `parallel()` fan-out). `acp` holds ONE long-lived `kiro-cli acp` process and runs one Agent-Client-Protocol session per turn (JSON-RPC 2.0 over newline-delimited stdio), via a dependency-free client (`kiro-acp-client.js`) hand-rolled after the internal `AGIArsenalConsole` pattern — no `@agentclientprotocol/sdk`. The ACP path does `initialize` (protocol version 1) → `session/new` → `session/prompt`, accumulates streamed `session/update` `agent_message_chunk` text to the `stopReason`, auto-approves `session/request_permission`, and ignores `_kiro.dev/*` notifications; the wire contract was captured live from `kiro-cli 2.10.0`. Use ACP to amortize process/agent-init cost across long sequential runs. Caveats: `--agent`/`--model`/`--effort` are session-level under ACP (per-call overrides apply only in headless), and ACP turns are serialized on the single process (headless gives true wall-clock parallelism). Verified with a live workflow run over ACP (handshake, schema-validated turn, parallel turns over one persistent process, clean teardown) plus a mock-ACP-server offline test.

## [2.2.1] - 2026-06-30

### Fixed

- **`cursus-author-step` no longer silently clones the exemplar and drops a new step's distinguishing feature.** A known-answer run (authoring `TSATabularPreprocessing` against a real pipeline whose expected interface was known) exposed a fidelity miss: the workflow's Resolve/Challenge phases correctly identified *why* the node was a NEW step (it needs a `preprocessor` output the generic `TabularPreprocessing` exemplar lacks), but the Author phase copied the exemplar shape and dropped that output — reproducing the exemplar and defeating the step's reason to exist. Fixed in three parts, none touching the compile/runtime path (the workflow is a reference `.js` orchestration script): (1) **Resolve now emits a required `divergences_from_exemplar[]`** — the concrete deltas that justified rejecting the reuse rung (new prompt step + schema field); if a step is `new_step` with no nameable divergence, the reuse ladder was mis-applied. (2) **Author is reframed from "copy the exemplar shape" to exemplar-PLUS-deltas** — the divergences are injected as REQUIRED additions the agent applies on top of the exemplar and then self-confirms, with an explicit "a dropped divergence is a BUG" instruction. (3) **Validate gains a divergence-fidelity gate** (bounded fix loop) that reads the authored artifacts, confirms every declared divergence is actually present (e.g. an "ADD a `preprocessor` output" delta needs both `contract.outputs.preprocessor` and `spec.outputs.preprocessor` plus the script writing it), and loops a targeted fix if any is missing. A step with a missing divergence is no longer reported green, and the run report surfaces `divergences_required` / `divergences_present` / `divergences_missing`.

## [2.2.0] - 2026-06-30

The **agent-authoring surface** tranche: the tooling and orchestration that let an agent *create* a step and *configure* a pipeline by tool-calling, plus the framework hardening those flows exposed. Design B made a step small enough (one `.step.yaml` + one config + one script, builder synthesized at runtime) that authoring becomes a guided, gateable loop rather than a multi-file ritual; this tranche ships that loop. Predominantly additive (new `author.*` MCP namespace, two shipped dynamic-workflow scripts, enriched validators) with a handful of validator-hardening changes called out under Changed. No public pipeline-authoring API change.

### Added

- **`author.*` MCP namespace (new) — `src/cursus/mcp/tools/author.py`.** Six tools that turn the "add a step" SOP into a tool-calling loop: `author.checklist(sagemaker_step_type[, step_assembly])` (the ordered SOP as data, with the bound `PatternHandler` + a same-phase exemplar DERIVED from the live `strategy_registry` so it can't drift), `author.rules(topic)` (a restriction oracle that introspects the live enforcement objects for `naming` / `packaging` / `sdk_carveout` / `reuse_class` / `closure`), `author.config_constraints(step_name)` (each config field's `allowed_values` + `case_sensitive` + declared type + the required-no-default set), `author.preflight_config` (offline `model_validate` of a value set), `author.preflight_step` (the offline constructibility proof — the same gates CI runs: interface + registry parity + `RegistryBindingValidator` B3 + routability), and `author.check_script` (script↔contract alignment in BOTH directions — the builder's `--args` are argparse'd and required env vars are read). Registered via `mcp/registry.py`; served alongside the existing namespaces.
- **`validate.step_interface` promotion.** The author-time interface gate is a verbatim promotion of `cursus validate step-interface` into the MCP surface, so the agent gate, the CLI gate, and the CI `validate step-interface --all` gate are literally one code path (green-local ⇒ green-CI).
- **Shipped dynamic-workflow orchestrators — `src/cursus/mcp/workflows/`.** Two Claude Code dynamic-workflow scripts (reference/runnable artifacts, not importable Python) that drive the loop deterministically, plus a self-contained README. `cursus-author-step.js` is **DAG-driven**: its input is a `PipelineDAG` in which one node is a NEW step sitting between a producer and a consumer, and it aligns both edges (the new dependency-spec to the producer output-spec, the new output-spec to the consumer dependency-spec — each must resolve at the resolver's 6-component compatibility score ≥ 0.5, editing the consumer's `compatible_sources` + output aliases for room) before authoring, then gates on `validate.step_interface` + `author.check_script` + an executable py_compile/yaml parse oracle + a whole-DAG completeness critic. `cursus-configure-pipeline.js` drives `DAGConfigFactory` config.json generation gated per-node by `author.config_constraints` + `author.preflight_config`, then a cross-node **DagCheck** phase.
- **ScriptAnalyzer reverse-alignment checks — `validation/alignment/analyzer/script_analyzer.py`.** Beyond the existing forward (script-uses → contract-declares) direction, the analyzer now runs reverse checks (contract-declares → script-parses/reads): `extract_argparse_flags`, `extract_env_reads` (two-tier all-vs-consuming), and `validate_reverse_alignment(contract, sagemaker_step_type)` (Category 4 Processing-gated + Category 5), so a custom script missing the `--job_type` argparse the builder passes, or a required env var declared-but-unread, is caught at author time.
- **Field-constraint extraction — `api/factory/field_extractor.py`.** `extract_field_constraints` surfaces each config field's closed value set (from `Literal` / `Enum` + an AST scrape of `allowed={...}` in validator source) and `case_sensitive`, and `extract_field_requirements` attaches `allowed_values` — the data `author.config_constraints` serves.
- **`PipelineDAG.validate_node_declarations()` + `strict` mode — `api/dag/base_dag.py`.** `add_edge` silently auto-creates any endpoint not already `add_node`'d, so a single edge-name typo spawns a phantom unconfigured node and orphans the real one (construction never raises; the serializer's dangling-edge check is fooled because the phantom is already in `nodes`). `validate_node_declarations()` reports every edge endpoint never explicitly declared (the only reliable typo detector), and `PipelineDAG(strict=True)` turns the same condition into a `ValueError` at `add_edge`/constructor time. `PipelineDAGResolver.validate_dag_integrity` gains an `undeclared_edge_nodes` issue category so `dag.validate_integrity` surfaces it.

### Changed

- **`registry.sagemaker_step_type` is now a closed-enum Pydantic gate.** A field audit found this routing key — the one string that decides Processing-vs-Training-vs-… — loaded any typo silently. `RegistrySection._SAGEMAKER_STEP_TYPES` is pinned and validated in the model validator (with a conformance test keeping the pin equal to the live `get_valid_sagemaker_step_types()`), so a mistyped routing verb is caught at author time instead of mis-routing at build.
- **Config enum validators normalized case-insensitive.** 29 validators across the step config classes (e.g. `output_format` `{CSV,TSV,Parquet}`) now normalize case consistently so a legal value in the wrong case no longer hard-fails a config it should accept.
- **11 step scripts standardized to canonical contract keys** (xgboost / pytorch / lightgbm / lightgbmmt model_eval + inference, model_metrics_computation, model_wiki_generator, temporal_split_preprocessing) so the script's I/O keys match the `.step.yaml` contract logical names exactly.
- **`dag.*` MCP tool no longer unions edge endpoints into the node list.** `_build_dag` previously merged edge endpoints into `nodes` to avoid a KeyError; the constructor now tolerates dangling edges (via `setdefault`) without that union, so `dag.validate_integrity` can flag an edge endpoint the caller never declared instead of masking it.

## [2.1.0] - 2026-06-29

A step-script and interface hardening tranche covering **31 step scripts**, **30 `.step.yaml` interfaces**, **19 processing modules**, and **2 test files**. Three threads: (1) a defensive fail-fast sweep that turns opaque downstream crashes (NoneType, divide-by-zero, missing-file, malformed-JSON) into descriptive `ValueError` / `ImportError` / `RuntimeError` with operator guidance, and removes silent successes; (2) an env-var contract alignment pass that relaxes previously-required env vars to optional-with-defaults and enumerates the knobs each script actually reads; and (3) a Bedrock structured-output upgrade that makes model output schema-aligned by default via Anthropic `tool_use`. Predominantly additive and backward-compatible (minor bump) — but a handful of **behavior-changing default values** are called out under Changed: pipelines that relied on the old defaults must now set them explicitly.

### Added

- **Interface env-var contracts now declare the knobs their scripts read.** Across the `.step.yaml` interfaces, the `env_vars.optional` blocks were expanded to enumerate previously-undeclared environment knobs: Bedrock retry/rate-limit/truncation settings (`BEDROCK_MAX_RETRIES`, `BEDROCK_TOP_P`, `BEDROCK_ADAPTIVE_RATE_LIMITING`, `BEDROCK_BATCH_MODE`, `BEDROCK_TRUNCATION_ENABLED`, …), streaming/worker controls (`ENABLE_TRUE_STREAMING`, `MAX_WORKERS`, `BATCH_SIZE`, `STREAMING_BATCH_SIZE`), PyTorch device tuning (`DEVICE=auto`, `ACCELERATOR=auto`, `NUM_WORKERS`), eval comparison flags (`COMPARISON_METRICS`, `COMPARISON_PLOTS`), and imputation/feature/risk-table strategy knobs (`DEFAULT_TEXT_STRATEGY`, `AUTO_DETECT_CATEGORICAL`, `SMOOTH_FACTOR`, `COUNT_THRESHOLD`, …).
- **`WORKING_DIRECTORY` defaults** added to the MIMS `package` (`/tmp/mims_packaging_directory`) and `payload` (`/tmp/mims_payload_work`) contracts.
- **Runtime-discovered env vars documented.** `missing_value_imputation` now describes per-column `COLUMN_STRATEGY_<column_name>` overrides and `payload` documents `SPECIAL_FIELD_<field_name>` overrides in their contract descriptions rather than leaving them undeclared.
- **User-supplied-package boundary documentation.** The PyTorch scripts (`pytorch_training.py`, `pytorch_model_eval.py`, `pytorch_model_inference.py`) and the LightGBM-MT scripts (`lightgbmmt_training.py`, `lightgbmmt_model_eval.py`, `lightgbmmt_model_inference.py`) gain `OPEN SECTION` banners clarifying the cursus-owned vs. user-implemented boundary, with import guards that re-raise an instructive `ImportError` pointing at the package layout the user must bundle into `source_dir`.
- **`bspline_calibration.py` documented as an alternative `ModelCalibration` entry point.** A module banner clarifies it is fully interchangeable with `model_calibration.py` — it satisfies the same `model_calibration.step.yaml` interface and sets `CALIBRATION_METHOD=bspline` where `model_calibration.py` defaults to `gam` — and deliberately has no `.step.yaml`/registry entry of its own.
- **Bedrock structured output via tool-calling in batch processing.** `bedrock_batch_processing.py` now forces schema-aligned model output by issuing an Anthropic `tool_use` request — a forced `structured_response` tool whose `input_schema` is the step's `validation_schema` — bringing it to parity with `bedrock_processing.py` (0% JSON parse failures). A shared `_build_request_body` applies the same request shape to **both** the realtime invoke path and the async batch JSONL `modelInput` records, and response parsing serializes the returned `tool_use` block's `input` so downstream handling is identical. Falls back to the legacy assistant-prefill heuristic when no `validation_schema` is present or structured output is disabled.

### Changed

- **Required-to-optional env-var relaxation across interfaces.** Previously-required fields — `ID_FIELD`, `LABEL_FIELD`, `TRAIN_RATIO`, `TEST_VAL_RATIO`, `TEXT_FIELD`, `SELECTION_STRATEGY`, `CALIBRATION_METHOD`, `CURRENCY_CONVERSION_VARS`, `SEQUENCE_GROUPING_FIELD`, `TIMESTAMP_FIELD`, `TEMPORAL_FIELD`, `VALUE_FIELDS`, `TASK_LABEL_NAMES`, and others — were moved into `env_vars.optional` with inline defaults, so step contracts no longer hard-fail when these are unset.
- **Default-value changes a pipeline author would observe.** `temporal_sequence_normalization`: `SEQUENCE_LENGTH` 100 → 51, `PADDING_STRATEGY` `zero` → `pre`. `feature_selection`: `N_FEATURES_TO_SELECT` 50 → 10, `FEATURE_SELECTION_METHODS` JSON-list → comma-string `variance,correlation,mutual_info,rfe`. `active_sample_selection`: `MAX_SAMPLES` 10000 → 0. `model_metrics_computation`: `AMOUNT_FIELD` `order_amount` → empty. `model_calibration`: `MULTICLASS_CATEGORIES` `[0, 1]` → empty. **Pipelines relying on the prior defaults must set these explicitly.**
- **`BEDROCK_USE_STRUCTURED_OUTPUT` now defaults ON** in both `bedrock_processing` and `bedrock_batch_processing` (previously `false`/unset → assistant-prefill heuristic). When a `validation_schema` is present, Bedrock output is now schema-enforced via `tool_use` by default; set `BEDROCK_USE_STRUCTURED_OUTPUT=false` to restore the legacy prefill behavior.
- **`pytorch_model_inference.step.yaml` `entry_point` corrected** `pytorch_inference.py` → `pytorch_model_inference.py` (the prior value referenced a non-existent script).
- **`temporal_sequence_normalization.py` made config-driven.** Module-global constants (`MISSING_INDICATORS`, `TEMPORAL_FIELD`, `SEQUENCE_SEPARATOR`, `OUTPUT_FORMAT`, `INCLUDE_ATTENTION_MASKS`, …) were replaced with explicit parameters threaded through the operation classes and helpers (`SequencePaddingOperation` gains `include_attention_masks`), with `entity_id_field`/`id_field` derived from env-driven `sequence_grouping_field`/`record_id_field` so operations honor runtime config instead of globals.
- **Exception/diagnostic hygiene.** `dummy_data_loading.detect_file_format` replaces three bare `except: pass` blocks with specific exception types and `logger.debug` messages; `label_ruleset_execution` narrows broad `except Exception` to `(KeyError, TypeError, ValueError, AttributeError)` with row/exception context so unexpected errors propagate; `streaming_risk_table_processor` and `streaming_numerical_imputation_processor` move `logger` definition above the `tqdm` import fallback to avoid a `NameError` on its log path.
- **`TemporalMaskProcessor.__init__`** reorders the reserved, currently-unused `mask_value` to the trailing keyword position (after `padding_value`/`output_format`) so positional callers are unaffected, and documents that the mask is derived as `(input != padding_value)` — i.e. `mask_value` is presently inert.

### Fixed

- **Numeric/edge-case safety in data-stage scripts.** `currency_conversion` falls back to a 1.0 rate (with one aggregated warning) on non-positive `conversion_rate` instead of emitting inf/NaN; `dummy_training.create_tarfile` guards its compression-ratio log with `if total_size > 0`; `feature_selection` guards empty score dicts against `ZeroDivisionError` (substituting `[0.0]`) while `pseudo_label_merge.extract_split_ratios` now raises `ValueError("Base splits must contain non-empty data")` on empty input instead of dividing by zero; `missing_value_imputation` treats NaN skewness as not-skewed and recommends `mean`.
- **Fail-fast validation of required env/config.** `bedrock_processing` (`BEDROCK_PRIMARY_MODEL_ID`), `redshift_data_loading` (`CLUSTER_SPECIFICATION`/`QUERY_SPECIFICATION`), `model_wiki_generator` (`metrics_input`/`wiki_output` dirs), and `lightgbm_model_eval` (model/eval/output dirs before `os.makedirs`) now raise descriptive `ValueError` instead of failing opaquely later.
- **Malformed-JSON handling.** `payload.extract_hyperparameters_from_tarball` and `stratified_sampling` (both `reference_counts.json` and the `REFERENCE_COUNTS_JSON` env var) convert `json.JSONDecodeError` into a clear `ValueError` naming the offending source; `temporal_split_preprocessing` hardens `BATCH_SIZE` parsing (empty → default 10, non-integer → `RuntimeError`).
- **Artifact/feature validation in model scripts.** `lightgbm_model_eval.load_model_artifacts` wraps each artifact load in a descriptive `ValueError` and skips malformed `<2-field` `feature_columns.txt` lines with a warning; `xgboost_model_eval` raises when none of the model's expected feature columns are present (replacing a silent zero-column inference); `percentile_model_calibration` guards divide-by-zero on duplicate consecutive percentiles and raises when `task_calibrations` is empty (no longer reporting an all-skipped run as success); `bspline_calibration` coerces score/label via `pd.to_numeric(errors='coerce')`, drops/logs invalid rows, and fails gracefully with a `No valid numeric rows` result.
- **Dropped-conversion bug fixed in inference scripts.** `xgboost_model_inference.save_predictions` and `lightgbm_model_inference.save_predictions` now assign the float cast back (`output_df_json[col] = output_df_json[col].astype(float)`) so float columns are actually converted before JSON export.
- **Missing runtime dependency pins** added to `xgboost_training` (`xgboost==1.7.6`, `numpy>=1.19.0`, `boto3>=1.26.0`, `matplotlib>=3.0.0`).
- **Falsy-value correctness in processing.** `categorical_imputation_processor` drops the placeholder `"My Text String"` default and uses explicit `key in` checks so legitimately-falsy configured defaults (`0`, `False`, `""`) are honored; `multiclass_label_processor` switches its guard to `label_list is not None` and adds `_normalize()` so `1`, `1.0`, and `"1"` map to the same key; `pipeline_iterable_datasets` removes the `if self.cat_field_list` gate so an empty list no longer skips numeric coercion.
- **NaN-safe fitting and fitted-state integrity.** `numerical_categorical_processor` switches to `nanmin`/`nanmax`/`nanpercentile` with all-NaN and zero-variance guards, and `minmax_scaling_processor` switches to `nanmin`/`nanmax` with a zero-variance/divide-by-zero guard, so a single NaN can't corrupt learned params; `fit()`/from-params paths in `numerical_categorical_processor`, `numerical_binning_processor`, and `numerical_imputation_processor` refuse to set `is_fitted` unless all state `transform()` needs is present (raising `ValueError` for missing custom bins/thresholds, all-NaN columns, NaN imputation values, or missing `actual_labels_`), and `numerical_imputation_processor` makes tied-mode selection deterministic.
- **Aliasing/mutation and cardinality fixes in text/temporal processors.** `bpe_tokenizer` copies via `list(texts)` before `random.shuffle` so the caller's list isn't mutated; `gensim_tokenize_processor` builds independent zero pad-vectors instead of shared references; `bert_tokenize_processor` keeps empty/whitespace chunks as `""` for 1:1 alignment; `dialogue_processor` recomputes `token_count` from the re-encoded truncated message; `time_delta_processor`, `sequence_padding_processor`, and `sequence_ordering_processor` add explicit `ValueError`/`KeyError` guards (unknown `reference_strategy`, non-positive `target_length`, non-2D input) in place of opaque downstream errors; `IdentityProcessor` and `CSAdapter` add missing `super().__init__()` calls.
- **`ValidationResult.__dict__()` renamed to `to_dict()`** in `label_ruleset_generation` (the dunder shadowed the built-in attribute); the four call sites and the corresponding test were updated, with identical output.

### Removed

- **Legacy env vars dropped from contracts.** `USE_SECURE_PYPI` (`pytorch_training`, `tokenizer_training`, `model_calibration`), `USE_PRECOMPUTED_FEATURES` (`pytorch_training`), and `AWS_STS_REGIONAL_ENDPOINTS`/`AWS_DEFAULT_REGION` (`edx_uploading`) are no longer set by their steps; pipelines that depended on these being injected must supply them another way.
- **Dead code.** Unused helpers `find_input_shards`/`aggregate_shard_results` (`risk_table_mapping.py`), unused `random`/`unicodedata` imports (`tokenizer_training.py`), and unused `numpy` imports (`model_wiki_generator.py`, `missing_value_imputation.py`).

## [2.0.0] - 2026-06-29

The **classless-factory** release — the second half of the specification-unification arc begun in 1.8.0. Where 1.8.0 collapsed the per-step *data* (the `*_contract.py` + `*_spec.py` pairs) into one declarative `.step.yaml`, 2.0.0 collapses the per-step *code*: **all 45 hand-written `builder_*.py` step-builder classes are deleted** and synthesized at runtime from the `.step.yaml` interface onto a single shared `TemplateStepBuilder` facade, routed by five `PatternHandler` construction strategies. The registry stops being a maintained artifact and is *derived* from the interface files; the alignment-validation framework is re-grounded on the three boundaries construction can't self-check; and two new agent/CLI surfaces (`steps`, `strategies`) expose the now-data-driven build so it stays introspectable. Net across `src/cursus` since 1.9.0: **185 files changed, +5,914 / −22,340 (≈ −16,400 LOC), 53 modules deleted, 10 added.** This is a **major** version bump: the per-step `*StepBuilder` source files no longer exist (they remain importable by name — see below — but are no longer files you can open, subclass in place, or monkeypatch), `cursus.steps.builders` is now a lazy import surface, and the `ValidationLevel` enum drops a member. The public pipeline-authoring API (`PipelineDAGCompiler` / `compile_dag_to_pipeline` / `compile_with_report`) is **unchanged** — the same DAG + the same config compiles to the same pipeline, which is the whole point of the rewrite (validated end-to-end on real SageMaker/SAIS infrastructure).

### The invariance guarantee

The rewrite is designed to be **invisible at the public API**. The DAG compiler, the assembler, the dependency resolver, and `pipeline.definition()` behave identically to 1.9.0 for any given DAG + config, because:

- The assembler instantiates a builder by calling `builder_cls(**five_kwargs)` with **no `isinstance` gate** — a runtime-synthesized class is indistinguishable from a hand-written one at that call site.
- The step↔step wiring graph keys entirely on `.step.yaml` **spec data** (`spec.step_type`, `compatible_sources`, `property_path`, `logical_name`) carried on `builder.spec`, plus the DAG node name — never on a Python class — so collapsing 45 classes into one facade leaves every edge intact.

A full compile of a real 11-node pipeline through a SAIS session was confirmed to produce the same SageMaker pipeline definition with only the per-step builder *source* gone.

### Added

- **`core/base/builder_templates.py` (new) — the `TemplateStepBuilder` facade + five `PatternHandler` strategies.** A step builder is now one shared facade class whose abstract methods (`create_step` / `_get_inputs` / `_get_outputs` / `_get_environment_variables`) delegate to a construction-verb handler bound at `__init__`. The five handlers are `ProcessingHandler` (covers both the `code=` "2A" and `processor.run()`→`step_args` "2B" assembly modes), `TrainingHandler`, `ModelCreationHandler`, `TransformHandler`, and `SDKDelegationHandler` (for the MODS/SAIS-predefined steps). `resolve_handler(sagemaker_step_type, step_assembly)` is the dispatch table; per-step behavior that isn't shared is expressed as declarative **knobs** read from the `.step.yaml` `patterns:` section, not as overridden Python.
- **Runtime builder synthesis (`step_catalog/builder_discovery.py::_synthesize_builder`).** For any registry step that has a `.step.yaml` interface, routes via `resolve_handler`, and has no physical builder file, the catalog fabricates `type(f"{Name}StepBuilder", (TemplateStepBuilder,), {"STEP_NAME": Name})` on demand and caches it per process. This is the deletion mechanism for the 45 shells: a step needs no `builder_*.py` at all.
- **PEP-562 lazy import surface for `steps/builders/__init__.py`.** `from cursus.steps.builders import XGBoostTrainingStepBuilder` still works — resolved lazily by a module-level `__getattr__` that routes through the same synthesizer the runtime uses, so the name resolves to a synthesized `TemplateStepBuilder` subclass with the correct `STEP_NAME`. `__all__` is data-derived (registry ∩ actually-loadable), so star-imports and introspection keep working and never trip over an offline-only SDK builder.
- **`registry/strategy_registry.py` (new) — the construction-strategy routing table.** `resolve_handler` / `axis_name_for_step_type` map a step's `sagemaker_step_type` (+ the `Processing` sub-discriminator `step_assembly`) onto a handler verb, backed by a self-registering `STRATEGY_REGISTRY`. This is the single source both the build path and the introspection tools read, so the two cannot drift.
- **`registry/interface_registry_loader.py` (new) — registry-by-construction.** `STEP_NAMES` and its derived maps (`CONFIG_STEP_REGISTRY`, `BUILDER_STEP_NAMES`, `SPEC_STEP_TYPES`) are now built by walking the `.step.yaml` `registry:` blocks (`build_registry_from_interfaces()`) instead of being read from a standalone table. The interface file is the single source of truth for step metadata as well as I/O.
- **`step_catalog/sdk_bindings.py` (new) — lazy SAIS-SDK step-class thunks.** The four SDK-delegation steps (Cradle / Redshift / Registration / DataUploading) need a live SAIS `*Step` class injected as a knob (not serializable to YAML). These are resolved through lazy thunks so the package imports and discovers cleanly **offline** (without the SAIS SDK installed); the SDK class is materialized only when actually building one of those steps.
- **`validation/alignment/validators/registry_binding_validator.py` (new) — the B3 boundary.** `RegistryBindingValidator` replaces the five deleted per-step-type validators with one check that proves a step is *constructible*: the handler resolves, the builder loads/synthesizes, and the config covers the fields the handler reads. If B3 passes, the step can be built at runtime.
- **`cursus steps` CLI + `steps.*` MCP tools (new).** `cursus steps io <Step>` and `cursus steps patterns <Step>` expose, for a step, its resolved inputs/outputs/dependencies and which construction pattern + knobs it binds — read from the same `io_view` the build uses (`steps/interfaces/io_view.py`, new), so the view provably cannot drift from behavior.
- **`cursus strategies` CLI + `strategies.*` MCP tools (new).** `cursus strategies axes | list | show | for --sagemaker-step-type <T> | knobs` make the strategy library self-describing and queryable — the replacement for "open the builder class to see how a step is built" now that there is no per-step class to open. Backed by `STRATEGY_REGISTRY`.
- The MCP tool surface grows from 8 to **10 namespaces** (adds `steps`, `strategies`).
- **`STEP_NAME` is now a first-class attribute on `StepBuilderBase`** (`core/base/builder_base.py`); `_get_step_name()` prefers it over stripping the class name, since a synthesized/shared shell's class name no longer carries the canonical step name.

### Changed

- **Step builders are synthesized, not written.** Authoring a new step is now: one `.step.yaml` (with a `registry:` block) + one config class — no builder file. The difference between, say, a Processing step and a Training step is one string (`sagemaker_step_type`) in the interface. `O(1)` step addition; zero builder boilerplate.
- **Alignment validation re-grounded on three boundaries (FZ 31e1d3h).** The old four-level model assumed three separately-maintained artifacts; with one self-aligning `.step.yaml` and source-less builders, that premise is gone. Validation now proves the single `StepInterface` object against the three things construction *can't* self-check: **B1 script↔interface fidelity** (AST, the former Level 1), **B2 cross-step DAG-resolvability** (the former Level 3, now also owning the SageMaker property-path check folded in from the old Level 2), and **B3 registry↔handler↔config binding** (`RegistryBindingValidator`). The ~14 `inspect.getsource(builder_class)` scans in the old validators — which now return identical shell text for every step and carry zero signal — are deleted.
- **`StepCatalog` gains import-free routability + a duck-typed SDK output accessor.** `has_builder_provider()` answers "is this step buildable?" without importing a class; `get_sdk_output_location(built_step, output_type=None)` reads a built MODS step's own output locations generically (this let `CradleDataLoading` drop its bespoke helpers and become a pure shell like the rest).
- **`cursus.steps.builders` is now a lazy (PEP-562) module** rather than ~45 eager `from .builder_x import X` statements. Importing the package no longer imports every builder (and no longer fails when an optional SDK builder's dependency is missing).

### Removed

- **All 45 per-step `builder_*.py` files** under `steps/builders/` (xgboost / pytorch / lightgbm(+mt) training·model·eval·inference, tabular_preprocessing, cradle / redshift / edx / registration / data-uploading SDK-delegation, bedrock ×3, model_calibration / percentile / metrics / wiki, risk_table_mapping, stratified_sampling, package, payload, temporal ×3, currency_conversion, missing_value_imputation, feature_selection, pseudo_label_merge, dummy_*, tokenizer_training, …). They are synthesized at runtime; their names remain importable via the lazy surface.
- **Seven validation modules** made redundant by the boundary redesign: `validation/alignment/core/contract_spec_alignment.py` (the old Level-2 Contract↔Spec tier — now a construction-time Pydantic invariant, i.e. a tautology to re-check), the five per-step-type validators (`processing_step_validator.py`, `training_step_validator.py`, `transform_step_validator.py`, `createmodel_step_validator.py`, `step_type_specific_validator.py`), and `validator_factory.py` that dispatched to them. Replaced by the single `RegistryBindingValidator`.
- **`registry/step_names.yaml`** — the standalone step-name table is gone; `STEP_NAMES` is now derived from the `.step.yaml` `registry:` blocks (see `interface_registry_loader.py`).
- **`ValidationLevel.CONTRACT_SPEC` (value 2)** removed from the enum. The surviving members keep their names and non-contiguous integer values (`SCRIPT_CONTRACT=1`, `SPEC_DEPENDENCY=3`, `BUILDER_CONFIG=4`) so `ValidationLevel(1/3/4)` coercion and existing references keep working — only the value-2 member is gone.

### Fixed

- **`ProcessingHandler` 2B passed both `step_args` and `processor` to `ProcessingStep`.** In the `processor.run()`→`step_args` assembly mode, the handler's shared `common` kwargs dict carried `processor=processor`, so the constructed `ProcessingStep(step_args=…, **common)` supplied both sides of SageMaker's mutual-exclusion check and raised `ValueError: either step_args or processor need to be given, but not both.` at build time. The `processor` kwarg now lives only on the `code=` (2A) branch; the `step_args` (2B) branch passes `step_args` alone. This surfaced only at real pipeline execution because unit tests mock `ProcessingStep` (the check lives in the SDK constructor, below the seam mocks replace); a regression test now drives the **real** `ProcessingStep` (processor on a `PipelineSession` so `run()` defers offline) for both assembly modes.

### Notes

- **Migration:** for typical pipeline authoring, no changes are required — the DAG-compiler API and every `.step.yaml`/config are unchanged. The breaking surface is for code that **imported a per-step builder module by path** (`from cursus.steps.builders.builder_xgboost_training_step import XGBoostTrainingStepBuilder`) — import the name from the package instead (`from cursus.steps.builders import XGBoostTrainingStepBuilder`), which the lazy surface resolves to the synthesized class. Code that **subclassed or monkeypatched a per-step builder in place** must move that per-step behavior into the step's `.step.yaml` (`patterns:` knobs) or a handler.
- **SageMaker step-construction constraints:** every pipeline step type enforces a `step_args` **XOR** object-handle rule (`processor`/`estimator`/`transformer`/`model`/`tuner`) in its constructor; a handler must pass exactly one, and `step_args` must come from the matching deferred call (`processor.run()` / `estimator.fit()` / etc.) on a `PipelineSession`. The `ProcessingHandler` fix above is the one place this was violated.

## [1.10.0] - 2026-06-26

A path-resolution feature: a **caller-hook project-root anchor (Strategy 0)** that lets a pipeline's entry point push its own project folder, so a step's relative `source_dir` / `processing_source_dir` resolves directly as `project_root / source_dir` — making both the `CURSUS_PROJECT_BASE` env var and the `project_root_folder` config field **optional**. Fully backward compatible; the new public API is additive (minor bump). 5 source files + 4 new test files (38 tests).

### Added

- **Caller-hook project-root anchor (Strategy 0)** (`core/utils/hybrid_path_resolution.py`). New `set_project_root()` / `get_project_root()` (re-exported from `cursus.core.utils`) push the project folder process-wide, and a new top-priority `_pushed_project_root_discovery` resolves a relative path directly as `project_root / relative_path`. Because the caller's `__file__` always lives inside the project, this is the most robust anchor across deployment modes (Lambda/MODS, SAIS notebook, pip-installed). It is **distinct from `CURSUS_PROJECT_BASE`**: that env var is a *package base* under which a named `project_root_folder` sibling is searched, whereas the pushed value is the *project folder itself*. The previous strategies become 0b (explicit `CURSUS_PROJECT_BASE`) and 1–4 fallbacks.
- **`project_root` parameter on `PipelineDAGCompiler.__init__`.** Pipelines pass `Path(__file__).parent` from the module defining `generate_pipeline()`; when omitted it is inferred from `config_path` (walking up past a `pipeline_config[s]/` directory to the project root). The resolved root is pushed process-wide so the config → resolver → builder chain anchors on it.
- **`project_root` parameter on `build_and_compile`** (`pipeline_catalog/core/builders.py`), forwarded to the compiler. The generated MODS template (`build_mods_pipeline`) now passes `project_root=caller_dir`, since its `generate_pipeline()` lives in the project folder.

### Changed

- **`BasePipelineConfig.resolve_hybrid_path` no longer requires `project_root_folder`.** It now proceeds when *either* `project_root_folder` is set *or* a project root was pushed by the caller hook, so a config without `project_root_folder` still resolves (anchored on the pushed folder).
- Dropped pre-existing unused imports and lint nits in the touched files — `ConfigurationError` / `ValidationError` / `RegistryError` imports and two `f""`-without-placeholders log strings in `dag_compiler.py`, and an unused `PipelineDAG` import in `builders.py`.

## [1.9.0] - 2026-06-26

A brittleness-review fix tranche: three live-bug fixes (job-type variant loading, CLI exit codes, dropped processing fields), centralization of duplicated step-name/job-type vocabulary into `step_catalog/naming.py`, and a sweep that surfaces previously-swallowed exceptions. 19 source + 11 test files (3 new); no dependency changes. Full suite: **1758 passed**. New public helpers are purely additive (minor bump).

### Fixed

- **Job-type variant loading silently broke for variants that restate only a subset of ports** (`core/base/step_interface.py`). `StepInterface.from_yaml_dict` shallow-merged a `.step.yaml` job-type variant over the base sections (`{**base, **variant}`), so a variant that listed only the ports it tweaked **dropped every base port it omitted** — e.g. `hyperparameters_s3_uri` vanished from `RiskTableMapping`'s variants, violating the contract↔spec alignment invariant and raising at construction. Now uses a recursive `_deep_merge`, so a variant overrides just the fields it names and preserves the rest. Fixes `RiskTableMapping` / `BatchTransform` variant loading.
- **CLI commands exited 0 on failure** (`cli/catalog_cli.py`, `cli/compile_cli.py`). Error paths did `return 1`, but Click ignores a command function's return value, so failures reported success to the shell. All error exits now `raise SystemExit(1)` (propagated past the outer handlers). `catalog_cli` was refactored onto a shared `safe_cli_command` decorator, removing 17 per-command `try/except` blocks in favor of one central error→exit-code path.
- **`ConfigurationGenerator` silently dropped the 9 processing-specific base fields** (`api/factory/configuration_generator.py`). `_inherits_from_processing_config` scanned the MRO for the substring `"BaseProcessingStepConfig"` — a transposition of the real class name `ProcessingStepConfigBase` — so it matched nothing and always returned `False`, disabling the processing-inheritance branch. Now uses `issubclass` against the imported base class (mirroring `DAGConfigFactory`); `_inherits_from_base_config` was hardened the same way.
- **Compound-acronym phantom catalog entries** (`step_catalog/step_catalog.py`). A naive `str.capitalize()` join turned `pytorch_training` into `Pytorch_Training` instead of `PyTorchTraining`, orphaning PyTorch/XGBoost/LightGBM file components into phantom entries. Now delegates to `naming.parts_to_pascal`.
- **Custom validation rules that raised were swallowed even in strict mode** (`processing/categorical/categorical_validation_processor.py`). A misconfigured rule's exception was caught and logged regardless of `validation_strategy`. Now re-raises as a `ValueError` under `strict`, and the intentional `ValueError` for actual violations was moved outside the `try` so strict-mode rejection can never be downgraded to a log.

### Added

- **Registry health signals** (`registry/step_names.py`). When the hybrid (workspace-aware) `UnifiedRegistryManager` fails to initialize, the code silently falls back to a static manager — previously invisible. New `is_hybrid_active()` and `get_registry_health()` (`{hybrid_active, init_error}`) surface the degraded state, and the fallback now logs the full traceback.
- **Centralized job-type / base-config vocabulary** (`step_catalog/naming.py`). `JOB_TYPE_SUFFIXES`, `JOB_TYPE_KEYWORDS`, `BASE_CONFIGS`, and `is_job_type_variant()` are now the single source of truth, replacing duplicated (and drifted) literals in `step_catalog.py`, `spec_discovery.py`, and `validation/builders/universal_test.py`. The two genuinely-distinct concepts (variant-suffix detection vs. classification keywords) are kept separate and documented.
- **Cache-reset / hot-reload helpers** — `reset_unified_config_manager_cache()`, `reset_inheritance_aware_field_generator_cache()`, and `clear_interface_cache()` (for tests and dev hot-reload of edited `.step.yaml` files).
- **`EdxUploadingStepBuilder`** is now exported from `steps/builders/__init__.py`.

### Changed

- **Per-context cache keying for config-field singletons** (`core/config_fields/{unified_config_manager,inheritance_aware_field_generator}.py`). Both `get_*` factories cached a single global on first call, so a later call with **different** `workspace_dirs` / `project_id` silently returned the stale wrong-context instance. Each is now keyed by its context (`workspace_dirs`, and additionally `project_id` for the field generator).
- **Surfaced previously-swallowed exceptions** across the discovery/tool layer (Theme 1): the MCP `catalog.step_info` and `dag.deserialize` tools now attach a `warning` when best-effort framework/metadata detection fails (so `None`-because-undetected is distinguishable from `None`-because-error); `mcp/server.py` chains the original `ImportError` (`raise … from exc`); `universal_test` logs unresolved SageMaker step types at debug instead of a bare `except`; and `ScriptExecutionRegistry._node_specs` is initialized in `__init__` to avoid an `AttributeError` on an uninitialized registry.

### Removed

- **Dead module-level `@property def STEP_NAMES`** (`registry/step_names.py`) — `@property` has no effect at module scope and the descriptor was immediately shadowed by the real `STEP_NAMES = get_step_names()` snapshot bound below it. Removed to stop implying a dynamic behavior that never existed; workspace-aware reads must call `get_step_names(workspace_id)`.

### Docs

- **`CategoricalLabelProcessor` multi-worker hazard documented.** With `update_on_new=True` (the default) `process()` mutates the category→label mapping on first sight of a new category, which is unsafe under `num_workers > 0` (each forked worker assigns divergent ids). The class/arg docstrings now direct multi-worker / distributed use to a fixed `initial_categories` list with `update_on_new=False`.

## [1.8.1] - 2026-06-26

Post-1.8.0 step-catalog hardening, a new MCP tool, and two CLI bug fixes found while testing the 1.8.0 CLI. No public-API or dependency changes; `step_catalog`/`registry`/`mcp`/`steps`/`pipeline_catalog` test suites green.

### Added

- **`catalog.step_spec` MCP tool.** Exposes a step's resolved dependencies and outputs (`{step_type, node_type, dependencies[], outputs[]}`) over the agent tool surface; abstract steps with no spec return a `not_found` result + remedy. Brings the MCP registry to **42 tools** (the `catalog` namespace goes from 6 → 7).
- **`step_catalog/naming.py` (new) — single source of truth for step-name conversion.** Centralizes PascalCase ↔ snake_case conversion (`canonical_to_snake`, `parts_to_pascal`, `canonical_key`); script, builder, and contract discovery now delegate to it.

### Fixed

- **`cursus --version` crashed** with `RuntimeError: 'upstream' is not installed`. The CLI's `version_option` looked up the internal distribution name; the published distribution is `cursus`. Now looks up `cursus` (matching the 1.8.0 `__version__` fix), and the last `upstream` reference in the package (a comment) was removed — `cursus --version` now prints `cursus 1.8.0`.
- **`cursus pipeline-catalog recommend --framework X` returned unrelated DAGs** (e.g. `--framework xgboost` surfaced bedrock/pytorch/dummy). `recommend_for_agent()` ignored framework, ranked, and truncated to top-5; `agent_tool` then post-filtered with an `... or results[:3]` fallback, so when the truncated set held no framework match it fell back to unrelated DAGs. `framework` is now a **hard filter inside `recommend_for_agent`, applied before scoring/truncation**, and the misleading fallback is gone. Also fixes the MCP `pipeline_catalog` recommend tool and the LLM `agent_tool`, which share that path. (The weak `test_recommend_framework_filter` that missed this was strengthened to assert every result matches the requested framework.)
- **Compound-acronym name resolution.** A latent `contract_discovery` bug resolved `PyTorchTraining` → `py_torch_training`, silently breaking contract imports for compound-acronym step types (PyTorch, XGBoost, LightGBM). Fixed via the centralized `naming.py`.
- **Hardened `step_name → .step.yaml` resolution** with a normalized fallback scan (`steps/interfaces/__init__.py`) so interface lookup tolerates naming variants.
- **Surfaced previously-silent discovery failures.** Bare `except` blocks in the discovery layer now log; `SyntaxError` is separated out in `config_discovery`; `StepCatalog.get_metrics_report()` gains `index_build_error` + `pipeline_interface_available` health signals; pipeline-interface methods degrade gracefully instead of raising `AttributeError` on `None`.

### Removed

- **Dead code in `step_catalog`** — two unused `StepCatalog` caches and a discarded redundant full-scan in `script_discovery`.

### Docs

- **README aligned with the refactored 1.8.0 repo** — corrected CLI commands (flag-based `compile`/`validate`, new `projects`/`pipeline-catalog`/`mcp` groups), replaced the removed class-based catalog examples with the data-driven `recommend_dag`/`load_shared_dag`/`build_and_compile` API, rewrote the Architecture section, added the Agent-Ready (MCP) feature, and fixed moved doc links.

## [1.8.0] - 2026-06-26

This release completes the **specification-unification** and **agent-enablement** arc. Per-step Python spec/contract pairs are merged into a single declarative `.step.yaml` per step; the pipeline catalog becomes data-driven; a new framework-neutral `cursus.mcp` tool surface is added for LLM agents; the dead `cursus.workspace` module is removed; and the CLI is reorganized into focused command groups. Repo-wide: 610 files changed (+115,624 / −38,213) across 12 commits since 1.7.1 (the bulk of the additions is a 139-file test-suite sync).

### Added

- **`cursus.mcp` module (new) — framework-neutral agentic tool surface.** A new top-level module parallel to `core`/`steps`, exposing **41 JSON-in / JSON-out tools across 8 namespaces** (`catalog`, `dag`, `config`, `compile`, `validate`, `execdoc`, `pipeline_catalog`, `info`) over the pipeline engine, leaving the human-facing `cursus.cli` untouched.
  - `mcp/envelope.py` — the `ToolResult` / `ToolError` response contract.
  - `mcp/registry.py` — single-source-of-truth `ToolDef` registry + `call_tool` invoker; a namespace that fails to import (e.g. a missing optional engine dependency) is skipped with a warning, not fatal.
  - `mcp/tools/*.py` — per-namespace tool implementations (`catalog`, `compile`, `config`, `dag`, `execdoc`, `info`, `pipeline_catalog`, `validate`, plus a `shared` helper).
  - `mcp/server.py` — thin optional MCP server (`python -m cursus.mcp.server`; lazy SDK import), with OpenAI/Claude + MCP tool-spec exporters.
  - Wired into the CLI as `cursus mcp`.
- **Declarative `.step.yaml` step interfaces (`steps/interfaces/`).** 45 `<step>.step.yaml` files + `steps/interfaces/__init__.py` (`load_step_interface()`), each carrying one unified `StepInterface` (contract I/O + spec demand/supply). New foundation classes `core/base/step_interface.py` and `core/base/step_contract.py`. Packaged via `*.yaml`/`*.yml` package-data so the YAML ships in the wheel.
- **Data-driven pipeline catalog (`pipeline_catalog/shared_dags/`).** 42 shared DAGs as `*.dag.json` (XGBoost, PyTorch, LightGBM, MTGBM, Bedrock, Dummy, …) indexed by `catalog_index.json`, replacing the previous class-based catalog. New `pipeline_catalog/core/`:
  - `builders.py` — `build_and_compile` (SAIS) and `build_mods_pipeline` (generates a `@MODSTemplate` class on demand),
  - `router.py` — deterministic scoring (`recommend_dag` / `auto_select_dag` / `recommend_for_agent`) over the index,
  - `agent_tool.py` — exposes the router as an LLM agent tool.
- **Reorganized CLI into focused command groups.** New `cursus` subcommands `config`, `dag`, `mcp`, `pipeline-catalog`, `projects`, `validate` (modules `config_cli.py`, `dag_cli.py`, `mcp_cli.py`, `pipeline_catalog_cli.py`, `project_cli.py`, `validate_cli.py`, plus a shared `_shared.py`).
- **YAML-backed step registry.** `registry/step_names.yaml` + `registry/step_names_base.py` load the canonical step-name table from data instead of a Python literal.
- **Pipeline-project discovery.** `core/utils/project_discovery.py` + `cursus projects list|show` — a read-only helper that became the standard way to enumerate pipeline projects (a `pipeline_config[s]/` + `dockers/` + pipeline `.py` layout, with `metadata.config_types` mapping nodes to config classes). Replaces the removed workspace module's only goal-relevant use.
- **Full test-suite sync** — 139 test files across `tests/` (alignment validators, step_catalog, core/deps, core/compiler, registry, mcp, cli, integration, …).

### Changed

- **Merged step spec + script contract into one `.step.yaml` per step.** The legacy `ScriptContract` (I/O) and `StepSpecification` (demand/supply) data classes are unified into a single Pydantic `StepInterface` sourced from YAML; contract↔spec alignment is now a **construction-time invariant** (a `@model_validator` raises if contract paths aren't a subset of the spec's deps/outputs) rather than a separate runtime validation tier. Step builders now call `load_step_interface("StepType")`.
- **Validation framework collapsed.** The redundant Contract↔Spec (Level-2) tier was removed (now an invariant) and the unified alignment tester refactored; dead modules deleted (see Removed).
- **DAG / API optimization and agent ergonomics.** `api/dag/{base_dag,pipeline_dag_resolver,pipeline_dag_serializer}.py` refactored; `step_catalog` discovery / config-resolver / builder-discovery updated; CLI and tool surfaces tidied for agent consumption.
- **Registry loads from YAML.** The canonical step table moved from the Python literal `step_names_original.py` to `step_names.yaml`, read through `step_names_base.py` (the workspace-aware facade in `step_names.py` is preserved).
- **NVMe/KMS compiler patch.** The compiler strips `VolumeKmsKeyId` from NVMe-backed `ProcessingStep`s in the generated pipeline definition.
- **Renamed `steps/scripts/edx_upload.py` → `steps/scripts/edx_uploading.py`** for naming consistency with the step type.

### Removed

- **`cursus.workspace` module (~1,949 LOC).** Removed `workspace/{__init__,api,integrator,manager,validator}.py` — a dead island (only a `try/except` re-export in `core/__init__.py`, zero instantiations anywhere). Its one goal-relevant use is replaced by `core/utils/project_discovery.py`. **Potentially breaking** for any *external* code importing `cursus.workspace`, though no in-tree caller existed. (The separate load-bearing `workspace_dirs` parameter and `step_catalog/adapters` were explicitly preserved.)
- **113 per-step Python spec/contract files.** 71 `*_spec.py` + 42 `*_contract.py` under `steps/specs/` and `steps/contracts/`, superseded by the 45 `.step.yaml` interfaces. Both packages are now emptied to `__init__.py`, except one shared `steps/contracts/training_script_contract.py` retained as the path-tolerant base for training-step contracts.
- **`core/base/specification_base.py`** — replaced by `step_interface.py` + `step_contract.py`.
- **`registry/step_names_original.py`** — replaced by `step_names.yaml` + `step_names_base.py`.
- **Dead validation modules** — `validation/alignment/reporting/validation_reporter.py`, `validation/alignment/validators/contract_spec_validator.py`, `validation/alignment/validators/method_interface_validator.py`, `validation/shared/chart_utils.py`, `validation/utils/import_resolver.py`.

### Fixed

- **Packaged the runtime data files.** `tool.setuptools.package-data` used a non-recursive `cursus = ["*.yaml", "*.json", …]` glob that only matched the top-level package directory, so the new `steps/interfaces/*.step.yaml` (45), `pipeline_catalog/shared_dags/**/*.dag.json` (42) + `catalog_index.json`, and `registry/step_names.yaml` were **silently omitted from the wheel** — a clean `pip install cursus==1.8.0` would have failed at runtime the moment a builder called `load_step_interface()`. Switched to a per-sub-package `"*"` key and added a matching `recursive-include src/cursus *.yaml *.yml *.json` to `MANIFEST.in` so the data ships in both the wheel and the sdist. (New gap: these data files did not exist in 1.7.1.)
- **`cursus.__version__` no longer resolves to `None` on the installed package.** `_resolve_metadata()` looked up `importlib.metadata.metadata("upstream")`, but the distribution published to PyPI is named **`cursus`**; that lookup failed and, with no `VERSION` file present in `site-packages`, `__version__` fell through to `None`. Now looks up `cursus`.

### Notes

- No dependency changes since 1.7.1 (`sagemaker>=2.248.0,<3`, `pydantic<3`, and `PyYAML>=6.0.0` — now load-bearing at runtime — were already declared). The 1.x line continues to target **SageMaker SDK 2.x**; the forthcoming 2.x line will target SDK 3.x.

## [1.7.1] - 2026-05-28

### Fixed

- **`sagemaker` dependency upper bound** — `sagemaker>=2.248.0` → `sagemaker>=2.248.0,<3`. The SageMaker SDK 3.x release ships a fundamentally restructured layout (the `sagemaker.workflow.pipeline_context` submodule was removed; even `sagemaker.__version__` is gone). With no upper bound, `pip install cursus` resolved `sagemaker==3.12.0`, after which any `from cursus.steps import *` raised `ModuleNotFoundError: No module named 'sagemaker.workflow'` at `cursus/core/base/builder_base.py:8`. Every fresh public-PyPI install of cursus 1.5.x — 1.7.0 was broken in this way. The 1.7.1 cap pins to the 2.x line until cursus is ported to the 3.x layout. Same defensive-cap shape as the `pydantic<3` cap added in 1.5.2.
- **Add `pytz` to direct dependencies** — sagemaker 2.x's `sagemaker/workflow/pipeline.py` does `import pytz` at module top-level but does *not* declare it in `Requires-Dist`. cursus imports that module transitively via `cursus.core.assembler.pipeline_assembler`, so a clean `pip install cursus` produced a `ModuleNotFoundError: No module named 'pytz'` at first import. Declaring `pytz` as a direct cursus dep papers over the sagemaker packaging gap so users get a functional install out of the box.

## [1.7.0] - 2026-05-28

### Added

- **`RedshiftDataLoading` step (new step type)** — SQL execution against a Redshift cluster, writing CSV output to S3 (and optionally uploading to EDX as a side effect). Adopted from the SAIS SDK MODS workflow. Source node in the DAG (no upstream dependencies). Ships the full five-layer step bundle:
  - `cursus/steps/scripts/redshift_data_loading.py` — container script. Supports two connectors: `pg8000` (PostgreSQL wire protocol) and `redshift_connector` (Amazon's official client). Optional `Andes3` IAM authentication with session tags. Reads a flattened `step_config.json` (`clusterSpecification` / `querySpecification` / `outputSpecification`).
  - `cursus/steps/configs/config_redshift_data_loading_step.py` — `RedshiftDataLoadingConfig(BasePipelineConfig)`. Tier-1 essential fields: `cluster_id`, `db_name`, `role_arn`, `cluster_endpoint`, `query`. Tier-2 with defaults: `connector_type` (default `redshift_connector`), `port` (5439), `is_using_andes3` (True), `output_data_source_type` (`S3` / `EDX`), `edx_arn` (required iff EDX mode), `drop_header`, `job_type`, `max_runtime_in_seconds` (12h default). Pydantic model-level validation enforces "EDX mode requires `edx_arn`" and "`connector_type` ∈ {pg8000, redshift_connector}". The computed `step_config_json` re-assembles the 3-spec dict that the container script consumes.
  - `cursus/steps/contracts/redshift_data_loading_contract.py` — script contract.
  - `cursus/steps/specs/redshift_data_loading_spec.py` — step specification (source node, no dependencies, two outputs).
  - `cursus/steps/builders/builder_redshift_data_loading_step.py` — step builder.
  - Registered in `cursus/registry/step_names.py` + `step_names_original.py` as `RedshiftDataLoading`.
- **`munged_address_pytorch/dockers/pytorch_training.py`** (NEW, 597 lines) — simplified DistilBERT binary classifier for Munged Address Detection. Text-only (no multimodal, no risk tables, no imputation, no streaming), ~400 actual lines vs the generic 1,631-line trainer. Reads `/opt/ml/input/data/{train,val,test}/{split}_processed_data.csv` with columns `shippingAddress`, `__tag__`, `orderDate`, `marketplaceId`. Outputs `model.pth`, `model.onnx`, tokenizer, hyperparameters, predictions, and TensorBoard logs.
- **`munged_address_pytorch/dockers/scripts/bedrock_processing.py`** (NEW, 2,493 lines) — project-local Bedrock processing docker script tailored to the munged-address use case.
- **`munged_address_pytorch/dockers/scripts/tabular_preprocessing.py`** (NEW, 2,159 lines) — project-local tabular preprocessing docker script.
- **`munged_address_pytorch/generate_config_na.py`** (NEW, 508 lines) — programmatically generates `pipeline_configs/config_NA.json` from code using `DAGConfigFactory`, `CradleJobSpecificationConfig`, `EdxDataSourceConfig`, and friends — replaces hand-edited JSON with a code-driven build.
- **`munged_address_pytorch/dockers/requirements-gpu-secure.txt`** (NEW) — security-scanned GPU requirements for the SageMaker container build.
- **`munged_address_pytorch/dockers/scripts/calibration/standard_calibration_dictionary.json`** (NEW) — calibration reference data used by the inference handler.

### Changed

- **`EdxUploadingConfig` — dual-mode input** (`+96 / -23` lines). Now accepts either a direct ARN OR the previous component-based form, matching the pattern already used by `EdxDataSourceConfig` in CradleDataLoading:
  1. **Direct ARN mode**: pass `edx_arn=arn:amazon:edx:iad::manifest/{provider}/{subject}/{dataset}/["{key}"]`; components are parsed automatically.
  2. **Component-based mode** (previous default): pass `edx_provider`, `edx_subject`, `edx_dataset`, `edx_manifest_key`.

  All five Tier-1 fields are now `Optional`, with a `model_validator` enforcing "at least one mode satisfied" at construction time. Backwards-compatible — existing component-based configs continue to work unchanged.
- **`munged_address_pytorch` docker scripts trimmed** — `pytorch_inference_handler.py` shrunk from 1,905 → much smaller, `pytorch_model_inference.py` shrunk from 1,638 → much smaller. The deletions match the same scope reduction as the new training script — multimodal / risk-table / imputation paths removed because Munged Address Detection is text-only.
- **`bedrock_processing` script (top-level)** — small adjustments (+31 lines) to align with the project-local docker variant.
- **Spec touch-ups for Redshift compatibility:**
  - `edx_uploading_spec.py`: added `RedshiftDataLoading` to compatible upstream sources (Redshift → EDX upload as a side-effect pipeline).
  - `tabular_preprocessing_spec.py`: added `RedshiftDataLoading` to compatible sources on both primary and secondary `DATA` dependencies (×2).
  - `cradle_data_loading_spec.py`: minor `aliases=[]` placeholder addition.
- **Project orchestrator small touches** — `munged_address_pytorch_na.py` (-1 line), `package.py` and `payload.py` (top-level + project copies, ±3-4 lines each), `tabular_preprocessing_sampling_munged.py` (+148/-… reorg), `tabular_preprocessing_training_munged.py` (+109/-… reorg), `hyperparameters_NA.json` (6-line adjustment).

## [1.6.0] - 2026-05-26

### Added

- **`EdxUploading` step (new step type)** — S3 → EDX manifest uploads via `EdxDataLoader`, runs in the SAIS Docker container as a SageMaker `ProcessingStep`. Ships the full five-layer step bundle:
  - `cursus/steps/scripts/edx_upload.py` — reads files from `/opt/ml/processing/input/data` and uploads each to an EDX manifest. Sink step (no S3 output). Parallel uploads via `ThreadPoolExecutor(max_workers=4)`. Uses `secure_ai_sandbox_python_lib.session.Session` (`SandboxSession`) under the hood.
  - `cursus/steps/configs/config_edx_uploading_step.py` — `EdxUploadingConfig(ProcessingStepConfigBase)`. Tier-1 essential fields: `edx_provider`, `edx_subject`, `edx_dataset`, `edx_manifest_key`. The manifest key supports two modes: (a) static key like `'munged_na'` → ARN ends with `/["munged_na"]`; (b) template key with placeholders like `'{marketplace},{dataset_date},{job_id}'`, resolved at runtime from `edx_manifest_key_parts` + environ. Derived `edx_arn_base` builds `arn:amazon:edx:iad::manifest/{provider}/{subject}/{dataset}` automatically.
  - `cursus/steps/contracts/edx_uploading_contract.py` — script contract.
  - `cursus/steps/specs/edx_uploading_spec.py` — step specification.
  - `cursus/steps/builders/builder_edx_uploading_step.py` — step builder.
- **`projects/munged_address_pytorch/`** — new end-to-end example project: a Munged Address Detection DistilBERT training pipeline (NA marketplace). Demonstrates a 4-phase Cursus DAG (`create_munged_address_training_dag()`):
  1. Tag generation → upload to EDX via the new `EdxUploading` step,
  2. Cradle data loading using the EDX tags as join key,
  3. Bedrock LLM strangeness scoring,
  4. Stratified sampling + tabular preprocessing + DistilBERT training.
  Includes `munged_address_pytorch_na.py` orchestrator (`MungedAddressTrainingPipelineNA`), `pipeline_configs/config_NA.json`, and SageMaker Docker scripts (`percentile_model_calibration.py`, `tabular_preprocessing_sampling_munged.py`, `tabular_preprocessing_training_munged.py`).
- **Logical-name aliases on data-loading and packaging specs** — `cradle_data_loading_spec.py` outputs now expose alias lists (`DATA` ↔ `input_data` / `data` / `training_data` / `processed_data`; `METADATA` ↔ `metadata`; `SIGNATURE` ↔ `signature` / `schema`); `package_spec.py` and `bedrock_processing_spec.py` gain matching alias entries. Aliases let downstream DAG edges bind to either the canonical logical name or a domain-friendly synonym without breaking existing wiring.

### Changed

- **`bedrock_processing` script + config — major rework** (`+467 / -45` lines in the script, `+82` in the config). Introduces three invocation modes selectable by config:
  1. `invoke_model` (Anthropic format) with assistant prefilling for JSON output — default;
  2. **Structured output via `tool_use`** — guaranteed schema compliance, 0% parse failures;
  3. **Converse API** — model-agnostic, supports Nova / Llama / Mistral without per-model format changes.
  Adds **self-contained mode** so an upstream `BedrockPromptTemplateGeneration` step is no longer required: prompt template (`bedrock_user_prompt_template`), system prompt (`bedrock_system_prompt`), input placeholders (`bedrock_input_placeholders`), and validation schema (`bedrock_validation_schema`) can all be embedded in the config. Fallback chain: upstream step output > config-embedded > error. Also adds: dynamic Pydantic model creation from JSON validation schemas; configurable thread pool + adaptive rate limiting (auto-tunes req/s from observed throttle rate); circuit breaker that trips after N consecutive failures; checkpoint/resume per batch; inference-profile management with on-demand fallback; input-field truncation; Unicode quote repair for German-style quotes in LLM responses.
- **`stratified_sampling` script + builder + config + contract — substantially expanded** (`+218 / -37` lines in the script). Four allocation strategies are now supported: `balanced` (equal per stratum), `proportional_with_minimum` (proportional + floor constraints), `optimal` (Neyman variance-weighted), and `external_proportional` (sample to match a reference distribution with multiplier). Adds sampling-with-replacement for oversampling, NaN-stratum guard, empty-DataFrame guard, per-split diagnostics JSON (requested vs achieved per stratum), CSV/TSV/Parquet format preservation, split-aware processing (train/val sampled, test copied unchanged), and reference-counts loading via sidecar `reference_counts.json` or env-var fallback.
- **`tabular_preprocessing` script + spec** (`+140 / -40` lines). Adds `optimize_dtypes` shared utility for memory-efficient dtype downcasting and additional preprocessing affordances used by the munged-address pipeline; spec gains 24 lines of output-binding additions.

## [1.5.3] - 2026-05-21

### Added

- **NVMe-aware MODSWorkflowHelper security patch** (`cursus/core/utils/nvme_security.py`) — auto-installed at `import cursus` time. Patches `SagemakerPipelineHelper._patch_training_step` to skip `VolumeKmsKeyId` injection on NVMe-backed GPU instances (`ml.p4d`, `ml.g5`, `ml.g4dn`, `ml.p3dn`, `ml.trn`, `ml.inf`, any `*d` / `*dn` family) which reject `VolumeKmsKeyId` because they use hardware encryption. All other security settings (subnets, security groups, `encrypt_inter_container_traffic`, `output_kms_key`) are preserved. Idempotent and a no-op when `mods_workflow_helper` / `sagemaker` aren't importable. Ref: OfficeHour-1553, CMLS-Model-1049 — temporary shim until MODSWorkflowHelper ships its own NVMe gate.
- **Hybrid path resolution** (`cursus/core/utils/hybrid_path_resolution.py`) — five-strategy resolver (`HybridPathResolver`) that locates project files across Lambda/MODS bundled, monorepo `src/cursus`, and pip-installed-separated deployment scenarios. Strategies tried in order: explicit `CURSUS_PROJECT_BASE` env var → package-location discovery → working-directory walk → generic recursive search → default `cursus/steps/scripts` fallback. Per-strategy metrics surfaced via `get_hybrid_resolution_metrics()`. Behavior gated by `CURSUS_HYBRID_RESOLUTION_ENABLED` (default on) and `CURSUS_HYBRID_RESOLUTION_MODE` (`full` / `fallback_only` / `disabled`).

### Fixed

- **`cursus.__version__` regression** — the metadata block in `src/cursus/__init__.py` had been temporarily switched to look up `importlib.metadata.version("upstream")` with a hardcoded `"1.4.1"` fallback. On the public PyPI distribution (`name = "cursus"`) the lookup always raised, so `cursus.__version__` reported `"1.4.1"` for every installed user. Restored the prior `VERSION`-file + `importlib.metadata.metadata("cursus")` resolution so `__version__` tracks the actual installed/working-tree version.
- **README install commands** — two `pip install cursus[xgboost]` references corrected to `cursus[gbm]` (the actual declared extra; `xgboost` is not an extra group). Users copy-pasting from the README got `WARNING: cursus does not provide the extra xgboost` and silently ended up without xgboost installed.

## [1.5.2] - 2026-05-14

### Changed

- **Defensive upper bound on Pydantic** — `pydantic>=2.11.0` → `pydantic>=2.11.0,<3`. Pydantic v1→v2 broke public validator APIs; pre-emptively capping prevents a future Pydantic 3 release from auto-upgrading users into breakage. The codebase uses Pydantic v2-only APIs (`field_validator`, `model_validator`, `ConfigDict`, `PrivateAttr`) across `cursus/steps/configs/` and `cursus/core/`.

### Removed

Dependency-tree pruning: 6 declared deps that have **zero imports in `src/cursus/`** (the only path shipped in the wheel). All removals are either pulled transitively by other declared deps (so install behavior is unchanged) or genuine orphans.

- **Core**:
  - `requests>=2.32.0` — pulled transitively by `sagemaker` (declared) and `transformers` (declared in `[nlp]`).
  - `packaging>=24.2.0` — pulled transitively by `sagemaker` (`packaging<25,>=23.0`), `transformers` (`>=20.0`), and `pytorch-lightning` (`>=20.0`).
  - `typing_extensions>=4.14.0` — pulled transitively by `pydantic` v2.x.
- **`[pytorch]` extra**:
  - `torchmetrics>=1.0.0` — pulled transitively by `pytorch-lightning>=2.0.0` (`torchmetrics>=0.7.0`).
- **`[nlp]` extra**:
  - `spacy>=3.7.0` — genuine orphan, no imports anywhere in the shipped package.
  - `huggingface-hub>=0.20.0` — pulled transitively by `transformers>=4.30.0` (`>=0.30.0`) and `tokenizers>=0.15.0` (`>=0.16.4`).

### Notes

- Net effect: 10 → 7 core deps, 4 → 3 in `[pytorch]`, 4 → 2 in `[nlp]`. Smaller user-visible dep tree on `pip install cursus`; runtime install graph unchanged because each removed dep is pulled via another declared dep (except `spacy`, which was unused entirely).
- `[viz]` and `[jupyter]` extras retain `seaborn`, `plotly`, `ipython`, etc. even though they don't appear in the shipped `src/cursus/` tree — they support the demo notebooks at the repo root which users cloning the repo may run.
- Audit method: AST-walked every `.py` under `src/cursus/`, collected top-level imports, cross-referenced against the distribution-name → import-name mapping for each declared dep. Only deps with zero direct imports AND covered by a transitive path (or proven orphan) were removed.

## [1.5.1] - 2026-05-14

### Changed

- **Lower Python floor to 3.9** — `requires-python` widened from `>=3.11` to `>=3.9`. Codebase audit confirms no use of Python 3.10/3.11-only syntax (no `match`/`case` pattern matching, no PEP-604 runtime unions without `__future__`, no `typing.Self` / `Required` / `LiteralString`, no `tomllib`, no `except*`).
- **`click>=8.1.0`** (was `>=8.2.0`) — click 8.2 dropped Python 3.7–3.9 support; relaxing to 8.1+ lets pip's resolver pick click 8.1.x on Python 3.9 and click 8.2+ on Python 3.10+. Only dependency adjustment required to lower the Python floor.
- **PyPI classifiers** — added `Python :: 3.9` and `Python :: 3.10` to advertise the broader support range.

### Notes

- All other pinned dependency floors (`pandas>=2.1.0`, `numpy>=1.26.0`, `sagemaker>=2.248.0`, `pydantic>=2.11.0`, `torch>=2.0.0`, etc.) already supported Python 3.9. No other dependency edits were needed.
- The `>=3.9` floor is unverified by CI in this release; if you exercise it in production on Python 3.9 / 3.10, please report any issues so a verification matrix can be added.

## [1.5.0] - 2026-05-14

### Added

- **Streaming-Mode Support Across Training & Preprocessing** — broad rollout of streaming-mode data handling for memory-efficient large-corpus training and preprocessing.
  - **Streaming-mode PyTorch training** — added streaming-mode datasets to PyTorch training; multi-GPU + multi-worker streaming support including FSDP.
  - **Iterable datasets for distributed training** — new pipeline iterable datasets that work with FSDP; larger inference batch sizes enabled.
  - **Streaming-mode tabular preprocessing** — full-parallel streaming-mode tabular preprocessing; combines shards on the fly; avoids intermediate shard files; memory-efficient implementation.
  - **Streaming-mode missing-value imputation** — full-parallel streaming-mode imputation with on-the-fly `fit_stream` adjustments.
  - **Streaming-mode risk-table mapping** — streaming variant for risk-table mapping in tabular pipelines.
  - **Streaming-mode temporal-split preprocessing** — parallel streaming-mode temporal split with cid-dedup handling.
  - **Streaming-mode dummy data loading** — streaming dummy-data loader for end-to-end pipeline development.
- **Custom BPE Tokenizer Pipeline** — first-class custom BPE tokenizer processor with installation, training, processor registry integration, and collation alignment across pytorch train/eval/inference.
- **SingleNodeCompiler** — new compiler that compiles a single DAG node for testing; documentation and `pytorch training/eval` fixes shipped alongside.
- **PipelineDAG Save/Load** — `PipelineDAG` instances can now be serialized to and loaded from file.
- **Pipeline Connection Analyzer** — new analyzer for inspecting pipeline connections.
- **CLI Refactor & Subcommands** — `cli compile`, `cli exe-doc`, `cli catalog` added; CLI tooling cleanup.
- **Hyperparameters Surface** — new hyperparameters configuration plus alignment fixes across pytorch + tokenizer pathways.

### Changed

- **PyTorch Training Realignment** — multi-pass PyTorch training updates aligning model-eval, inference handler, and tokenizer-aware preprocessing; multiple bug-fix iterations for stability.
- **Pytorch Model Eval & Inference Handler** — repeated updates to model-eval and inference-handler paths, including streaming-mode integration and tokenizer alignment.
- **Tabular Preprocessing Memory Efficiency** — streaming-mode roll-in plus targeted memory-efficiency improvements for tabular preprocessing.
- **DAG Config Factory Performance** — caching introduced; performance improvements for repeated factory calls.
- **Multitask Percentile Calibration** — calibration logic updates.
- **Tokenizer Folder Layout** — `tokenizer/` renamed to avoid module/import conflicts.
- **Names3Risk Text Field Conversion** — text-field handling reworked.
- **Package Metadata Refactor** — `src/cursus/__init__.py` now derives `__title__` / `__version__` / `__description__` / `__author__` from `pyproject.toml` (via `importlib.metadata`) + `VERSION` file as the single source of truth — no hardcoded values; removed the obsolete `src/cursus/__version__.py`.
- **Internal package-name references standardized to `Cursus`** — naming cleanup; `pyproject.toml` already declared `name = "cursus"` so no PyPI distribution-name change.

### Fixed

- **PyTorch Model Eval Bugs** — multiple eval-path bugs squashed.
- **Tokenizer / Processor Alignment** — fixed errors at the pytorch-training × tabular-preprocessing × tokenizer × processor boundary.
- **Sensitive Info Removal** — removed sensitive content from a tracked path.

### Removed

- **Redundant CLI Tests** — `test/cli/test_builder_test_cli.py`, `test/cli/test_pipeline_cli.py`, `test/cli/test_workspace_cli.py` and adjacent stale tests deleted as part of CLI refactor.
- **Config-UI Surface** — `src/cursus/api/config_ui/` removed (enhanced widget, README/docs, sub-modules) — superseded by other configuration paths.
- **Legacy `__version__.py` Module** — superseded by the metadata refactor (pyproject + VERSION single source of truth).
- **Claude Workspace Artifacts** — `add claude` commit `f2fc4472` added Claude Code workspace artifacts (not user-facing).
- **Presentation Slides** — `add slides` commit `ca9c8ed9` (assets in `docs/`).

### Technical Notes

- ~50 commits across 253 changed files (+55,475 / −31,594) since `v1.4.7`.
- Streaming-mode rollout is the headline feature: PyTorch training, tabular preprocessing, missing-value imputation, risk-table mapping, temporal-split, and dummy-data loading all gain streaming variants in this release.
- Package metadata is now declarative: bumping `VERSION` is the only file you must edit to ship a new release; everything else flows from `pyproject.toml`.

## [1.4.7] - 2026-01-07

### Added
- **Tokenizer Training Support** - Added new tokenizer training step and related components
  - **Tokenizer Training Step** - New processing step for training tokenizers
  - **BPE Tokenizer Implementation** - Complete implementation of BPE tokenizer for NLP workflows
  - **Tokenizer Training Integration** - Integration with PyTorch training workflows

- **Semi-Supervised Learning Framework** - Enhanced support for semi-supervised learning workflows
  - **PyTorch SSL Support** - Extended PyTorch training to support semi-supervised learning
  - **LightGBM SSL Support** - Added semi-supervised learning capabilities to LightGBM training
  - **SSL Pipeline Integration** - Integrated SSL workflows into existing pipeline templates

### Enhanced
- **PyTorch Training Improvements** - Major enhancements to PyTorch training capabilities
  - **Model Path Configuration** - Added model path to Lightning models for testing data saving
  - **Name3risk Training Updates** - Updated PyTorch training specifically for names3risk models
  - **Tokenizer Training Alignment** - Updated tokenizer training to fit PyTorch training patterns

- **Model Architecture Refactoring** - Significant improvements to model organization and structure
  - **Name3risk Model Updates** - Renamed and reorganized lstm2risk and transformer2risk models
  - **Common Structure Extraction** - Extracted common structures from TSA, names3risk, and bimodal BERT models
  - **PyTorch Module Expansion** - Expanded PyTorch modules under TSA for better modularity

- **Folder and Code Organization** - Major improvements to project structure and organization
  - **Model Reorganization** - Multiple reorganization efforts for better code structure
  - **Name3risk Model Structure** - Reorganized folder structure for names3risk models
  - **PyTorch Design Reorganization** - Reorganized design for PyTorch models for better maintainability

### Fixed
- **MTGBM Legacy Cleanup** - Removed redundant legacy implementations
  - **MTGBM Redundancy Removal** - Removed redundant legacy implementation of MTGBM
  - **Code Cleanup** - General code cleanup and organization improvements

### Technical Details
- **Tokenizer Training Architecture** - Complete tokenizer training framework with BPE implementation
- **SSL Integration** - Full integration of semi-supervised learning across multiple frameworks
- **PyTorch Training Enhancements** - Enhanced PyTorch training with better model path handling and testing support
- **Model Organization** - Improved model organization through restructuring and common pattern extraction
- **Code Quality** - Enhanced code quality through refactoring and cleanup efforts

### Quality Assurance
- **Enhanced Testing** - Improved testing for tokenizer training and SSL workflows
- **Integration Testing** - Comprehensive integration testing for model reorganization
- **Code Quality Validation** - Validation of code quality improvements through refactoring

### Performance Improvements
- **Training Performance** - Improved training performance with better model path handling
- **Tokenizer Training Efficiency** - Optimized tokenizer training workflows
- **Model Loading** - Enhanced model loading and saving capabilities for testing data

## [1.4.6] - 2025-12-11

### Added
- **Multi-Task Learning Enhancements** - Extended multi-task learning capabilities and model correspondences
  - **MTL Model Correspondence** - Enhanced correspondence between multi-task learning models
  - **MTL Calibration Support** - Expanded precision-based calibration for multi-task models
  - **MTL Metric Computation** - Extended model metric computation to support multi-task scenarios

- **LightGBM Multi-Task Improvements** - Enhanced LightGBM multi-task training and evaluation
  - **LightGBMMT Training Updates** - Updated LightGBM multi-task training to collect environment variables in config
  - **LightGBMMT Processing Steps** - Enhanced processing steps to support secure PyPI installation
  - **LightGBMMT Model Evaluation** - Improved LightGBM multi-task model evaluation with customized module support
  - **LightGBMMT Input Validation** - Added input validation to LightGBM multi-task training

- **Security and Package Management** - Improved security and package management capabilities
  - **Secure PyPI Installation** - Added support for optional secure PyPI package installation
  - **Environment Variable Management** - Enhanced environment variable handling in training configurations
  - **Security Updates** - Updated multi-task learning models with security improvements

### Enhanced
- **Inference System Improvements** - Improvements to inference performance and reliability
  - **PyTorch Inference Alignment** - Enhanced alignment between PyTorch inference handler and training
  - **Inference Handler Refinements** - Refined output field handling in PyTorch inference
  - **ONNX Performance Optimization** - Further optimized ONNX inference performance
  - **Inference Latency Reduction** - Reduced runtime latency in preprocessing steps

- **Processing Infrastructure Improvements** - Major enhancements to data processing capabilities
  - **Preprocessing Optimization** - Additional optimizations for preprocessing steps
  - **Concurrent Processing** - Fixed concurrency issues in BERT optimization
  - **Temporal Processing** - Enhanced temporal processing capabilities for time series data

- **Pipeline Configuration Enhancements** - Improved configuration handling and management
  - **Multi-Region Hyperparameter Support** - Added support for multi-region hyperparameters in training
  - **Task Label Configuration** - Adjusted configuration for task label naming fields
  - **Payload Method Expansion** - Expanded payload method to allow customer-provided inputs and support multimodal inputs

### Fixed
- **Concurrency and Safety Issues** - Resolved critical concurrency and safety issues
  - **Thundering Herd Mitigation** - Addressed safety issues and mitigated thundering herd problems
  - **Distributed Training Fixes** - Fixed synchronization issues in distributed training
  - **Race Condition Prevention** - Added locker mechanisms to prevent racing issues in GPU instances

- **Model Artifact Handling** - Improved handling of model artifacts and temporary files
  - **Temporary Folder Issues** - Fixed errors when saving to temporary folders in inference handlers
  - **Artifact Path Resolution** - Resolved issues with model artifact path resolution
  - **Model Registration** - Added registration step for better model management

- **Script and Configuration Fixes** - Various fixes to scripts and configuration handling
  - **Missing Environment Variables** - Fixed missing environment variables in builders
  - **Test Configuration Issues** - Resolved issues in test configurations
  - **Requirements Updates** - Updated requirements to support pandas 2.0.0+ and matplotlib 3.9

- **Pipeline Component Fixes** - Enhanced reliability of various pipeline components
  - **Dummy Data Loading** - Added job type support to dummy data loading builder
  - **Bedrock Batch Processing** - Fixed test issues in Bedrock batch processing
  - **Label Ruleset Execution** - Fixed bugs in label ruleset execution
  - **Active Sampling** - Enhanced active sampling capabilities

### Technical Details
- **Multi-Task Architecture** - Enhanced multi-task learning architecture with improved calibration and metric computation
- **LightGBMMT Improvements** - Comprehensive improvements to LightGBM multi-task training, evaluation, and inference
- **Security Enhancements** - Added secure package installation options and improved environment variable management
- **Performance Optimizations** - Continued performance improvements in preprocessing and inference operations
- **Concurrency Management** - Enhanced concurrency handling with locker mechanisms and thundering herd mitigation

### Quality Assurance
- **Enhanced Testing** - Improved test coverage for multi-task learning and LightGBM multi-task components
- **Concurrency Validation** - Thorough validation of concurrency fixes and safety improvements
- **Security Testing** - Enhanced security testing with secure PyPI installation validation
- **Integration Testing** - Comprehensive integration testing of pipeline components and configurations

### Performance Improvements
- **Inference Performance** - Optimized ONNX inference and reduced preprocessing latency
- **Training Efficiency** - Improved training efficiency with better environment variable management
- **Memory Management** - Enhanced memory management in distributed training scenarios
- **Processing Speed** - Optimized processing steps for better overall pipeline performance

## [1.4.5] - 2025-11-22

### Added
- **Multi-GPU Training Support** - Enhanced PyTorch training with multi-GPU capabilities
  - **Gradient Checkpointing** - Enabled gradient checkpointing for memory-efficient training
  - **Multi-GPU Synchronization** - Fixed synchronization issues for distributed training
  - **Multi-GPU Model Evaluation** - Updated model evaluation to support multi-GPU configurations
  - **PyTorch Training Instances** - Added new instance configurations for PyTorch training

- **Model Parallelism Design** - Comprehensive design for model parallelism implementation
  - **Model Parallelism Architecture** - Detailed design documentation for model parallelism
  - **Model Design Index** - Created model design index for better organization and reference

- **Training Configuration Enhancements** - Improved training configuration and management
  - **Max Run Time Configuration** - Added maximum run time configuration with default of 2 days
  - **Instance Type Updates** - Updated training instance types for better performance
  - **Training Pipeline Updates** - Enhanced pipeline configurations for testing scenarios

### Enhanced
- **Inference System Improvements** - Major improvements to inference performance and reliability
  - **Version Removal** - Removed __version__ from inference scripts for cleaner deployments
  - **Model Calibration** - Converted model calibration results to dictionary format for better handling
  - **Latency Reduction** - Reduced inference latency through optimization

- **Data Processing Enhancements** - Improved data handling and processing capabilities
  - **Text Field Handling** - Enhanced text field handling with optional text names
  - **Text Truncation** - Added text truncation for better memory management
  - **Field Name Cleanup** - Removed redundant suffixes in field names for cleaner data structures
  - **Field Filtering** - Enhanced field filtering in TensorBoard for better visualization

- **Model Artifact Support** - Extended support for pretrained models
  - **Pretrained Model Input** - Updated specifications to allow pretrained models as model artifact input
  - **Model Artifact Integration** - Enhanced integration of model artifacts in pipeline workflows

### Fixed
- **Multi-GPU Synchronization Issues** - Comprehensive fixes for distributed training
  - **Sync Issue Resolution** - Fixed synchronization issues affecting multi-GPU training
  - **Anti-Deadlock Mechanism** - Implemented anti-deadlock measures for distributed training
  - **Dataset Synchronization** - Fixed dataset synchronization across multiple GPUs

- **Data Processing Fixes** - Critical fixes for data handling and processing
  - **Chunker Bug Fix** - Fixed bug in data chunking functionality
  - **Attention Mask Alignment** - Fixed errors due to attention mask and input ID mismatch
  - **Text Field Overwriting** - Fixed text field overwriting issues in dataset processing

- **Field Mapping Improvements** - Enhanced field mapping and risk table handling
  - **Risk Table Mapping** - Avoided risk table mapping on text fields for better accuracy
  - **Field Name Redundancy** - Removed redundant suffixes and improved field naming consistency

- **TensorBoard Optimization** - Improved TensorBoard logging and visualization
  - **Excessive Logging Prevention** - Avoided saving too much data in TensorFlow for better performance
  - **Field Filtering** - Enhanced field filtering to reduce TensorBoard clutter

- **Configuration Management** - Improved configuration handling and management
  - **Gitignore Updates** - Updated .gitignore file for better version control
  - **Hyperparameter Registry Cleanup** - Removed hyperparameter registry for cleaner architecture
  - **BSM Hyperparameter Updates** - Updated BSM hyperparameter naming conventions

### Technical Details
- **Multi-GPU Architecture** - Complete multi-GPU training support with gradient checkpointing and synchronization
- **Model Parallelism** - Comprehensive model parallelism design with detailed documentation
- **Inference Optimization** - Reduced inference latency and removed version dependencies
- **Training Configuration** - Enhanced training configuration with max run time and instance type updates
- **Data Processing** - Improved text field handling, truncation, and field mapping

### Quality Assurance
- **Distributed Training Validation** - Comprehensive validation of multi-GPU training functionality
- **Synchronization Testing** - Thorough testing of synchronization mechanisms and anti-deadlock measures
- **Performance Testing** - Validation of inference latency improvements and TensorBoard optimizations
- **Integration Testing** - End-to-end testing of pretrained model artifact integration

### Performance Improvements
- **Multi-GPU Training Performance** - Optimized multi-GPU training with gradient checkpointing and synchronization
- **Inference Latency** - Reduced inference latency through optimization and cleanup
- **Memory Efficiency** - Improved memory efficiency with text truncation and gradient checkpointing
- **TensorBoard Performance** - Enhanced TensorBoard performance by reducing excessive logging

## [1.4.4] - 2025-11-17

### Added
- **Semi-Supervised Learning Framework** - Comprehensive support for semi-supervised learning workflows
  - **Semi-Supervised Learning Training Steps** - New training steps for SSL workflows with labeled and unlabeled data
  - **Active Sample Selection Step** - Complete step for active learning and sample selection strategies
  - **Pseudo Label Merge Step** - New step for merging pseudo-labeled data with labeled datasets
  - **Labeled Data Merger** - Enhanced data merger for combining labeled and pseudo-labeled datasets
  - **SSL Training Pipelines** - Complete pipeline templates for semi-supervised learning workflows

- **Active Learning Support** - Advanced active learning capabilities for model improvement
  - **Active Sampling Steps** - New steps for active sampling strategies and data selection
  - **Active Sample Selection Contract** - Enhanced contract definitions for active sampling workflows
  - **Sample Selection Builder** - Complete builder for active sample selection step

### Enhanced
- **Dependency Management** - Updated dependencies for better compatibility
  - **LightGBM Dependency** - Added LightGBM to core dependencies for extended ML framework support
  - **Python 3.11+ Requirement** - Updated README to reflect Python 3.11+ requirement for modern features
  - **PyProject Configuration** - Fixed pyproject.toml configuration for better package management

- **Contract and Configuration Updates** - Improved contract definitions and configurations
  - **SSL Contract Updates** - Enhanced contracts for semi-supervised learning workflows
  - **Active Sampling Configuration** - Improved configuration handling for active sampling steps
  - **Data Merger Contracts** - Enhanced contract definitions for data merging operations

### Fixed
- **PyProject Configuration** - Fixed issues in pyproject.toml affecting package installation and dependency resolution
- **Contract Alignment** - Updated contracts to align with semi-supervised learning requirements
- **Step Registration** - Added missing steps to __init__ for proper module discovery

### Technical Details
- **SSL Architecture** - Complete semi-supervised learning framework with training, active sampling, and pseudo-labeling
- **Active Learning System** - Advanced active learning capabilities with sample selection and data merging
- **Pipeline Integration** - Full integration of SSL workflows into pipeline catalog and execution system
- **LightGBM Support** - Extended support for LightGBM in addition to existing XGBoost and PyTorch frameworks

### Quality Assurance
- **Workflow Testing** - Comprehensive testing of semi-supervised learning workflows
- **Contract Validation** - Enhanced validation of SSL-related contracts and specifications
- **Pipeline Validation** - Thorough validation of SSL pipeline templates and execution

### Performance Improvements
- **Data Merging Performance** - Optimized data merging operations for large-scale SSL workflows
- **Sample Selection Efficiency** - Enhanced efficiency of active sample selection algorithms
- **Pipeline Execution** - Improved execution performance for SSL training pipelines

## [1.4.3] - 2025-11-17

### Added
- **Multi-Task Gradient Boosting (MT-GBM) Framework** - Comprehensive multi-task learning capabilities
  - **MT-GBM Training Step** - New training step for multi-task gradient boosting models
  - **MT-GBM Inference Step** - Dedicated inference step for MT-GBM models
  - **LightGBM Multi-Task Support** - Complete LightGBM multi-task compiled C code implementation
  - **MT-GBM Analysis Tools** - Analysis and evaluation tools for multi-task models

- **LightGBM Framework Integration** - Full LightGBM support for various ML workflows
  - **LightGBM Training Step** - New SKLearn-based LightGBM training step using SKLearn container
  - **LightGBM Model Evaluation** - Enhanced model evaluation capabilities for LightGBM
  - **LightGBM Model Inference** - Complete inference pipeline for LightGBM models
  - **LightGBM Multi-Task Extensions** - Extended support for multi-task LightGBM workflows

- **Multi-Label Classification Support** - Comprehensive multi-label ML capabilities
  - **Multi-Label Preprocessing Step** - New preprocessing step for multi-label classification tasks
  - **Multi-Label Ruleset System** - Label ruleset generation and execution for multi-label scenarios
  - **Multi-Label Extensions** - Extended label ruleset transform for multi-label workflows

- **Label Ruleset System** - Advanced label transformation and optimization
  - **Label Ruleset Generation Step** - Automated generation of label transformation rules
  - **Label Ruleset Execution Step** - Execution engine for label transformation rulesets
  - **Label Ruleset Optimization** - Optimization algorithms for ruleset efficiency
  - **Ruleset-Based Pipelines** - Complete pipelines with ruleset-based label transformation

- **Bedrock Enhancements** - Expanded AWS Bedrock integration capabilities
  - **Bedrock Batch Processing Role** - Enhanced role management for batch processing
  - **Multi-Job Bedrock Processing** - Support for multiple concurrent Bedrock jobs
  - **Improved Prompt Generation** - Enhanced prompt template generation with better JSON handling
  - **Bedrock Pipeline Expansion** - Extended Bedrock pipelines with ruleset integration

### Enhanced
- **Format Preservation Strategy** - Comprehensive format handling across processing steps
  - **Universal Format Detection** - Automatic format detection for all processing steps
  - **Metric Computation Format** - Enhanced format preservation for metric computation
  - **Input/Output Format Consistency** - Maintained format consistency across pipeline steps

- **Processing Infrastructure Improvements** - Major enhancements to data processing
  - **Processor Alignment** - Aligned processors with corresponding processing steps
  - **Processor Reorganization** - Renamed and restructured processors for better clarity
  - **Iterative Data Loading** - Tabular preprocessing now supports iterative loading for large datasets
  - **Lightning Model Reorganization** - Restructured projects to better organize Lightning models

- **Path and Input/Output Management** - Enhanced path handling and I/O configurations
  - **Input/Output Path Alignment** - Fixed multiple input/output path misalignment issues
  - **Unique Logical Names** - Renamed input/output logical names for global uniqueness
  - **Artifact Passing** - Preprocessing steps now properly pass model artifacts down the pipeline
  - **XGBoost Artifact Integration** - XGBoost training now accepts preprocessing artifacts

- **Configuration and Contract Improvements** - Better configuration handling
  - **Currency Conversion Simplification** - Simplified and structured input/output/env variables for currency conversion
  - **Tabular Preprocessing Contract** - Corrected contract to allow no-label input scenarios
  - **JSONL Output Handling** - Changed expected output to jsonl.out file from batch mode
  - **Nested Instance Handling** - Enhanced handling of nested multiple instances

- **Script and Step Catalog Enhancements** - Improved script discovery and catalog management
  - **Internal Script Search Strategy** - Added strategy to search internal scripts more effectively
  - **Step Catalog LightGBM Support** - Adjusted step catalog to support LightGBM framework
  - **Verbose Log Suppression** - Suppressed verbose logging in step catalog for cleaner output
  - **Generic Path Discovery** - Added generic path discovery capabilities

### Fixed
- **Dependency and Package Management** - Critical dependency and installation fixes
  - **Dependency Updates** - Updated package dependencies for better compatibility
  - **Package Install Issues** - Fixed package installation issues across various scripts
  - **PyPI Package Installation** - Added public/secure PyPI install in train/inference for XGBoost
  - **Pip Install Upgrade** - Upgraded pip installation process for reliability
  - **Boto3 Upgrade** - Updated boto3 for latest AWS features and bug fixes

- **Parsing and Format Issues** - Comprehensive parsing error fixes
  - **Output Parsing Issues** - Fixed output parsing errors and truncation problems
  - **JSON Format Handling** - Fixed JSON output format to handle {{ and special characters
  - **German Double Quote Fix** - Fixed German double quote issues causing parsing errors
  - **Output Format Preservation** - Fixed output format issues for JSON output from LLM
  - **Input/Output Ordering** - Preserved input and output ordering in configurations

- **Script and Contract Alignment** - Enhanced alignment between scripts and contracts
  - **PyTorch Training Fix** - Fixed error in PyTorch training script with new optional input
  - **Script Contract Mismatch** - Corrected multiple script-contract mismatches
  - **Duplicate Alias Fix** - Resolved duplicate alias issues in configurations
  - **Filename Mismatch** - Fixed filename output and mismatch issues

- **Bedrock Processing Fixes** - Specific fixes for Bedrock workflows
  - **Bedrock Batch Processing** - Fixed errors in calling Bedrock batch processing
  - **Job Name Issues** - Corrected job name generation for Bedrock jobs
  - **Prompt Config Location** - Changed prompt config location to be local in source directory
  - **Uncompressed File Handling** - Fixed handling of uncompressed files in Bedrock workflows
  - **File Size Limit Checks** - Added file size limit validation

- **Test Infrastructure** - Test reliability improvements
  - **Test Error Fixes** - Fixed errors in various test suites
  - **Bedrock Batch Processing Tests** - Fixed tests for Bedrock batch processing
  - **Script Test Corrections** - Corrected script testing issues
  - **DAG Config Test Fixes** - Fixed DAG configuration test issues

- **Code Quality and Organization** - General code cleanup and improvements
  - **Redundant Step Specs Removal** - Removed redundant step specifications with duplicate I/O
  - **Legacy Code Separation** - Separated legacy code from refactored implementations
  - **Debug Message Cleanup** - Removed unnecessary debug messages for cleaner logs
  - **Reformat and Cleanup** - Applied code formatting and cleanup across multiple modules

### Technical Details
- **MT-GBM Architecture** - Complete multi-task gradient boosting framework with LightGBM integration
- **LightGBM Integration** - Full LightGBM support using SKLearn framework containers
- **Multi-Label Framework** - Comprehensive multi-label classification with ruleset-based transforms
- **Label Ruleset System** - Advanced label transformation with generation, execution, and optimization
- **Format Preservation** - Universal format detection and preservation across all processing steps
- **Bedrock Enhancements** - Extended Bedrock capabilities with batch processing and multi-job support
- **Path Management** - Enhanced input/output path handling with global unique logical names
- **Processor Architecture** - Aligned and restructured processor system for better maintainability

### Quality Assurance
- **Enhanced Testing** - Improved test coverage for MT-GBM, LightGBM, and multi-label workflows
- **Configuration Validation** - Better validation for complex multi-task and multi-label configurations
- **Contract Alignment** - Systematic alignment improvements between scripts, contracts, and specifications
- **Format Consistency** - Ensured format consistency across all processing steps and pipelines

### Performance Improvements
- **Iterative Loading** - Memory-efficient iterative loading for large datasets in tabular preprocessing
- **Package Installation** - Optimized package installation with environment variable controls
- **Processing Efficiency** - Improved processing efficiency with aligned processors and format preservation
- **Catalog Performance** - Reduced verbose logging for faster step catalog operations

## [1.4.2] - 2025-11-03

### Added
- **AWS Bedrock Integration** - Comprehensive support for AWS Bedrock services in ML pipelines
  - **Bedrock Processing Step** - New processing step for Bedrock-based operations with job type support
  - **Bedrock Prompt Template Generation** - Advanced prompt template generation capabilities for Bedrock models
  - **Bedrock Prompt Generator** - Dedicated prompt generator for Bedrock workflows
  - **Bedrock Pipeline Templates** - Complete pipeline templates integrating Bedrock processing with PyTorch models
  - **Bedrock Script Processing** - Enhanced script processing utilities for Bedrock operations

- **Trimodal Model Support** - Advanced multimodal machine learning capabilities
  - **Trimodal BERT Integration** - Complete trimodal BERT implementation in PyTorch RNR framework
  - **Trimodal Data Handling** - Enhanced data loaders to handle trimodal input with proper field name and dimension matching
  - **Trimodal Model Training** - Updated model training infrastructure to support trimodal model requirements
  - **Trimodal Model Inference** - Enhanced model inference capabilities for trimodal architectures

- **Feature Selection Framework** - New feature selection capabilities for improved model performance
  - **Feature Selection Step** - Dedicated step builder for feature selection operations
  - **XGBoost Feature Selection Integration** - Enhanced XGBoost training to handle feature selection workflows
  - **Training Artifact Optimization** - Optimized training artifacts by removing unnecessary feature selection saves

- **Enhanced Model Evaluation** - Expanded model evaluation and inference capabilities
  - **PyTorch Model Evaluation Step** - New step builder for PyTorch model evaluation workflows
  - **PyTorch Model Inference Step** - Enhanced inference step for PyTorch models with improved performance
  - **Score Percentile Calculation** - New score percentile calculation utilities for model evaluation
  - **Model Performance Analytics** - Enhanced analytics for model performance assessment

- **Advanced Processing Infrastructure** - Improved processing capabilities and framework support
  - **Framework Processor Integration** - Enhanced Framework Processor support for risk table and percentile calibration
  - **XGBoost Model Loading** - Improved XGBoost model.tar.gz loading capabilities
  - **Calibration System Updates** - Enhanced calibration system with improved processing workflows
  - **Processing Pipeline Optimization** - Optimized processing pipelines for better performance and reliability

### Enhanced
- **Dependency Resolution System** - Major improvements to dependency matching and resolution
  - **Bedrock Dependency Matching** - Enhanced dependency resolution to properly match Bedrock processing steps with PyTorch models
  - **Job Type Integration** - Improved job type handling for Bedrock processing with better pipeline integration
  - **Cross-Framework Dependencies** - Better dependency resolution across different ML frameworks and processing types

- **Pipeline Template System** - Significant improvements to pipeline template architecture
  - **Bedrock Pipeline Integration** - New pipeline templates showcasing Bedrock and PyTorch integration patterns
  - **Multimodal Pipeline Support** - Enhanced pipeline templates supporting trimodal and multimodal workflows
  - **Feature Selection Pipelines** - New pipeline examples demonstrating feature selection integration

- **Configuration Management** - Enhanced configuration handling and validation
  - **Bedrock Configuration** - New configuration classes for Bedrock prompt template generation and processing
  - **Trimodal Configuration** - Enhanced configuration support for trimodal model requirements
  - **Template Configuration Updates** - Improved template configuration with redundant input removal

- **Model Training Infrastructure** - Major improvements to training capabilities
  - **PyTorch Training Enhancements** - Fixed and enhanced PyTorch training with better error handling and performance
  - **XGBoost Training Updates** - Updated XGBoost training with feature selection support and improved model inference
  - **Training Pipeline Optimization** - Optimized training pipelines for better resource utilization and performance

### Fixed
- **Contract and Configuration Issues** - Critical fixes to contracts and configurations
  - **Contract Base Fixes** - Fixed contract base implementation for better reliability and consistency
  - **Configuration Alignment** - Resolved configuration alignment issues across different step types
  - **Template Configuration** - Fixed template configuration issues and removed redundant inputs

- **Model Training and Inference Fixes** - Comprehensive fixes to model training and inference
  - **PyTorch Training Fixes** - Fixed critical issues in PyTorch training workflows and data handling
  - **Model Inference Updates** - Enhanced model inference reliability and performance
  - **Training Artifact Management** - Improved training artifact management and storage optimization

- **Processing Pipeline Stability** - Enhanced processing pipeline reliability
  - **Bedrock Processing Fixes** - Fixed issues in Bedrock processing workflows and script execution
  - **Data Loading Improvements** - Enhanced data loading reliability for trimodal and multimodal inputs
  - **Processing Error Handling** - Improved error handling and recovery in processing pipelines

- **Code Quality and Organization** - Systematic code cleanup and quality improvements
  - **Bug Fixes** - Multiple bug fixes across different components for improved stability
  - **Code Organization** - Better code organization and structure for maintainability
  - **Documentation Updates** - Enhanced documentation and planning documents

### Technical Details
- **Bedrock Integration Architecture** - Complete AWS Bedrock integration with processing steps, prompt generation, and pipeline templates
- **Trimodal Model Framework** - Comprehensive trimodal model support with BERT integration and enhanced data handling
- **Feature Selection System** - Advanced feature selection framework with XGBoost integration and training optimization
- **Model Evaluation Pipeline** - Enhanced model evaluation and inference capabilities with PyTorch support and score analytics
- **Processing Infrastructure** - Improved processing infrastructure with Framework Processor integration and enhanced calibration

### Quality Assurance
- **Enhanced Testing** - Comprehensive testing improvements with better coverage and reliability
- **Configuration Validation** - Improved configuration validation and alignment across all components
- **Pipeline Integration Testing** - Enhanced integration testing for Bedrock, trimodal, and feature selection workflows
- **Code Quality Standards** - Systematic code quality improvements with cleanup and standardization

### Performance Improvements
- **Training Performance** - Optimized training performance with better resource management and artifact handling
- **Inference Speed** - Enhanced inference performance with optimized model loading and processing
- **Processing Efficiency** - Improved processing efficiency with Framework Processor integration and pipeline optimization
- **Memory Management** - Better memory management in trimodal data handling and model operations

## [1.4.1] - 2025-10-24

### Added
- **Percentile-Based Model Calibration** - New advanced calibration capabilities for improved model performance
  - **Percentile Model Calibration Step** - New step for percentile-based model calibration with enhanced accuracy
  - **Percentile Inference and Evaluation** - Support for percentile-based inference and evaluation workflows
  - **Calibration Dictionary System** - Enhanced calibration dictionary for better model calibration management
  - **Percentile Pipeline Templates** - New pipeline templates with percentile-based calibration workflows

- **Temporal Self-Attention (TSA) Framework** - Comprehensive temporal modeling capabilities
  - **TSA Lightning Models** - Complete refactoring of temporal self-attention into PyTorch Lightning models
  - **Temporal Feature Engineering** - Advanced temporal feature engineering capabilities
  - **Temporal Sequence Normalization** - New step for temporal sequence normalization and preprocessing
  - **TSA Processing Module** - Comprehensive processing utilities for temporal self-attention workflows
  - **TSA Analysis Tools** - Analysis and visualization tools for temporal self-attention models

- **Enhanced Processing Infrastructure** - Expanded data processing capabilities
  - **New Processors Collection** - Additional processors for various data processing tasks
  - **Categorical Preprocessing Enhancements** - Improved categorical data preprocessing with better handling
  - **Signature Input Support** - Added signature input support for tabular preprocessing workflows
  - **Processing Module Expansion** - Extended processing module with new utilities and processors

- **Model Evaluation Enhancements** - Advanced model evaluation and comparison capabilities
  - **Model Score Comparison** - Added model score comparison in model evaluation and metric computation
  - **Model Wiki Generator Comparison** - Support for comparison features in model wiki generator
  - **Job Type Variants** - Enhanced job type variant support for model metric computation and dummy data loading

### Enhanced
- **Documentation and Organization** - Major improvements to project documentation and structure
  - **File Tagging System** - Comprehensive file tagging and organization system for better resource management
  - **Documentation Tags** - Enhanced documentation with improved tagging and categorization
  - **Planning Documentation** - Updated planning documents with current project status and roadmaps
  - **Entry Point Documentation** - New cursus package overview and entry point documentation
  - **Resource Organization** - Systematic moving of files to resource or archive directories for better organization

- **AWS Integration Improvements** - Enhanced AWS functionality and integration
  - **AWS Function Extensions** - New AWS function extensions for improved cloud integration
  - **Cache Management** - Disabled cache by default for better performance and reliability
  - **Robust Fallback Handling** - Enhanced fallback handling and test fixes for better reliability

- **Pipeline and Configuration Enhancements** - Improved pipeline configuration and management
  - **DAG Config Factory** - Fixed DAG config factory name matching for better pipeline generation
  - **Payload Config Improvements** - Fixed issues with payload config and removed unnecessary dependencies
  - **Hyperparameter Updates** - Updated hyperparameters for better model performance
  - **Development Set Management** - Cleaned up development set handling and removed redundant components

- **Testing Infrastructure** - Comprehensive testing improvements
  - **Script Testing Framework** - Enhanced script testing with better coverage and reliability
  - **Test Fixes** - Fixed all tests and improved test reliability across the system
  - **Robust Test Handling** - Improved test handling with better fallback mechanisms

### Fixed
- **Contract and Configuration Issues** - Critical fixes to contracts and configurations
  - **Dummy Data Loading Contract** - Fixed dummy data loading contract for better reliability
  - **Missing Value Imputation** - Corrected missing value imputation logic and implementation
  - **Script Errors** - Fixed various script errors and improved script reliability
  - **Contract Alignment** - Fixed contract alignment issues and improved validation

- **Processing and Pipeline Fixes** - Enhanced processing pipeline reliability
  - **Tabular Preprocessing** - Removed dependency on label field in tabular preprocessing for better flexibility
  - **Processing Pipeline Bugs** - Fixed various bugs in processing pipelines and data handling
  - **Pipeline Template Issues** - Fixed issues in pipeline templates and improved template reliability

- **Code Quality and Cleanup** - Systematic code cleanup and quality improvements
  - **File Cleanup** - Removed redundant files and cleaned up project structure
  - **Name Standardization** - Fixed naming inconsistencies across components
  - **Code Organization** - Improved code organization and removed unnecessary components

### Technical Details
- **Percentile Calibration Architecture** - Complete percentile-based calibration system with enhanced accuracy and performance
- **TSA Framework** - Comprehensive temporal self-attention framework with Lightning integration and processing utilities
- **Processing Pipeline Enhancement** - Expanded processing capabilities with new processors and improved data handling
- **Model Evaluation System** - Enhanced model evaluation with comparison capabilities and advanced metrics
- **Documentation System** - Improved documentation organization with tagging and better resource management

### Quality Assurance
- **Enhanced Testing** - Comprehensive testing improvements with better coverage and reliability
- **Contract Validation** - Improved contract validation and alignment across all components
- **Configuration Management** - Better configuration management with enhanced validation and error handling
- **Code Quality** - Systematic code quality improvements with cleanup and standardization

### Performance Improvements
- **Cache Optimization** - Disabled cache by default for better performance and reliability
- **Processing Performance** - Improved processing performance with enhanced algorithms and better resource management
- **Pipeline Execution** - Optimized pipeline execution with better resource utilization and performance
- **Model Calibration Performance** - Enhanced model calibration performance with percentile-based approaches

## [1.4.0] - 2025-10-17

### Added
- **Script Testing Infrastructure** - Revolutionary new script testing framework for ML pipeline validation
  - **Script Execution Registry** - Complete registry system for managing script execution contexts and dependencies
  - **Script Dependency Matcher** - Advanced dependency resolution system for script testing with intelligent matching
  - **Script Input Resolver** - Comprehensive input resolution system for script testing with contract-based validation
  - **Interactive Script Testing API** - New API for interactive script testing with Jupyter notebook integration
  - **Script Testing CLI** - Command-line interface for script testing operations and validation

- **Enhanced Validation Framework** - Major expansion of validation capabilities for script testing
  - **Input Collection System** - Advanced input collection with registry integration and dependency resolution
  - **Contract-Based Testing** - Script testing using contract definitions for input and output validation
  - **Message Passing System** - Sophisticated message passing for script testing with dependency resolution support
  - **Test Registry Integration** - Full integration between script testing and execution registry systems

- **Comprehensive Test Suite** - Extensive test coverage for script testing functionality
  - **Script Testing Tests** - Complete test suite for dummy data loading, missing value imputation, model metrics computation
  - **Stratified Sampling Tests** - Enhanced tests for stratified sampling with comprehensive validation
  - **XGBoost Model Inference Tests** - Comprehensive test suite for XGBoost model inference scripts
  - **Registry Integration Tests** - Tests for registry integration with input collection and dependency resolution

- **Interactive Development Tools** - Enhanced tools for interactive script development and testing
  - **Refactored Interactive Notebook** - New refactored notebook for interactive script testing with improved functionality
  - **Script Testing Documentation** - Comprehensive documentation for script testing implementation and design
  - **Implementation Plans** - Detailed implementation plans for dependency resolution and input collection

### Enhanced
- **Pipeline Template System** - Major improvements to pipeline template handling
  - **Template State Management** - Enhanced template state management in pipeline DAG compiler
  - **Execution Document Integration** - Improved integration between pipeline compilation and execution document generation
  - **Template Metadata Population** - Better metadata population during pipeline compilation process

- **CLI System Improvements** - Enhanced command-line interface capabilities
  - **Script Testing CLI** - New CLI module for script testing operations replacing runtime testing CLI
  - **Improved Integration** - Better integration between CLI components and validation systems
  - **Enhanced Error Handling** - Improved error handling and user experience in CLI operations

- **Code Organization and Architecture** - Major architectural improvements and code cleanup
  - **Module Refactoring** - Complete refactoring of runtime testing module to script testing module
  - **Code Redundancy Reduction** - Significant reduction in code redundancy through architectural improvements
  - **Simplified Architecture** - Streamlined architecture with clearer separation of concerns
  - **Enhanced Modularity** - Improved modularity with better component isolation and reusability

### Fixed
- **Runtime Testing Migration** - Complete migration from runtime testing to script testing architecture
  - **Module Restructuring** - Moved from `cursus.script_testing` to `cursus.validation.script_testing` for better organization
  - **Legacy Code Removal** - Removed deprecated runtime testing modules and replaced with script testing framework
  - **Import Path Updates** - Updated all import paths to reflect new script testing architecture
  - **Test Infrastructure Migration** - Migrated test infrastructure to support new script testing framework

- **Pipeline Compilation Issues** - Critical fixes to pipeline compilation and template handling
  - **Template State Issues** - Fixed template state management issues in pipeline DAG compiler
  - **Execution Document Generation** - Fixed timing issues with execution document generation and template metadata
  - **Compilation Sequencing** - Improved sequencing of compilation and document filling operations

- **Code Quality Improvements** - Major code quality enhancements through refactoring
  - **Redundant Code Elimination** - Removed significant amounts of redundant code through architectural improvements
  - **Module Organization** - Better organization of modules with clearer responsibilities and boundaries
  - **Import Cleanup** - Cleaned up import statements and dependencies throughout the codebase

### Technical Details
- **Script Testing Architecture** - Complete script testing framework with registry, dependency resolution, and input collection
- **Dependency Resolution System** - Advanced dependency resolution with message passing and contract-based validation
- **Interactive Testing Framework** - Comprehensive framework for interactive script testing with Jupyter integration
- **Registry Integration** - Full integration between script execution registry and testing infrastructure
- **Code Reduction Metrics** - Significant code reduction through architectural improvements and redundancy elimination

### Quality Assurance
- **Comprehensive Testing** - Extensive test coverage for all script testing components and functionality
- **Integration Validation** - Thorough validation of integration between script testing and existing systems
- **Documentation Quality** - Enhanced documentation with detailed implementation plans and design documents
- **Code Standardization** - Improved code quality through refactoring and architectural improvements

### Performance Improvements
- **Script Testing Performance** - Optimized script testing execution with improved dependency resolution and input collection
- **Registry Operations** - Enhanced performance of registry operations with better caching and optimization
- **Template Processing** - Improved template processing performance with better state management
- **CLI Responsiveness** - Enhanced CLI responsiveness with optimized script testing operations

## [1.3.9] - 2025-10-16

### Added
- **Interactive Runtime Testing Factory** - New comprehensive factory system for interactive runtime testing
  - **DAG Config Factory** - Complete factory system for DAG configuration generation with early validation
  - **Configuration Generator** - Advanced configuration generation with field extraction and mapping
  - **Field Extractor** - Intelligent field extraction system for configuration classes
  - **Config Class Mapper** - Enhanced mapping system for configuration class discovery and management

- **Enhanced UI System** - Major improvements to user interface and configuration management
  - **Universal Config UI** - Generalized UI system for configuration management across all step types
  - **Smart Default Values** - Intelligent default value population with base config inheritance
  - **Single Page Integration** - Unified single-page UI design for better user experience
  - **Jupyter Notebook Integration** - Complete integration with Jupyter notebooks for interactive development

- **New ML Pipeline Steps** - Expanded step library with additional ML capabilities
  - **LightGBM Training Step** - Complete LightGBM training step with JumpStart integration
  - **Dummy Data Loading Step** - Utility step for testing and development workflows
  - **Enhanced Cradle Data Loading** - Improved cradle data loading with dynamic data source support

### Enhanced
- **Configuration System Improvements** - Major enhancements to configuration management architecture
  - **Inheritance-Aware Field Generation** - Enhanced field generation with proper inheritance handling
  - **Unified Hyperparameter Location** - Standardized hyperparameter handling across all training steps
  - **Reduced Configuration Redundancy** - Eliminated redundant fields in payload step configurations
  - **Enhanced Config Factory** - Improved configuration factory with better validation and error handling

- **Runtime Testing Infrastructure** - Comprehensive improvements to runtime testing capabilities
  - **Script Runtime Testing** - Enhanced script-level runtime testing with better validation
  - **Script Discovery System** - Improved script discovery and validation mechanisms
  - **Runtime DAG Testing** - Enhanced DAG-level runtime testing with better error reporting
  - **API Config Factory Testing** - Complete test suite for API configuration factory components

- **Code Quality and Organization** - Major code cleanup and optimization efforts
  - **UI Code Redundancy Reduction** - Significant reduction in UI-related code redundancy
  - **Relative Import Migration** - Continued migration to relative imports for better modularity
  - **Path Security Improvements** - Fixed path reversal vulnerability and enhanced security measures
  - **Configuration Field Optimization** - Removed redundant hyperparameter fields and improved efficiency

- **User Experience Improvements** - Enhanced user interface and interaction design
  - **Verbose Message Support** - Added comprehensive verbose messaging for better debugging
  - **Smart UI Defaults** - Intelligent default value handling in UI components
  - **Previous Button Functionality** - Fixed navigation issues in UI workflows
  - **Window Duplication Fixes** - Resolved UI rendering issues and window management problems

### Fixed
- **UI System Stability** - Comprehensive fixes to user interface reliability
  - **Rendering Issue Resolution** - Fixed various UI rendering problems and display issues
  - **Window Management** - Resolved duplicated window issues and improved window handling
  - **Navigation Fixes** - Fixed Previous button functionality and navigation flow
  - **Widget Integration** - Improved widget integration and interaction handling

- **Configuration Management Fixes** - Enhanced configuration system reliability
  - **Hyperparameter Field Cleanup** - Removed redundant hyperparameter fields from configurations
  - **Config Validation** - Improved configuration validation and error handling
  - **Field Generation** - Fixed inheritance-aware field generation issues
  - **Default Value Handling** - Enhanced default value population and management

- **Testing Infrastructure Fixes** - Improved testing framework reliability
  - **Test Error Resolution** - Fixed multiple test errors across validation and runtime modules
  - **Mock Infrastructure** - Enhanced mock infrastructure for better test isolation
  - **Test Execution** - Improved test execution reliability and error reporting
  - **Integration Testing** - Fixed integration test issues and improved coverage

- **Security and Vulnerability Fixes** - Critical security improvements
  - **Path Reversal Vulnerability** - Fixed path reversal security vulnerability
  - **Input Validation** - Enhanced input validation and sanitization
  - **Security Hardening** - Improved overall security posture and vulnerability management

### Technical Details
- **Factory Architecture** - Complete factory system for configuration generation with intelligent field extraction
- **UI Framework** - Universal UI framework with smart defaults and inheritance-aware field handling
- **Runtime Testing** - Enhanced runtime testing infrastructure with comprehensive validation capabilities
- **Configuration Management** - Improved configuration system with reduced redundancy and better validation
- **Security Enhancements** - Critical security fixes and vulnerability remediation

### Quality Assurance
- **Test Coverage** - Expanded test coverage for factory system and UI components
- **Configuration Validation** - Enhanced configuration validation with early error detection
- **UI Testing** - Comprehensive UI testing with improved user experience validation
- **Security Testing** - Enhanced security testing and vulnerability assessment

### Performance Improvements
- **UI Performance** - Optimized UI performance with reduced code redundancy and better rendering
- **Configuration Processing** - Improved configuration processing speed with enhanced factory system
- **Runtime Testing Performance** - Optimized runtime testing execution with better resource management
- **Memory Usage** - Reduced memory footprint through code optimization and redundancy elimination

## [1.3.8] - 2025-10-05

### Enhanced
- **Testing Infrastructure Modernization** - Major overhaul of testing framework and coverage analysis
  - **Comprehensive Test Coverage Analysis** - New advanced coverage analysis tools with detailed reporting and gap identification
  - **Test Framework Refactoring** - Complete refactoring of test infrastructure with improved organization and reliability
  - **Coverage Metrics Enhancement** - Enhanced test coverage metrics with function-level analysis and redundancy detection
  - **Test Execution Optimization** - Improved test execution performance with better resource management and parallel execution

- **Configuration Field Management System Refactoring** - Complete redesign of configuration field management architecture
  - **Unified Config Manager** - New unified configuration manager with streamlined field handling and validation
  - **Step Catalog Aware Categorizer** - Enhanced categorizer with step catalog integration for better field classification
  - **Config Field Simplification** - Simplified configuration field system with reduced complexity and improved maintainability
  - **Performance Optimization** - Optimized configuration processing with better caching and reduced redundancy

- **Validation System Improvements** - Enhanced validation framework with better integration and user experience
  - **Simple Integration Validation** - New simplified validation integration for easier testing and validation workflows
  - **Universal Builder Tester Enhancements** - Improved universal builder testing with better coverage and reliability
  - **Validation Documentation Updates** - Comprehensive documentation updates reflecting new validation experience and best practices
  - **API Reference Improvements** - Enhanced API reference documentation for validation tools and testing frameworks

### Added
- **Advanced Testing Tools** - New comprehensive testing infrastructure and analysis capabilities
  - **Test Coverage Analysis Tools** - Advanced tools for analyzing test coverage with detailed reporting and recommendations
  - **Circular Import Testing** - Comprehensive circular import detection and prevention testing framework
  - **Config Field Testing Suite** - Complete test suite for configuration field management with end-to-end validation
  - **Integration Testing Framework** - Enhanced integration testing with better coverage and reliability

- **Documentation Expansion** - Comprehensive documentation improvements and new guides
  - **Pytest Best Practices Guide** - New guide for pytest best practices and troubleshooting
  - **Test Failure Categories Guide** - Detailed guide for test failure categories and prevention strategies
  - **Validation Tutorial Updates** - Updated validation tutorials with quick start guides and API references
  - **Coverage Analysis Documentation** - Comprehensive documentation for test coverage analysis and improvement

- **Configuration Management Tools** - New tools for configuration field management and optimization
  - **Circular Reference Tracker** - Enhanced circular reference detection and tracking for configuration fields
  - **Type-Aware Config Serializer** - New type-aware serialization system for better configuration handling
  - **Performance Optimizer** - Configuration performance optimization tools with caching and efficiency improvements
  - **Tier Registry System** - Enhanced tier registry for better configuration field classification and management

### Fixed
- **Test Infrastructure Stability** - Major improvements to test execution reliability and consistency
  - **Test Error Resolution** - Systematic resolution of test errors across validation and configuration modules
  - **Coverage Analysis Accuracy** - Fixed coverage analysis accuracy with better function detection and reporting
  - **Test Execution Consistency** - Improved test execution consistency across different environments and configurations
  - **Mock Infrastructure Reliability** - Enhanced mock infrastructure with better error handling and resource management

- **Configuration System Reliability** - Enhanced configuration system stability and performance
  - **Config Field Management Bugs** - Fixed critical bugs in configuration field management and validation
  - **Circular Reference Issues** - Resolved circular reference issues in configuration processing
  - **Type Serialization Problems** - Fixed type serialization issues with better type handling and validation
  - **Performance Bottlenecks** - Eliminated performance bottlenecks in configuration processing and validation

- **Code Quality Improvements** - Major code cleanup and organization enhancements
  - **Redundant Code Elimination** - Systematic removal of redundant code across testing and configuration modules
  - **Import Path Cleanup** - Fixed import path issues and improved module organization
  - **Legacy File Removal** - Removed legacy files and unused components for better maintainability
  - **Documentation Consistency** - Improved documentation consistency and accuracy across all modules

### Technical Details
- **Testing Architecture** - Comprehensive testing framework with advanced coverage analysis and reporting capabilities
- **Configuration Management** - Redesigned configuration field management system with unified manager and step catalog integration
- **Validation Framework** - Enhanced validation system with simplified integration and improved user experience
- **Code Organization** - Improved code organization with better separation of concerns and reduced redundancy
- **Performance Optimization** - Significant performance improvements in testing, configuration, and validation operations

### Quality Assurance
- **Test Reliability** - Improved test reliability with comprehensive error resolution and better infrastructure
- **Configuration Accuracy** - Enhanced configuration accuracy with better validation and type handling
- **Documentation Quality** - Improved documentation quality with updated guides and comprehensive API references
- **Code Standards** - Enhanced code quality standards with systematic cleanup and organization improvements

### Performance Improvements
- **Test Execution Speed** - Optimized test execution with improved performance and resource utilization
- **Configuration Processing** - Enhanced configuration processing performance with better caching and optimization
- **Validation Performance** - Improved validation performance with streamlined algorithms and better resource management
- **Memory Usage** - Reduced memory usage through redundant code elimination and better resource management

## [1.3.7] - 2025-10-04

### Added
- **New ML Pipeline Steps** - Expanded step library with advanced ML capabilities
  - **Missing Value Imputation Step** - Complete step for handling missing data with multiple imputation strategies
  - **Model Wiki Generator Step** - Automated model documentation and wiki generation capabilities
  - **Model Metrics Computation Step** - Comprehensive model performance metrics calculation and reporting
  - **XGBoost Model Inference Step** - Dedicated inference step for XGBoost models with optimized performance

### Enhanced
- **Validation System Improvements** - Major enhancements to validation framework reliability
  - **Builder Test CLI Enhancements** - Improved command-line interface for builder testing with better error handling
  - **Alignment Tester Refactoring** - Comprehensive refactoring of alignment validation system for better accuracy
  - **Validation Builder Test Optimization** - Reduced code redundancy and improved test execution performance
  - **Step Catalog Integration** - Enhanced step catalog system with deduplication and improved step discovery

- **Testing Infrastructure Modernization** - Comprehensive testing system improvements
  - **Pytest Best Practices Implementation** - Adopted pytest best practices across all test modules
  - **Test Structure Reorganization** - Improved test organization with better separation of concerns
  - **Dynamic Step Builder Testing** - Enhanced dynamic testing capabilities for step builders
  - **CLI Testing Framework** - Comprehensive testing framework for command-line interface components

- **Code Quality and Organization** - Major code cleanup and optimization efforts
  - **Workspace-Aware System Simplification** - Removed over-engineered workspace-aware components in favor of step catalog
  - **Validation System Refactoring** - Streamlined validation components with reduced complexity
  - **Code Redundancy Elimination** - Systematic removal of redundant code across validation and testing modules
  - **Rule-Based Validation** - Enhanced rule-based validation system with improved pattern matching

### Fixed
- **CLI System Stability** - Comprehensive fixes to command-line interface reliability
  - **Builder CLI Error Resolution** - Fixed critical errors in builder CLI execution and validation
  - **Alignment CLI Improvements** - Resolved alignment CLI issues with better error handling and reporting
  - **Validation CLI Enhancements** - Fixed validation CLI bugs and improved user experience
  - **Registry CLI Integration** - Enhanced registry CLI with better component discovery and management

- **Testing Framework Reliability** - Major improvements to test execution stability
  - **Test Error Resolution** - Systematic resolution of test errors across validation and builder test modules
  - **Step Catalog Test Fixes** - Fixed step catalog testing issues with improved integration
  - **Alignment Test Improvements** - Enhanced alignment testing with better rule matching and validation
  - **Builder Test Optimization** - Improved builder test reliability with better mock infrastructure

- **Validation System Accuracy** - Enhanced validation accuracy and reduced false positives
  - **Rule Matching Improvements** - Better rule matching algorithms for more accurate validation results
  - **Low Score System Fixes** - Fixed issues with low score detection and reporting in validation systems
  - **Alignment Validation Enhancements** - Improved alignment validation with better pattern recognition
  - **Factory Method Cleanup** - Cleaned up factory methods in alignment tester for better reliability

### Technical Details
- **Step Library Expansion** - Added 4 new production-ready ML pipeline steps with complete implementations
- **Validation Architecture** - Refactored validation system with rule-based approach and step catalog integration
- **Testing Framework** - Modernized testing infrastructure with pytest best practices and dynamic testing
- **CLI Enhancement** - Comprehensive CLI improvements with better error handling and user experience
- **Code Optimization** - Significant code reduction through redundancy elimination and architectural simplification
- **Dependency Updates** - Lowered networkx requirement from >=3.5.0 to >=2.8.0 for better package compatibility

### Quality Assurance
- **Test Reliability** - Improved test reliability with systematic error resolution and better infrastructure
- **Validation Accuracy** - Enhanced validation accuracy with improved rule matching and pattern recognition
- **Code Quality** - Better code quality through systematic cleanup and architectural improvements
- **CLI Stability** - Enhanced CLI stability with comprehensive error handling and improved user experience

### Performance Improvements
- **Test Execution Speed** - Optimized test execution with reduced redundancy and better resource management
- **Validation Performance** - Improved validation performance with streamlined algorithms and better caching
- **Step Catalog Operations** - Enhanced step catalog performance with deduplication and optimized discovery
- **CLI Responsiveness** - Improved CLI responsiveness with optimized execution paths and better error handling

## [1.3.6] - 2025-09-29

### Added
- **Stratified Sampling Step** - New step for stratified data sampling
  - **StratifiedSampling Step Builder** - Complete step builder for stratified sampling operations
  - **Stratified Sampling Script** - Processing script for stratified data sampling
  - **Stratified Sampling Contract** - Contract definition for stratified sampling step
  - **Step Specification** - Complete step specification for stratified sampling functionality

### Enhanced
- **Step Catalog System Improvements** - Major enhancements to step catalog architecture
  - **Universal Step Builder Tester** - Enhanced universal tester with improved step catalog integration
  - **Builder Test Variants** - Improved builder test variants for processing, createmodel, and training steps
  - **Step Catalog Mapping** - Enhanced step catalog mapping and discovery capabilities
  - **Unified Alignment Tester** - Improved alignment testing with better error handling and validation

- **Testing Infrastructure Modernization** - Comprehensive testing system improvements
  - **Test Execution Refinement** - Refined test execution with better signature handling and error recovery
  - **Processing Test Variants** - Enhanced processing test variants with improved coverage
  - **Builder Test Framework** - Simplified and improved builder test framework
  - **Test Error Resolution** - Systematic resolution of test errors and improved reliability

- **System Architecture Simplification** - Major architectural improvements
  - **Workspace-Aware System Cleanup** - Removed over-engineered workspace-aware system in favor of step catalog
  - **Workspace Module Redesign** - Completely removed cursus.workspaces and redesigned a simplified workspace module using step catalog system
  - **Code Redundancy Reduction** - Massive reduction in redundant code and files
  - **Registry System Consolidation** - Moved registry components to unified location under cursus/registry
  - **Step Builder Simplification** - Simplified createmodel and transform step builders

### Fixed
- **Test Infrastructure Stability** - Comprehensive fixes to testing framework
  - **Test Workspace Issues** - Fixed and cleaned up test workspace configuration
  - **Alignment Test Errors** - Resolved errors in alignment testing framework
  - **Processing Test Fixes** - Fixed processing test execution and validation
  - **Builder Test Reliability** - Improved reliability of builder test execution

- **Code Organization Improvements** - Major cleanup and organization enhancements
  - **Redundant File Removal** - Removed redundant files and placeholder components
  - **Import Path Cleanup** - Cleaned up import paths and removed redundant imports
  - **Registry Organization** - Consolidated registry system and removed duplicate components
  - **Test File Cleanup** - Removed redundant validation test files and improved organization

### Technical Details
- **Step Catalog Integration** - Enhanced step catalog system with better mapping and discovery
- **Testing Framework** - Improved testing framework with better error handling and variant support
- **Code Reduction** - Significant code reduction through removal of over-engineered components
- **Architecture Simplification** - Simplified architecture with clearer component boundaries
- **Registry Consolidation** - Unified registry system with better organization and functionality

### Quality Assurance
- **Test Reliability** - Improved test reliability with better error handling and recovery
- **Code Quality** - Enhanced code quality through redundancy removal and simplification
- **System Stability** - Improved system stability with simplified architecture
- **Integration Testing** - Better integration testing with enhanced step catalog system

### Performance Improvements
- **Test Execution Speed** - Improved test execution performance with optimized framework
- **System Responsiveness** - Enhanced system responsiveness through code simplification
- **Memory Usage** - Reduced memory footprint through redundant code removal
- **Build Performance** - Improved build performance with cleaner architecture

## [1.3.5] - 2025-09-27

### Fixed
- **Step Catalog System Stability** - Critical fixes to step catalog functionality and builder recovery
  - **Builder Recovery Issue** - Fixed builder recovery mechanism in step catalog system
  - **Step Builder Map Issues** - Resolved issues with step builder mapping and discovery
  - **Dynamic Template Integration** - Fixed critical bug in dynamic template where step_catalog wasn't passed to parent class
  - **Job Type Variants** - Fixed errors in job type variant handling and mapping

- **Pipeline Template System** - Enhanced pipeline template base to work with deprecated builder registry
  - **Builder Registry Deprecation** - Updated pipeline template base to reflect deprecated builder registry system
  - **Template Integration** - Improved integration between pipeline templates and step catalog system
  - **Logging Cleanup** - Removed extensive debug logging for cleaner production output

### Enhanced
- **Step Catalog Mapping** - Improved step catalog mapping system with better error handling
  - **Mapping Reliability** - Enhanced mapping reliability and error recovery mechanisms
  - **Builder Discovery** - Improved builder discovery with better fallback mechanisms
  - **Job Type Handling** - Enhanced job type variant handling in step catalog mapping

### Technical Details
- **Critical Bug Fix** - Fixed missing step_catalog parameter in dynamic template parent class initialization
- **Builder Recovery** - Enhanced builder recovery mechanism with better error handling and fallback options
- **Mapping System** - Improved step catalog mapping system with enhanced job type variant support
- **Template System** - Updated pipeline template system to work seamlessly with step catalog architecture

### Quality Assurance
- **System Stability** - Improved overall system stability with critical bug fixes
- **Error Handling** - Enhanced error handling and recovery mechanisms throughout step catalog system
- **Integration Testing** - Better integration between step catalog and pipeline template systems

## [1.3.4] - 2025-09-27

### Enhanced
- **Step Catalog System Revolution** - Major architectural transformation with unified step catalog system
  - **Step Catalog Expansion** - Comprehensive expansion of step catalog with mapping enhancement and registry absorption
  - **Builder Registry Deprecation** - Complete deprecation and removal of step builder registry in favor of unified step catalog
  - **Workspace-Aware Integration** - Full integration of step catalog into workspace-aware system architecture
  - **Redundancy Reduction** - Massive code redundancy reduction through step catalog consolidation

- **Documentation System Overhaul** - Comprehensive documentation updates reflecting new architecture
  - **Step Catalog Integration Guide** - New comprehensive guide for step catalog integration replacing builder registry documentation
  - **Developer Guide Updates** - Updated all developer guides to reflect removal of step builder registry
  - **Tutorial Modernization** - Modernized tutorials and API references to use step catalog system
  - **Documentation Cleanup** - Removed 750+ lines of outdated step builder registry documentation

- **Pipeline System Improvements** - Enhanced pipeline infrastructure with better integration
  - **Base Pipeline Enhancement** - Major improvements to base pipeline with 556+ lines of new functionality
  - **Workspace Core Integration** - Enhanced workspace core assembler and compiler with step catalog support
  - **Pipeline Catalog Updates** - Updated pipeline catalog to work seamlessly with new step catalog architecture

- **Validation System Enhancements** - Improved validation capabilities with step catalog integration
  - **Interface Validation Updates** - Enhanced interface validation with better error handling and reference removal
  - **Builder Validation Improvements** - Updated builder validation to work with new step catalog system
  - **Test Infrastructure Modernization** - Modernized test infrastructure to support step catalog architecture

### Added
- **Unified Step Catalog Architecture** - Revolutionary new architecture for component management
  - **Step Catalog Core** - Complete step catalog system with mapping and discovery capabilities
  - **Component Mapping System** - Advanced mapping system for step catalog components
  - **Registry Integration** - Seamless integration of registry functionality into step catalog
  - **Workspace Catalog Support** - Full workspace-aware step catalog implementation

- **Enhanced Project Structure** - Improved project organization and Docker integration
  - **Docker Container Reorganization** - Moved Docker projects to dedicated `dockers/` directory
  - **Project Structure Cleanup** - Better organization of PyTorch BSM extension and XGBoost projects
  - **Container Integration** - Enhanced Docker container integration with step catalog system

- **Advanced Testing Framework** - Expanded testing capabilities for new architecture
  - **Step Catalog Tests** - Comprehensive test suite for step catalog functionality
  - **Mapping Tests** - Detailed tests for step catalog mapping system
  - **Integration Tests** - Enhanced integration tests for workspace-aware step catalog

### Fixed
- **Architecture Migration Issues** - Resolved issues during migration to step catalog system
  - **Registry Removal** - Clean removal of deprecated step builder registry system
  - **Import Path Updates** - Updated all import paths to use step catalog instead of builder registry
  - **Test Infrastructure Fixes** - Fixed test infrastructure to work with new step catalog architecture
  - **XGBoost Model Evaluation** - Fixed XGBoost model evaluation step builder issues

- **Code Quality Improvements** - Major code quality enhancements through redundancy reduction
  - **Redundant Code Elimination** - Removed massive amounts of redundant code through step catalog consolidation
  - **Documentation Consistency** - Ensured consistency across all documentation after registry removal
  - **Test Reliability** - Improved test reliability with better integration testing

- **Project Organization Fixes** - Enhanced project structure and organization
  - **Folder Structure** - Improved folder structure with Docker projects properly organized
  - **Path Resolution** - Fixed path resolution issues in configuration and processing steps
  - **Test Configuration** - Fixed test configuration issues across multiple test modules

### Technical Details
- **Step Catalog Architecture** - Complete unified step catalog system replacing fragmented builder registry approach
- **Code Reduction Metrics** - Achieved significant code reduction through consolidation and redundancy elimination
- **Migration Strategy** - Seamless migration from builder registry to step catalog with full backward compatibility
- **Performance Optimization** - Improved performance through unified architecture and reduced complexity
- **Architectural Simplification** - Simplified architecture with clear component boundaries and responsibilities

### Quality Assurance
- **Comprehensive Testing** - Enhanced test coverage for step catalog system and workspace integration
- **Migration Validation** - Thorough validation of migration from builder registry to step catalog
- **Integration Testing** - Comprehensive integration testing across all system components
- **Documentation Quality** - Improved documentation quality with updated guides and references

### Performance Improvements
- **System Responsiveness** - Enhanced system responsiveness with streamlined step catalog architecture
- **Memory Usage** - Reduced memory footprint through elimination of redundant registry components
- **Code Execution Speed** - Improved execution speed through unified step catalog system
- **Resource Utilization** - Better resource utilization through unified step catalog system

## [1.3.3] - 2025-09-25

### Enhanced
- **Path Resolution System** - Major improvements to deployment-agnostic path resolution
  - **Hybrid Path Resolution Strategy** - New hybrid approach combining direct matching with intelligent fallback mechanisms
  - **Development Context Awareness** - Enhanced path resolution that adapts to development vs package deployment contexts
  - **Fallback Logic Improvements** - Improved fallback logic in path resolution for better reliability across environments
  - **Portable Configuration System** - Redesigned configuration portability system with package-aware path handling

- **Project Structure Improvements** - Enhanced project organization and Docker integration
  - **Docker Container Updates** - Updated Docker containers with improved folder structure and organization
  - **Project Folder Restructuring** - Reorganized project folders for better maintainability and clarity
  - **PyTorch BSM Extension** - Complete PyTorch BSM extension with Lightning models, processing utilities, and training scripts
  - **XGBoost Project Enhancements** - Enhanced XGBoost projects with improved hyperparameter handling and model evaluation

- **Step Builder System** - Comprehensive updates to step builders and configuration management
  - **Step Builder Modernization** - Updated all step builders with improved path resolution and configuration handling
  - **Configuration Portability** - Enhanced configuration classes with portable path resolution capabilities
  - **Hyperparameter Loading Fixes** - Fixed hyperparameter loading issues in XGBoost training with local folder support
  - **Model Evaluation Updates** - Updated model evaluation step scripts with improved functionality

### Added
- **CLI Pipeline Compilation Tools** - New command-line interface tools for pipeline compilation and management
  - **Pipeline Initiation CLI** - New CLI tools for pipeline project initiation and setup
  - **Compilation Tools** - Enhanced compilation tools with better error handling and user experience
  - **Project Template Support** - Support for different project templates and initialization patterns

- **Comprehensive Processing Libraries** - Extensive processing utilities for different ML frameworks
  - **PyTorch Processing Suite** - Complete processing library with BERT tokenization, binning, and data loading utilities
  - **XGBoost Processing Tools** - Comprehensive processing tools for XGBoost workflows including categorical processing and risk table handling
  - **Multimodal Model Support** - Added support for multimodal models including cross-attention, gate fusion, and mixture of experts
  - **Lightning Model Collection** - Complete collection of PyTorch Lightning models for various use cases

- **Enhanced Testing Infrastructure** - Expanded testing capabilities for hybrid path resolution
  - **Hybrid Path Resolution Tests** - Comprehensive test suite for hybrid path resolution strategies
  - **Integration Testing** - Enhanced integration tests for portable path resolution across different deployment contexts
  - **Configuration Validation Tests** - New tests for modernized path resolution and configuration validation

### Fixed
- **Path Resolution Issues** - Critical fixes to path resolution across different deployment contexts
  - **Execution Location Path Computation** - Fixed path computation to use execution location instead of definition location
  - **Hyperparameter Loading** - Fixed XGBoost training to properly load hyperparameter.json from local folders
  - **Portable Source Directory** - Resolved issues with portable source directory handling and removed redundant backup scenarios
  - **Development vs Package Context** - Fixed path resolution to work correctly in both development and packaged environments

- **Configuration Management Fixes** - Enhanced configuration handling and validation
  - **Config Base Portability** - Fixed configuration base classes to support portable path creation
  - **Step Builder Configuration** - Updated step builder configurations with improved path handling and validation
  - **Contract and Script Alignment** - Fixed alignment issues between contracts and training scripts

- **Project Structure Cleanup** - Comprehensive cleanup and organization improvements
  - **Redundant File Removal** - Removed redundant scenario backup files and cleaned up project structure
  - **Import Path Corrections** - Fixed import paths across all project modules for better consistency
  - **Documentation Updates** - Updated internal documentation and design documents

### Technical Details
- **Hybrid Path Resolution** - New hybrid strategy combining direct path matching with intelligent fallback mechanisms
- **Deployment Context Awareness** - System now adapts path resolution based on development vs deployment context
- **Configuration Portability** - Enhanced configuration system with package-aware and development-agnostic path handling
- **Project Template System** - Improved project template system with better organization and Docker integration
- **Processing Pipeline Architecture** - Comprehensive processing pipeline architecture supporting multiple ML frameworks

### Quality Assurance
- **Cross-Environment Testing** - Enhanced testing across different deployment environments and contexts
- **Path Resolution Validation** - Comprehensive validation of path resolution in various scenarios
- **Configuration Consistency** - Improved consistency in configuration handling across different project types
- **Integration Reliability** - Enhanced reliability of integration between different system components

### Performance Improvements
- **Path Resolution Performance** - Optimized path resolution algorithms for better performance across different contexts
- **Configuration Processing** - Improved configuration processing performance with better caching and validation
- **Docker Container Efficiency** - Enhanced Docker container efficiency with optimized folder structures and processing
- **Build System Optimization** - Optimized build system performance with better dependency management

## [1.3.2] - 2025-09-21

### Enhanced
- **Package Portability System** - Major improvements to package portability and path handling
  - **Relative Path Computation** - Changed relative path computation to use execution location instead of definition location
  - **Builder Class Portability** - Enhanced builder classes with relative path support for better portability
  - **Config Base Portability** - Updated config base to allow relative path creation for improved portability
  - **Step Catalog Dual Search Space** - Enhanced step catalog with improved search space and dual search capabilities

- **Step Catalog System Improvements** - Significant enhancements to step catalog architecture
  - **Pipeline DAG Resolver Redesign** - Complete redesign of pipeline DAG resolver using step catalog
  - **Contract Discovery Refactoring** - Refactored contract discovery with relative import support
  - **Builder Discovery System** - Created builder discovery class for step catalog integration
  - **Specification Discovery** - Added spec discovery and relative import module support

- **Configuration Management Enhancements** - Major improvements to configuration handling
  - **Config Field Management Refactor** - Complete refactor of config field management system
  - **Payload Config Simplification** - Simplified payload configuration for better usability
  - **Contract Name Standardization** - Standardized contract names and removed source directory validator
  - **Config Error Fixes** - Fixed various configuration errors and improved validation

- **PyTorch Integration Improvements** - Enhanced PyTorch support and Docker integration
  - **PyTorch Training Builder Fixes** - Fixed PyTorch training builder, script, contract, and step specifications
  - **PyTorch Docker Updates** - Updated PyTorch Docker containers with improved functionality
  - **Batch Transform Builder Fixes** - Fixed builder for batch transform operations

### Added
- **Execution Document Generation** - New execution document generation capabilities
  - **Demo Pipeline Integration** - Updated execution documents and demo pipeline examples
  - **Config Samples** - Added comprehensive configuration samples and examples
  - **Test Execution Location** - New notebook for testing execution vs definition location

- **Enhanced Documentation** - Comprehensive documentation and tutorial updates
  - **Tutorials and Guides Updates** - Updated tutorials and guides with latest features
  - **Notebook Root Removal** - Removed notebook root references for better portability
  - **Demo Configuration** - Enhanced demo configuration notebooks and examples

### Fixed
- **Import System Improvements** - Major fixes to import and module loading
  - **Relative Import Support** - Enhanced relative import support throughout the system
  - **Importlib Issues** - Fixed importlib issues with step catalog system when system path doesn't include cursus
  - **Module Discovery** - Improved module discovery and loading mechanisms
  - **System Path Independence** - Reduced dependency on system path configuration

- **Test Infrastructure Fixes** - Comprehensive test system improvements
  - **Test Bug Fixes** - Fixed multiple test bugs and improved test reliability
  - **Test Integration** - Enhanced test integration and execution
  - **Load Config Error Fixes** - Fixed load_config errors and improved error handling

- **Portability and Path Issues** - Systematic fixes to path handling and portability
  - **Relative Path Creation** - Fixed relative path creation across different components
  - **Package Module Definition** - Improved package module definition using relative imports
  - **Path Computation Logic** - Enhanced path computation logic for better cross-platform support

### Technical Details
- **Step Catalog Architecture** - Enhanced step catalog system with dual search space and improved discovery
- **Portability Framework** - Comprehensive portability improvements across configuration, builders, and path handling
- **Import System** - Modernized import system with relative imports and reduced system path dependencies
- **Configuration Management** - Streamlined configuration management with simplified payload handling
- **Docker Integration** - Updated Docker containers with improved PyTorch support and functionality

### Quality Assurance
- **Test System Reliability** - Improved test system reliability with comprehensive bug fixes
- **Configuration Validation** - Enhanced configuration validation and error handling
- **Import Stability** - Improved import stability with better module discovery and loading
- **Cross-Platform Support** - Enhanced cross-platform support with improved path handling

### Performance Improvements
- **Step Catalog Performance** - Optimized step catalog operations with enhanced search capabilities
- **Configuration Processing** - Improved configuration processing performance with streamlined management
- **Import Performance** - Enhanced import performance with optimized relative import system
- **Path Resolution Speed** - Faster path resolution with improved computation algorithms

## [1.3.1] - 2025-09-18

### Enhanced
- **Pipeline Execution System** - Major improvements to pipeline execution and portability
  - **Temporary Directory Management** - Enhanced temporary directory parameter passing from DAG compiler to assembler
  - **Portability Enhancements** - Improved portability with better parameter passing throughout the pipeline execution chain
  - **Pipeline Template Base** - Enhanced pipeline template base with improved parameter handling and execution context
  - **Dynamic Template System** - Streamlined dynamic template system with better integration and reduced complexity

- **Step Builder Architecture** - Significant improvements to step builder system
  - **Floating S3 Support** - Refactored XGBoost training and dummy training steps to support floating S3 configurations
  - **Risk Table Mapping Refactoring** - Major refactoring of risk table mapping to remove hyperparameter saving and improve Join compatibility
  - **Model Calibration Cleanup** - Removed deprecated model calibration step builder and cleaned up unused code
  - **Training Step Improvements** - Enhanced PyTorch and XGBoost training steps with better parameter handling

- **Pipeline Catalog System** - Major updates to pipeline catalog infrastructure
  - **Core Pipeline Enhancements** - Significant improvements to base pipeline functionality with 117+ lines of new features
  - **Pipeline Catalog Core** - Added new pipeline catalog core infrastructure for better pipeline management
  - **Test Infrastructure** - Recreated comprehensive pytest framework for pipeline catalog testing
  - **Catalog Organization** - Improved pipeline catalog organization and structure

### Added
- **Join Operation Support** - Enhanced support for Join operations in pipeline execution
  - **String Handling** - Fixed bug to support Join operations with string parameters in log.debug statements
  - **F-String Compatibility** - Removed f-string usage to ensure compatibility with Join operations
  - **Pipeline Assembler Integration** - Enhanced pipeline assembler with better Join operation support

- **Comprehensive Test Suite** - Expanded testing infrastructure for new features
  - **Pipeline Execution Tests** - New comprehensive tests for pipeline execution with temporary directory integration
  - **Template Base Tests** - Enhanced tests for pipeline template base functionality
  - **Builder Base Tests** - New tests for builder base functionality and parameter handling
  - **DAG Compiler Tests** - Enhanced DAG compiler tests with better coverage

- **Documentation Updates** - Comprehensive documentation improvements
  - **Implementation Plans** - Updated hyperparameters source directory refactor implementation plans
  - **Progress Tracking** - Enhanced progress tracking in implementation plans
  - **Tutorial Updates** - Updated documentation, tutorials, and prompts for better user experience

### Fixed
- **Pipeline Execution Issues** - Critical fixes to pipeline execution system
  - **Join Operation Compatibility** - Fixed compatibility issues with Join operations in various step builders
  - **String Parameter Handling** - Improved string parameter handling in log.debug and other operations
  - **Temporary Directory Integration** - Fixed temporary directory parameter passing throughout the execution chain

- **Step Builder Fixes** - Comprehensive fixes to step builder implementations
  - **Risk Table Mapping** - Fixed risk table mapping step to remove unnecessary hyperparameter saving
  - **Training Step Alignment** - Fixed alignment issues in PyTorch and XGBoost training steps
  - **Model Step Configuration** - Improved model step configuration handling and parameter validation

- **Code Quality Improvements** - Major code quality enhancements
  - **Code Cleanup** - Removed deprecated and unused code, including model calibration step builder
  - **Import Optimization** - Improved import statements and module organization
  - **Parameter Validation** - Enhanced parameter validation and error handling throughout the system

### Technical Details
- **Pipeline Architecture** - Enhanced pipeline execution architecture with better portability and parameter passing
- **Step Builder System** - Improved step builder system with floating S3 support and better Join compatibility
- **Pipeline Catalog** - New pipeline catalog core infrastructure with comprehensive testing framework
- **Execution Context** - Enhanced execution context management with temporary directory support

### Quality Assurance
- **Comprehensive Testing** - Extensive testing of new pipeline execution features and improvements
- **Integration Validation** - Thorough validation of integration between different system components
- **Performance Testing** - Performance validation of enhanced pipeline execution system
- **Compatibility Testing** - Comprehensive compatibility testing for Join operations and string handling

### Performance Improvements
- **Execution Speed** - Improved pipeline execution speed with optimized parameter passing
- **Memory Management** - Better memory management in pipeline execution with enhanced cleanup
- **Resource Utilization** - Optimized resource utilization in step builders and pipeline assembly
- **Code Efficiency** - Improved code efficiency through cleanup and refactoring efforts

## [1.3.0] - 2025-09-18

### Added
- **Unified Step Catalog System** - Revolutionary step catalog architecture with 95%+ code reduction
  - **Step Catalog Core** - Complete step catalog system with models, discovery, and integration capabilities
  - **Adapter Architecture** - Modular adapter system for config resolution, contract discovery, and file resolution
  - **Workspace Integration** - Full integration of step catalog with workspace-aware system architecture
  - **Legacy Migration** - Seamless migration from legacy systems with backward compatibility

- **Massive Code Consolidation** - Achieved 95%+ reduction in codebase complexity
  - **File Consolidation** - Consolidated 40+ scattered files into unified step catalog architecture
  - **Adapter Modularization** - Split monolithic adapters into focused, single-responsibility modules
  - **Dependency Graph System** - New dependency graph system for better component relationships
  - **Inventory Management** - Enhanced inventory system for workspace component tracking

- **Enhanced Pipeline Integration** - Step catalog fully integrated into pipeline systems
  - **Pipeline DAG Resolver** - Updated pipeline DAG resolver to use step catalog for component discovery
  - **Runtime Testing Integration** - Step catalog integrated into runtime testing and validation systems
  - **Pipeline Catalog Utils** - Enhanced pipeline catalog utilities with step catalog support
  - **Registry System Updates** - Updated registry system to leverage step catalog architecture

### Enhanced
- **Workspace-Aware Architecture** - Major improvements to workspace system with step catalog integration
  - **Discovery System Refactoring** - Completely refactored workspace discovery system using step catalog
  - **File Resolution Enhancement** - Enhanced file resolution with step catalog adapters
  - **Alignment Testing** - Improved workspace alignment testing with step catalog integration
  - **Module Loading** - Enhanced workspace module loading with better dependency management

- **Validation System Improvements** - Comprehensive validation system updates
  - **Contract Discovery** - Streamlined contract discovery using step catalog adapters
  - **Runtime Validation** - Enhanced runtime validation with step catalog integration
  - **Alignment Validation** - Improved alignment validation with unified step catalog approach
  - **Builder Validation** - Enhanced builder validation using step catalog registry

- **Configuration Management** - Major improvements to configuration handling
  - **Config Class Detection** - Enhanced config class detection with step catalog integration
  - **Config Resolution** - Improved configuration resolution using step catalog adapters
  - **Hyperparameter Management** - Updated hyperparameter handling with step catalog support
  - **Three-Tier Config** - Enhanced three-tier configuration system with step catalog integration

### Fixed
- **Legacy System Migration** - Comprehensive fixes during migration to step catalog
  - **Import Path Updates** - Updated all import paths to use step catalog architecture
  - **Pydantic Compatibility** - Fixed Pydantic deprecated features and updated to latest version
  - **Alignment Errors** - Resolved alignment errors during legacy system migration
  - **Test Integration** - Fixed test integration issues with step catalog system

- **Code Quality Improvements** - Major code quality enhancements
  - **Redundancy Elimination** - Eliminated massive code redundancy through step catalog consolidation
  - **Module Organization** - Improved module organization with clear separation of concerns
  - **Dependency Management** - Enhanced dependency management with step catalog architecture
  - **Error Handling** - Improved error handling throughout step catalog system

- **System Integration Fixes** - Comprehensive integration fixes
  - **Registry Integration** - Fixed registry system integration with step catalog
  - **Workspace Integration** - Resolved workspace integration issues with step catalog
  - **Pipeline Integration** - Fixed pipeline system integration with step catalog
  - **Validation Integration** - Resolved validation system integration with step catalog

### Technical Details
- **Step Catalog Architecture** - Complete unified step catalog system with modular adapter architecture
- **Code Reduction Metrics** - Achieved 95%+ reduction in codebase complexity through consolidation
- **Migration Strategy** - Seamless migration from legacy systems with full backward compatibility
- **Performance Optimization** - Significant performance improvements through code consolidation
- **Architectural Simplification** - Simplified architecture with clear component boundaries

### Quality Assurance
- **Comprehensive Testing** - Enhanced test coverage for step catalog system and integrations
- **Migration Validation** - Thorough validation of migration from legacy systems
- **Integration Testing** - Comprehensive integration testing across all system components
- **Performance Testing** - Performance validation of consolidated step catalog system

### Performance Improvements
- **Code Execution Speed** - Improved execution speed through code consolidation and optimization
- **Memory Usage** - Reduced memory footprint through elimination of redundant code
- **System Responsiveness** - Enhanced system responsiveness with streamlined architecture
- **Resource Utilization** - Better resource utilization through unified step catalog system

### Breaking Changes
- **Import Path Changes** - Some import paths updated to reflect step catalog architecture
- **API Consolidation** - Legacy APIs consolidated into unified step catalog interfaces
- **Configuration Updates** - Some configuration structures updated for step catalog compatibility

### Migration Guide
- **Legacy System Migration** - Comprehensive migration guide for transitioning from legacy systems
- **Import Path Updates** - Guide for updating import paths to use step catalog
- **Configuration Migration** - Instructions for migrating configurations to step catalog format
- **Testing Updates** - Guide for updating tests to work with step catalog system

## [1.2.6] - 2025-09-16

### Added
- **Step Catalog System** - New unified step catalog architecture for better component discovery and management
  - **Step Catalog Infrastructure** - Complete step catalog system with models, discovery, and integration capabilities
  - **Config Discovery System** - Automated configuration discovery and validation for step catalog components
  - **Step Catalog Integration** - Enhanced integration between step catalog and existing pipeline components
  - **Catalog CLI Tools** - New command-line interface tools for step catalog management and operations

- **Execution Document Generator** - Standalone execution document generation system
  - **MODS Execution Document Generator** - Complete execution document generator for MODS pipelines
  - **Cradle Helper System** - Enhanced cradle data loading helper with improved configuration management
  - **Registration Helper** - New registration helper for model registration and management workflows
  - **Execution Document Utilities** - Comprehensive utilities for execution document generation and management

- **Enhanced Pipeline Catalog** - Major improvements to pipeline catalog architecture
  - **Pipeline Catalog Refactoring** - Complete refactoring of pipeline catalog with improved organization
  - **MODS Pipeline Collection** - Expanded collection of MODS pipeline templates and examples
  - **Pipeline Execution System** - New pipeline execution utilities and generator capabilities
  - **Catalog Registry System** - Enhanced catalog registry with better validation and discovery

### Enhanced
- **Core Type Safety Improvements** - Comprehensive MyPy type safety enhancements across core modules
  - **MyPy Core Optimization** - 12 iterations of MyPy improvements for better type safety and code quality
  - **Type Safety Validation** - Enhanced type checking and validation across all core components
  - **Import Path Optimization** - Improved import paths and module organization for better maintainability
  - **Error Handling Enhancement** - Better error handling and type safety in core modules

- **Pipeline Generation Independence** - Major architectural improvement separating pipeline and execution generation
  - **Independent Generation Systems** - Pipeline generation and execution document generation now operate independently
  - **Template State Management** - Enhanced template state management for better pipeline compilation
  - **Execution Document Separation** - Clean separation between pipeline compilation and execution document filling
  - **Improved Workflow Flexibility** - Better workflow flexibility with independent generation capabilities

- **Docker Infrastructure Updates** - Enhanced Docker containers with improved scripts and functionality
  - **XGBoost Container Improvements** - Updated XGBoost containers with enhanced training, calibration, and inference scripts
  - **Script Error Fixes** - Fixed errors in training, calibration, packaging, and payload scripts
  - **Container Reliability** - Improved reliability and functionality of Docker container infrastructure
  - **Script Standardization** - Standardized scripts across different container types for consistency

### Fixed
- **Pipeline Catalog Organization** - Resolved issues with pipeline catalog structure and organization
  - **Catalog Structure Cleanup** - Cleaned up pipeline catalog structure with better organization
  - **MODS Pipeline Integration** - Fixed integration issues with MODS pipeline catalog components
  - **Catalog Index Management** - Improved catalog index management and validation
  - **Migration Guide Updates** - Updated migration guides for pipeline catalog changes

- **Core Module Stability** - Enhanced stability and reliability of core modules
  - **Import Path Corrections** - Fixed import path issues across core modules and components
  - **Module Organization** - Improved module organization and dependency management
  - **Configuration Management** - Enhanced configuration management with better validation and error handling
  - **Test Infrastructure** - Improved test infrastructure with better coverage and reliability

- **Builder CLI Improvements** - Fixed issues in builder CLI and related tools
  - **CLI Error Handling** - Enhanced error handling in CLI tools and commands
  - **Builder Test Integration** - Improved integration between builder tests and CLI tools
  - **Command Reliability** - Enhanced reliability of CLI commands and operations
  - **User Experience** - Better user experience with improved CLI feedback and error messages

### Technical Details
- **Step Catalog Architecture** - Complete step catalog system with discovery, models, and integration capabilities
- **Execution Document System** - Standalone execution document generator with comprehensive helper systems
- **Type Safety Framework** - Enhanced type safety with comprehensive MyPy improvements across all modules
- **Pipeline Independence** - Architectural improvement separating pipeline and execution document generation
- **Docker Infrastructure** - Updated Docker containers with improved scripts and enhanced functionality

### Quality Assurance
- **Comprehensive Testing** - Enhanced test coverage for step catalog, execution document generation, and core improvements
- **Type Safety Validation** - Comprehensive type safety validation with MyPy across all core modules
- **Integration Testing** - Improved integration testing for step catalog and execution document systems
- **Docker Validation** - Enhanced validation of Docker container functionality and script reliability

### Performance Improvements
- **Catalog Performance** - Optimized step catalog operations with better discovery and validation algorithms
- **Generation Performance** - Improved performance of pipeline and execution document generation processes
- **Type Checking Speed** - Enhanced type checking performance with optimized MyPy configurations
- **CLI Responsiveness** - Improved responsiveness of CLI tools and catalog operations

## [1.2.5] - 2025-09-14

### Added
- **Runtime Inference Testing Infrastructure** - New comprehensive testing framework for pipeline runtime inference
  - **Inference Runtime Tester** - New inference runtime testing system with comprehensive validation capabilities
  - **Runtime Testing Integration** - Enhanced integration between runtime testing and inference validation
  - **Inference Test Framework** - Complete test framework for inference pipeline validation
  - **Runtime Code Updates** - Updated runtime code with improved testing and validation capabilities

- **Model Calibration System** - Enhanced model calibration capabilities with improved workflow support
  - **Calibration Step Integration** - Updated package dependencies for calibration step integration
  - **Calibrated Model Support** - Enhanced package to accept and process calibrated models
  - **Calibration Model Organization** - Improved calibration model folder structure and organization
  - **CSV Data Format Support** - Modified calibration data handling to support CSV format
  - **Model File Format Updates** - Enhanced output model file format handling for calibration workflows

- **Enhanced Documentation System** - Comprehensive documentation updates and improvements
  - **API Reference Updates** - Updated API reference documentation with latest features and capabilities
  - **Developer Guide Enhancements** - Enhanced developer guides with current system architecture
  - **Documentation Cleanup** - Removed outdated documentation files and improved content organization
  - **Tutorial Updates** - Updated tutorials with improved examples and current best practices

### Enhanced
- **Dependency Resolution System** - Major improvements to automatic dependency resolution
  - **Auto Dependency Resolution** - Enhanced automatic dependency resolution for package management
  - **Logical Name Alignment** - Improved logical name alignment across system components
  - **Package Dependency Management** - Enhanced package dependency management for calibration and other steps
  - **Dependency Validation** - Improved dependency validation and resolution accuracy

- **CLI System Improvements** - Enhanced command-line interface with better functionality
  - **CLI Updates** - Updated CLI system with improved commands and user experience
  - **Command Integration** - Better integration of new features into CLI interface
  - **User Experience** - Enhanced user experience with improved command structure and help

- **Testing Infrastructure** - Continued improvements to testing framework and capabilities
  - **Test Updates** - Updated test suite with improved coverage and reliability
  - **Runtime Testing** - Enhanced runtime testing capabilities with better validation
  - **Test Integration** - Improved integration between different testing components

- **Docker Infrastructure** - Enhanced Docker container support and functionality
  - **Docker Updates** - Updated Docker containers with improved functionality and reliability
  - **Container Integration** - Better integration of Docker containers with pipeline system
  - **Script Error Fixes** - Fixed errors in training and processing scripts within Docker containers

### Fixed
- **Script and Parameter Issues** - Comprehensive fixes to script and parameter handling
  - **Parameter Parsing** - Fixed parameter parsing issues in various pipeline components
  - **Training Script Errors** - Fixed errors in training scripts affecting pipeline execution
  - **Script Formatting** - Applied black formatting for consistent code style across scripts
  - **Script Reliability** - Improved script reliability and error handling

- **Documentation and Organization** - Major cleanup and organization improvements
  - **File Organization** - Removed outdated files and improved project organization
  - **Documentation Accuracy** - Updated documentation to reflect current system state
  - **Content Extraction** - Extracted and summarized important content from outdated documentation
  - **Tag Standardization** - Updated documentation and tag standards for consistency

- **Import System Improvements** - Fixed non-relative imports for better modularity
  - **Validation Module Imports** - Fixed non-relative imports in validation/alignment modules
  - **CLI Module Imports** - Fixed non-relative imports in CLI scripts for consistency
  - **Import Standardization** - Ensured consistent relative import usage across codebase

### Technical Details
- **Runtime Testing Architecture** - Comprehensive runtime testing system with inference validation capabilities
- **Calibration Workflow** - Complete calibration workflow with CSV data support and improved model handling
- **Dependency Management** - Enhanced automatic dependency resolution with logical name alignment
- **Documentation System** - Improved documentation system with better organization and current content
- **CLI Integration** - Full integration of new features into command-line interface

### Quality Assurance
- **Runtime Validation** - Comprehensive runtime validation with inference testing capabilities
- **Calibration Testing** - Enhanced testing for calibration workflows and model handling
- **Documentation Quality** - Improved documentation quality with updated content and organization
- **Code Formatting** - Consistent code formatting with black formatter across all scripts

### Performance Improvements
- **Runtime Testing Performance** - Optimized runtime testing execution with improved efficiency
- **Dependency Resolution Speed** - Enhanced dependency resolution performance with better algorithms
- **Documentation Access** - Improved documentation organization for faster access to information
- **CLI Responsiveness** - Enhanced CLI responsiveness with optimized command execution

## [1.2.4] - 2025-09-10

### Enhanced
- **Testing Infrastructure Modernization** - Comprehensive migration from unittest to pytest framework
  - **Pytest Migration** - Converted all test modules from unittest to pytest across validation, workspace, and core components
  - **Test Framework Standardization** - Standardized testing patterns and improved test organization
  - **Enhanced Test Coverage** - Improved test coverage analysis and reporting capabilities
  - **Test Execution Reliability** - Fixed pytest import errors and improved test isolation

- **Documentation System Improvements** - Major enhancements to documentation infrastructure
  - **Sphinx Documentation** - Automated documentation generation using Sphinx with API references
  - **API Reference Documentation** - Comprehensive API reference documentation with improved structure
  - **Workspace Documentation** - Enhanced documentation for workspace-aware system architecture
  - **Developer Guide Updates** - Updated developer guides with current system architecture and best practices

- **Step Catalog System Development** - New unified step catalog architecture
  - **Unified Step Catalog** - Design and implementation of unified step catalog system
  - **Step Catalog Integration** - Enhanced integration between step catalog and existing pipeline components
  - **Catalog System Design** - Comprehensive design documentation for step catalog architecture

- **Validation System Enhancements** - Continued improvements to validation framework
  - **Validation Runtime Testing** - Enhanced runtime validation testing with improved reliability
  - **Contract Discovery** - Improved contract discovery and path retrieval mechanisms
  - **Pipeline Testing Specifications** - Enhanced PipelineTestingSpecBuilder to support DAG-to-specification tracking
  - **Validation Alignment** - Continued refinement of validation alignment algorithms

### Added
- **Docker Integration** - New Docker containers for different ML frameworks
  - **PyTorch BSM Extension** - Docker container for PyTorch-based BSM models with training and inference
  - **XGBoost A-to-Z** - Complete XGBoost pipeline Docker container with training, evaluation, and inference
  - **XGBoost PDA** - Specialized XGBoost container for PDA (Predictive Data Analytics) workflows

- **Enhanced Testing Infrastructure** - Expanded testing capabilities
  - **Runtime Script Tester** - Improved runtime script testing with better error handling
  - **Validation System Analysis** - New analysis tools for validation system performance and accuracy
  - **Test Coverage Tools** - Enhanced test coverage analysis and reporting tools

- **Documentation Enhancements** - Comprehensive documentation improvements
  - **CLI Documentation** - Complete command-line interface documentation
  - **Registry System Documentation** - Detailed documentation for registry system architecture
  - **DAG Documentation** - Enhanced documentation for DAG compilation and execution

### Fixed
- **Import System Improvements** - Comprehensive fixes to import-related issues
  - **Relative Import Migration** - Continued migration to relative imports for better modularity
  - **Import Error Resolution** - Fixed various import errors across test and core modules
  - **Test Import Stability** - Improved stability of test imports and execution

- **Test Infrastructure Fixes** - Major improvements to test execution reliability
  - **Pytest Configuration** - Fixed pytest configuration issues and import errors
  - **Test Isolation** - Improved test isolation by removing problematic `__init__.py` files
  - **Test Execution Consistency** - Enhanced consistency in test execution across different environments

- **Validation System Fixes** - Continued improvements to validation accuracy
  - **Runtime Validation** - Fixed issues in runtime validation testing
  - **Syntax Error Resolution** - Fixed syntax errors in validation components
  - **Test Error Resolution** - Systematic resolution of test errors and failures

### Technical Details
- **Testing Framework** - Complete migration from unittest to pytest with improved test patterns and execution
- **Documentation System** - Sphinx-based documentation generation with comprehensive API references
- **Step Catalog Architecture** - New unified step catalog system with enhanced integration capabilities
- **Docker Infrastructure** - Complete Docker containers for PyTorch and XGBoost ML workflows
- **Import System** - Continued improvements to relative import system for better modularity

### Quality Assurance
- **Test Framework Modernization** - Modern pytest-based testing infrastructure with improved reliability
- **Documentation Quality** - Enhanced documentation quality with automated generation and API references
- **Validation System Reliability** - Continued improvements to validation system accuracy and performance
- **Code Organization** - Better code organization with improved import structure and modularity

### Performance Improvements
- **Test Execution Speed** - Improved test execution performance with pytest framework
- **Documentation Generation** - Faster documentation generation with Sphinx automation
- **Import Performance** - Enhanced import performance with relative import system
- **Validation Performance** - Optimized validation system performance with improved algorithms

## [1.2.3] - 2025-09-06

### Enhanced
- **Runtime Testing Infrastructure** - Major improvements to pipeline runtime testing system
  - **Runtime Testing Refactoring** - Comprehensive refactoring of runtime testing components for better organization and maintainability
  - **Testing Framework Simplification** - Streamlined testing framework with reduced redundancy and improved clarity
  - **CLI Module Standardization** - Enhanced CLI module with better structure and standardized command patterns
  - **Dependency Management** - Improved dependency resolution and management across testing components

- **Documentation and Tutorials** - Significant improvements to user-facing documentation
  - **Tutorial Updates** - Enhanced tutorials with better examples and clearer explanations
  - **Reference Documentation** - Updated API reference documentation with improved coverage
  - **Quick Onboarding Guide** - New quick start guide for faster user onboarding
  - **Progress Tracking** - Enhanced progress tracking and reporting in documentation

- **Workflow Orchestration System** - New workflow orchestration capabilities
  - **Workflow Orchestrator** - New workflow orchestrator for managing complex pipeline workflows
  - **Agentic Workflow Integration** - Enhanced agentic workflow capabilities with better automation
  - **Workflow Analysis** - Comprehensive analysis tools for workflow optimization and monitoring

- **Code Quality and Organization** - Major code cleanup and standardization efforts
  - **Import System Improvements** - Fixed test imports and standardized to relative imports throughout codebase
  - **Code Redundancy Reduction** - Systematic analysis and removal of redundant code in runtime testing
  - **Module Organization** - Better organization of modules with clearer separation of concerns
  - **Utility Functions** - Enhanced utility functions with improved functionality and reliability

### Added
- **Visual Documentation** - New visual elements to enhance documentation
  - **Workflow Plots** - Added comprehensive plots and diagrams for workflow visualization
  - **README Updates** - Enhanced README with better visual elements and clearer structure
  - **Documentation Plots** - New plots and charts to illustrate system architecture and workflows

- **Testing Infrastructure Expansion** - Enhanced testing capabilities
  - **Relative Import Testing** - Comprehensive testing of relative import system
  - **Runtime Testing Utilities** - New utilities for runtime testing and validation
  - **Test Data Management** - Improved test data management with better cleanup and organization

### Fixed
- **Import System Issues** - Comprehensive fixes to import-related problems
  - **Relative Import Migration** - Successfully migrated from absolute to relative imports throughout codebase
  - **Test Import Fixes** - Fixed import issues in test modules for better reliability
  - **Module Path Resolution** - Improved module path resolution and import consistency

- **Code Redundancy Issues** - Systematic elimination of code duplication
  - **Runtime Testing Cleanup** - Removed redundant code in runtime testing components
  - **Dependency Cleanup** - Cleaned up unnecessary dependencies and imports
  - **Version Management** - Removed redundant version definitions and standardized version handling

- **Data Management** - Improved data handling and cleanup
  - **Data Removal** - Cleaned up unnecessary data files and improved data management
  - **Test Data Organization** - Better organization of test data with proper cleanup procedures

### Technical Details
- **Import Architecture** - Migrated to relative import system for better modularity and maintainability
- **Testing Framework** - Enhanced testing framework with better organization and reduced redundancy
- **Workflow System** - New workflow orchestration system with agentic capabilities
- **Documentation System** - Improved documentation system with visual elements and better organization
- **Code Quality** - Comprehensive code quality improvements with standardization and cleanup

### Quality Assurance
- **Import System Validation** - Comprehensive validation of relative import system
- **Testing Reliability** - Improved test reliability with better import handling and organization
- **Documentation Quality** - Enhanced documentation quality with better examples and visual elements
- **Code Standardization** - Systematic code standardization across all modules

### Performance Improvements
- **Import Performance** - Improved import performance with relative import system
- **Testing Performance** - Enhanced testing performance with reduced redundancy and better organization
- **Documentation Generation** - Faster documentation generation with improved tooling
- **Workflow Execution** - Optimized workflow execution with better orchestration

## [1.2.2] - 2025-09-05

### Enhanced
- **Developer Documentation System** - Major improvements to developer guides and documentation
  - **Pipeline Integration Guide** - New comprehensive guide for pipeline catalog integration
  - **Creation Process Updates** - Enhanced step creation process documentation with better workflow guidance
  - **Step Builder Documentation** - Updated step builder documentation with improved examples and patterns
  - **Prerequisites Guide** - Enhanced prerequisites documentation for better developer onboarding
  - **Standardization Rules** - Updated rules and guidelines for consistent development practices

- **Hybrid Registry System Completion** - Finalized the hybrid registry system architecture
  - **Registry System Optimization** - Completed code redundancy reduction and registry system optimization
  - **Workspace-Aware Registry** - Finished implementation of workspace-aware registry management
  - **Registry Integration** - Completed integration between hybrid registry and core system components
  - **Registry Testing** - Comprehensive testing and validation of hybrid registry functionality

- **Validation Framework Enhancements** - Significant improvements to validation system
  - **Step Name Validation** - Added comprehensive validation for step naming conventions
  - **Enhanced Validation Logic** - Improved validation algorithms with better error detection and reporting
  - **Validation Testing** - Enhanced test coverage for validation framework components
  - **CLI Validation Tools** - Updated CLI tools with improved validation capabilities

- **Testing Infrastructure Improvements** - Major enhancements to testing framework
  - **Registry Testing** - Enhanced testing for registry system components
  - **Step Builder Testing** - Improved step builder test framework with better coverage
  - **Validation Testing** - Comprehensive testing of validation framework functionality
  - **Test Organization** - Better organization and structure of test components

- **Design Documentation Updates** - Comprehensive updates to design and planning documentation
  - **Architecture Documentation** - Enhanced design documentation with updated architectural patterns
  - **Planning Documentation** - Updated project planning documents with current development status
  - **User Guide Planning** - Created comprehensive plan for user guide updates and improvements
  - **Analysis Documentation** - Enhanced analysis and evaluation documentation

### Added
- **Pipeline Catalog Integration Guide** - New comprehensive guide for integrating with pipeline catalog system
  - **Zettelkasten Integration** - Documentation for Zettelkasten-inspired catalog organization
  - **Catalog Management** - Guidelines for managing pipeline catalog entries and metadata
  - **Integration Patterns** - Best practices for pipeline catalog integration workflows

- **Enhanced CLI Tools** - Expanded command-line interface capabilities
  - **Validation Commands** - New CLI commands for comprehensive validation workflows
  - **Step Management** - Enhanced step management and discovery tools
  - **Registry Operations** - Improved registry management through CLI interface

### Fixed
- **Documentation Consistency** - Resolved inconsistencies in developer documentation
  - **Guide Alignment** - Ensured consistency across all developer guide documents
  - **Reference Updates** - Updated cross-references and links throughout documentation
  - **Format Standardization** - Standardized documentation format and structure

- **Validation System Stability** - Improved stability and reliability of validation components
  - **Error Handling** - Enhanced error handling in validation workflows
  - **Test Reliability** - Improved reliability of validation tests and framework
  - **Registry Validation** - Fixed issues in registry system validation

### Technical Details
- **Documentation Architecture** - Comprehensive documentation system with improved organization and cross-referencing
- **Validation Framework** - Enhanced validation system with step name validation and improved testing
- **CLI Integration** - Full integration of validation and management tools into command-line interface
- **Registry System** - Continued improvements to registry system with better testing and validation

### Quality Assurance
- **Documentation Quality** - Comprehensive review and improvement of all developer documentation
- **Validation Coverage** - Enhanced validation coverage with new step name validation capabilities
- **Test Reliability** - Improved test reliability and coverage across validation and registry components
- **Development Workflow** - Streamlined development workflow with better documentation and tools

### Performance Improvements
- **Documentation Access** - Improved organization and accessibility of developer documentation
- **Validation Performance** - Enhanced performance of validation operations with optimized algorithms
- **CLI Responsiveness** - Improved responsiveness of CLI tools with optimized execution paths

## [1.2.1] - 2025-09-03

### Enhanced
- **Registry System Refactoring** - Major improvements to component registry architecture
  - **Registry Migration** - Moved registry system into separate folder for better organization
  - **Import Path Optimization** - Comprehensive cleanup of import paths across registry components
  - **Builder Registry Updates** - Enhanced builder registry with improved component discovery
  - **Registry Integration** - Better integration between registry components and core system

- **Testing Infrastructure Improvements** - Significant enhancements to testing framework
  - **Script Alignment Testing** - Enhanced script alignment validation with improved accuracy
  - **Builder Registry Testing** - Comprehensive testing of builder registry functionality
  - **Test Error Resolution** - Fixed systematic test errors and improved test reliability
  - **Test Coverage Expansion** - Expanded test coverage across registry and validation components

- **Code Organization and Cleanup** - Major code organization improvements
  - **Import Path Standardization** - Removed remaining import path errors throughout codebase
  - **Code Simplification** - Phase 1 code simplification with redundancy removal
  - **Workspace-Aware System Updates** - Enhanced workspace-aware data structures organization
  - **Documentation Updates** - Updated design docs and planning documentation

### Fixed
- **Import Path Issues** - Comprehensive resolution of import path problems
  - **Registry Import Fixes** - Fixed import issues in registry system components
  - **Module Path Corrections** - Corrected module paths across builder and validation systems
  - **Circular Import Prevention** - Enhanced import structure to prevent circular dependencies
  - **Type Import Optimization** - Improved type imports for better performance and reliability

- **Testing System Stability** - Major improvements to test execution reliability
  - **Test Error Resolution** - Fixed systematic test errors affecting validation framework
  - **Mock Infrastructure** - Enhanced mock infrastructure for better test isolation
  - **Test Execution Consistency** - Improved consistency in test execution across different environments
  - **Validation Framework Fixes** - Fixed issues in alignment validation and script testing

### Technical Details
- **Registry Architecture** - Reorganized registry system with improved modularity and separation of concerns
- **Testing Framework** - Enhanced testing infrastructure with better error handling and reporting
- **Code Quality** - Improved code quality through systematic cleanup and standardization
- **Import Management** - Streamlined import management with better dependency resolution

### Quality Assurance
- **Code Standardization** - Comprehensive code standardization across registry and testing components
- **Test Reliability** - Improved test reliability with better error handling and mock infrastructure
- **Documentation Quality** - Enhanced documentation with updated design and planning documents
- **System Integration** - Better integration between registry, testing, and validation components

### Performance Improvements
- **Import Performance** - Optimized import performance through better path management
- **Test Execution Speed** - Improved test execution speed with enhanced infrastructure
- **Registry Operations** - Faster registry operations with optimized component discovery
- **Memory Management** - Better memory management in testing and validation operations

## [1.2.0] - 2025-09-02

### Added
- **Workspace-Aware System Architecture** - Major new infrastructure for multi-workspace development
  - **Workspace Isolation Principle** - Each workspace maintains independent configuration and execution context
  - **Shared Core Principle** - Core functionality remains shared across workspaces for consistency
  - **Extension-Based Design** - Backward-compatible architecture that extends existing functionality
  - **Workspace CLI Integration** - Enhanced CLI tools with workspace-aware capabilities

- **Enhanced Testing Infrastructure** - Comprehensive testing framework for workspace-aware components
  - **Universal Step Builder Testing** - 424 tests across Processing (280), CreateModel (72), and Training (72) step builders
  - **Alignment Validation System** - 4-level validation framework with 100% success rate across 9 production scripts
  - **Workspace-Aware Test Framework** - Testing infrastructure that validates workspace isolation and shared core principles
  - **Comprehensive Test Coverage** - 100% backward compatibility validation with existing functionality

- **Advanced Validation Framework** - Production-ready validation system with workspace support
  - **Multi-Level Alignment Validation** - Script ↔ Contract, Contract ↔ Specification, Specification ↔ Dependencies, Builder ↔ Configuration
  - **Workspace Context Validation** - Ensures proper workspace isolation while maintaining shared functionality
  - **Quality Scoring System** - Weighted performance metrics across validation levels
  - **Zero False Positives** - Eliminated systematic validation issues through enhanced pattern recognition

### Enhanced
- **CLI System Improvements** - Major enhancements for workspace-aware development
  - **Workspace Command Integration** - CLI commands now support workspace-specific operations
  - **Enhanced Error Handling** - Better error messages and workspace context awareness
  - **Improved User Experience** - Streamlined workspace setup and management commands
  - **Backward Compatibility** - All existing CLI functionality preserved while adding workspace features

- **Core Infrastructure Updates** - Fundamental improvements to support workspace architecture
  - **Configuration Management** - Enhanced configuration system with workspace-aware field management
  - **Registry System** - Updated registry system to support workspace-specific component discovery
  - **Dependency Resolution** - Improved dependency resolution with workspace context awareness
  - **Template System** - Enhanced template system supporting workspace-specific customizations

- **Testing System Reliability** - Major improvements to test execution and validation
  - **Test Execution Stability** - Resolved pytest import issues through direct Python execution
  - **Enhanced Mock Infrastructure** - Improved mock factory system with better error handling
  - **Performance Optimization** - Optimized test execution with average 32 seconds per builder
  - **Comprehensive Reporting** - Detailed test reports with performance metrics and quality analysis

### Fixed
- **Test Infrastructure Issues** - Resolved systematic problems affecting test execution
  - **Import Path Resolution** - Fixed "ModuleNotFoundError: No module named 'cursus.test'" by using direct Python execution
  - **Python Cache Issues** - Cleared cache files to resolve import conflicts
  - **Test Isolation** - Improved test isolation to prevent state contamination between workspace contexts
  - **Mock Configuration** - Enhanced mock setup for workspace-aware testing scenarios

- **Workspace Integration Issues** - Comprehensive fixes for workspace-aware functionality
  - **Configuration Merging** - Fixed configuration merging issues in workspace contexts
  - **Path Resolution** - Improved path resolution for workspace-specific resources
  - **Registry Consistency** - Ensured consistent component registration across workspace boundaries
  - **Template Generation** - Fixed template generation issues in workspace-aware environments

### Technical Details
- **Workspace Architecture** - Extension-based design with 5 major system blocks transformation scope
- **Test Coverage** - 424 step builder tests + 9 alignment validation scripts with 100% success rate
- **Quality Metrics** - Perfect backward compatibility with zero breaking changes to existing functionality
- **Performance** - Maintained existing performance while adding workspace capabilities
- **Documentation** - Comprehensive workspace-aware design documentation and implementation guides

### Quality Assurance
- **100% Backward Compatibility** - All existing functionality preserved during workspace-aware transformation
- **Comprehensive Validation** - Full validation of workspace isolation and shared core principles
- **Zero Regression** - No existing functionality broken during infrastructure updates
- **Production Readiness** - Workspace-aware system ready for production deployment

### Performance Improvements
- **Test Execution Optimization** - Improved test execution speed with optimized resource management
- **Workspace Context Switching** - Efficient workspace context management with minimal overhead
- **Configuration Processing** - Enhanced configuration processing performance in workspace environments
- **Registry Operations** - Optimized registry operations for workspace-aware component discovery

## [1.1.1] - 2025-08-25

### Added
- **Pipeline Runtime Testing Infrastructure** - Comprehensive testing framework for pipeline runtime execution
  - **Production Testing Suite** - New production-level testing with deployment validation, end-to-end testing, health checking, and performance optimization
  - **Jupyter Integration Testing** - Complete test suite for Jupyter notebook integration with advanced debugging, visualization, and template testing
  - **Integration Testing Framework** - Enhanced integration testing with real data testing, S3 data downloading, and workspace management
  - **Runtime CLI Enhancements** - Updated runtime CLI with improved pipeline execution and testing capabilities

- **Pipeline Execution System** - Enhanced pipeline execution infrastructure
  - **Pipeline Executor** - New comprehensive pipeline executor with advanced execution capabilities
  - **Real Data Tester** - Integration testing with real data scenarios and validation
  - **S3 Data Downloader** - Automated data downloading and management for testing scenarios
  - **Workspace Manager** - Enhanced workspace management for testing and execution environments

- **Testing Infrastructure Expansion** - Major expansion of testing capabilities
  - **Runtime Validation Tests** - Comprehensive runtime validation testing across multiple categories
  - **DAG Resolver Testing** - Enhanced testing for pipeline DAG resolution and validation
  - **Comprehensive Test Runner** - Updated test runner with improved coverage and execution
  - **Unit Test Enhancements** - Expanded unit test coverage with improved test reliability

### Enhanced
- **Project Planning and Documentation** - Updated project planning with detailed implementation plans
  - **Pipeline Runtime Integration Plans** - Detailed plans for Jupyter integration, S3 integration, and testing master implementation
  - **Design Documentation Updates** - Enhanced design documentation with latest architectural decisions
  - **Notebook Templates** - Updated Jupyter notebook templates with improved functionality and examples

- **Validation System Improvements** - Enhanced validation capabilities across the system
  - **Runtime Validation** - Improved runtime validation with better error detection and reporting
  - **Integration Validation** - Enhanced integration validation with real-world testing scenarios
  - **Performance Validation** - New performance validation and optimization capabilities

### Fixed
- **Test Infrastructure Stability** - Improved stability and reliability of testing infrastructure
  - **Test Execution Reliability** - Enhanced test execution with better error handling and recovery
  - **Data Management** - Improved data management for testing scenarios with better cleanup and validation
  - **Runtime Execution** - Fixed issues in runtime execution with better error handling and logging

### Technical Details
- **Testing Architecture** - Comprehensive testing framework covering production, Jupyter integration, and runtime validation
- **Pipeline Execution** - Enhanced pipeline execution system with real data testing and S3 integration
- **Validation Framework** - Improved validation framework with comprehensive testing across multiple levels
- **CLI Integration** - Enhanced CLI tools with better runtime execution and testing capabilities

### Quality Assurance
- **Comprehensive Testing** - Extensive testing infrastructure covering all aspects of pipeline runtime execution
- **Real Data Validation** - Testing with real data scenarios to ensure production readiness
- **Performance Monitoring** - Enhanced performance monitoring and optimization capabilities
- **Integration Validation** - Thorough validation of integration between different system components

### Performance Improvements
- **Test Execution Speed** - Optimized test execution with improved performance and resource utilization
- **Pipeline Runtime Performance** - Enhanced pipeline runtime performance with better resource management
- **Data Processing Efficiency** - Improved efficiency in data processing and management operations

## [1.1.0] - 2025-08-21

### Added
- **MODS Pipeline Support** - Major new feature for advanced pipeline management
  - **MODS DAG Compiler** - New compiler specifically designed for MODS (Model Operations and Data Science) pipelines
  - **MODS Pipeline Integration** - Full integration with existing pipeline infrastructure
  - **MODS Pipeline Catalog** - Dedicated catalog for MODS-specific pipeline templates and examples
  - **Enhanced Pipeline Metadata** - Extended metadata system to support MODS pipeline requirements

- **Pipeline Catalog Restructuring** - Complete reorganization of pipeline catalog architecture
  - **Shared DAG Structure** - New shared DAG architecture for better reusability and maintainability
  - **Catalog Folder Restructuring** - Improved organization with clear separation of concerns
  - **Enhanced Pipeline Templates** - Updated pipeline templates with better structure and documentation
  - **Catalog Index System** - New indexing system for better pipeline discovery and management

- **Model Calibration Enhancements** - Expanded model calibration capabilities
  - **Flexible Calibration Dependencies** - Model calibration steps can now depend on either training or evaluation steps
  - **Enhanced Calibration Workflows** - Improved calibration workflow patterns for better flexibility
  - **Calibration Step Variants** - Support for different calibration step configurations and dependencies

### Enhanced
- **CLI System Improvements** - Major enhancements to command-line interface
  - **Registry Updates** - Improved registry system with better component discovery and management
  - **Enhanced CLI Commands** - Updated CLI commands with better functionality and user experience
  - **Improved Error Handling** - Better error messages and handling in CLI operations

- **Pipeline Metadata System** - Significant improvements to pipeline metadata handling
  - **EnhancedDAGMetadata** - New enhanced metadata system with extended capabilities
  - **DAGMetadata Updates** - Improved core metadata system with better data structures
  - **Metadata Integration** - Better integration of metadata across pipeline components

- **Pipeline Output Management** - Improved pipeline output handling
  - **Non-MODS Pipeline Output Fix** - Fixed issues with non-MODS pipeline output generation
  - **Output Consistency** - Improved consistency in pipeline output across different pipeline types
  - **Better Output Validation** - Enhanced validation of pipeline outputs

### Fixed
- **Pipeline Catalog Issues** - Resolved various issues in pipeline catalog management
  - **Structure Cleanup** - Removed old structure and maintained only new, improved structure
  - **Catalog Consistency** - Fixed inconsistencies in catalog organization and structure
  - **Template Validation** - Improved validation of pipeline templates and examples

- **Registry System Fixes** - Enhanced registry system reliability
  - **Component Registration** - Fixed issues with component registration and discovery
  - **Registry Consistency** - Improved consistency in registry operations
  - **Better Error Recovery** - Enhanced error recovery in registry operations

### Technical Details
- **MODS Architecture** - Complete MODS pipeline architecture with dedicated compiler and catalog system
- **Catalog Restructuring** - New folder structure with shared DAGs and improved organization
- **Metadata Enhancement** - Extended metadata system supporting both traditional and MODS pipelines
- **CLI Integration** - Full integration of new features into command-line interface
- **Backward Compatibility** - Maintained backward compatibility while adding new MODS features

### Quality Assurance
- **Comprehensive Testing** - Extensive testing of new MODS pipeline features and catalog restructuring
- **Documentation Updates** - Updated documentation to reflect new features and architectural changes
- **Integration Validation** - Thorough validation of integration between MODS and existing pipeline systems
- **Performance Optimization** - Optimized performance for new features and improved existing functionality

### Performance Improvements
- **Catalog Performance** - Improved performance of pipeline catalog operations with new structure
- **Metadata Processing** - Enhanced performance of metadata processing and management
- **CLI Responsiveness** - Improved responsiveness of CLI operations with optimized registry system

## [1.0.12] - 2025-08-19

### Added
- **Comprehensive Documentation Expansion** - Major expansion of validation and testing documentation
  - **Builder Validation Documentation** - Detailed documentation for step builder validation patterns
  - **Test Variant Documentation** - Comprehensive guides for processing, createmodel, and registermodel test variants
  - **Interface Testing Guides** - Detailed documentation for interface compliance testing
  - **Specification Testing Documentation** - Enhanced guides for specification validation patterns
  - **Integration Testing Documentation** - Complete documentation for integration test patterns

- **Pipeline Catalog Documentation** - New pipeline catalog with documented examples
  - **Example Pipeline Collection** - Curated collection of pipeline examples with detailed documentation
  - **Best Practices Documentation** - Enhanced documentation of pipeline development best practices
  - **Usage Pattern Documentation** - Documented common usage patterns and implementation examples

### Enhanced
- **Developer Documentation** - Significant improvements to developer-facing documentation
  - **Validation Framework Documentation** - Enhanced documentation for the validation framework
  - **Testing Infrastructure Guides** - Improved documentation for testing infrastructure and patterns
  - **Step Builder Documentation** - Enhanced documentation for step builder development and testing
  - **Quality Assurance Documentation** - Comprehensive documentation for quality assurance processes

- **Technical Documentation** - Expanded technical documentation and guides
  - **Architecture Documentation** - Enhanced documentation of system architecture and design patterns
  - **Implementation Guides** - Detailed implementation guides for various components
  - **Testing Methodology Documentation** - Comprehensive documentation of testing methodologies and approaches

### Documentation Quality Improvements
- **Comprehensive Coverage** - Documentation now covers all major validation and testing components
- **Detailed Examples** - Added detailed examples and usage patterns throughout documentation
- **Developer Experience** - Improved developer experience with better organized and more accessible documentation
- **Quality Standards** - Enhanced documentation quality standards and consistency

### Technical Details
- **Documentation Structure** - Organized documentation into logical categories and hierarchies
- **Cross-References** - Enhanced cross-referencing between related documentation sections
- **Code Examples** - Added comprehensive code examples and usage patterns
- **Best Practices** - Documented best practices for development, testing, and validation

## [1.0.11] - 2025-08-17

### Enhanced
- **Alignment Validation Visualization System** - Revolutionary visual reporting and analysis capabilities
  - **Visual Score Charts** - Comprehensive PNG chart generation for all alignment validation reports
  - **Enhanced Validation Reports** - Improved HTML and JSON reports with detailed scoring breakdowns
  - **Interactive Visualization** - Score charts showing level-by-level validation performance with color-coded results
  - **Comprehensive Report Generation** - Automated generation of visual reports for all 9 production scripts

- **CLI Tools Enhancement** - Major improvements to command-line interface capabilities
  - **Enhanced Alignment CLI** - Improved alignment validation CLI with better error handling and reporting
  - **Builder Test CLI Improvements** - Enhanced builder test CLI with comprehensive test execution and reporting
  - **Streamlined Report Generation** - Optimized CLI tools for faster report generation and visualization
  - **Better User Experience** - Improved command-line interface with clearer output and progress indicators

- **Validation System Improvements** - Advanced validation capabilities with enhanced accuracy
  - **Alignment Scorer Enhancement** - Improved scoring algorithms with better accuracy and detailed breakdowns
  - **Pattern Recognition Improvements** - Enhanced pattern recognition for better validation accuracy
  - **False Positive Elimination** - Further reduction of false positives in validation alignment tests
  - **Unified Alignment Tester** - Enhanced unified tester with better integration and visualization support

### Added
- **Comprehensive Visualization Infrastructure** - New visualization capabilities for validation results
  - **Score Chart Generation** - Automated PNG chart generation for all validation reports
  - **Enhanced Validation Summary** - Comprehensive validation summary with visual elements and detailed metrics
  - **Multi-Level Visualization** - Visual representation of validation results across all four validation levels
  - **Interactive Report Integration** - Integration of visual charts into HTML reports for better user experience

- **Advanced Testing Framework** - Expanded testing capabilities with visualization support
  - **Visualization Integration Tests** - Complete test suite for visualization integration and chart generation
  - **Alignment Scorer Tests** - Comprehensive tests for alignment scoring algorithms and accuracy
  - **Full Validation Tests** - End-to-end validation tests with visual report generation
  - **Level-Specific Validation Tests** - Specialized tests for each validation level with visual feedback

- **Documentation and Design Updates** - Comprehensive documentation improvements
  - **Multi-Developer Workspace Management** - New design documentation for multi-developer workspace management
  - **Alignment Validation Data Structures** - Enhanced documentation for alignment validation data structures
  - **Visualization Integration Design** - Detailed design documentation for visualization integration
  - **Project Planning Updates** - Updated project planning documents with visualization integration plans

### Fixed
- **Validation Alignment False Positives** - Systematic elimination of remaining false positive issues
  - **Pattern Recognition Accuracy** - Improved pattern recognition algorithms to reduce false positives
  - **Configuration Analysis** - Enhanced configuration analysis with better error detection and reporting
  - **Builder Config Alignment** - Fixed remaining issues in builder-configuration alignment validation
  - **Report Generation Stability** - Improved stability of report generation with better error handling

- **CLI Tool Reliability** - Enhanced reliability and performance of command-line tools
  - **Error Handling Improvements** - Better error handling and recovery in CLI tools
  - **Report Generation Optimization** - Optimized report generation for faster execution and better performance
  - **Memory Management** - Improved memory management in CLI tools for better resource utilization
  - **Output Formatting** - Enhanced output formatting for better readability and user experience

- **Visualization System Stability** - Improved stability and reliability of visualization components
  - **Chart Generation Reliability** - Enhanced reliability of PNG chart generation with better error handling
  - **Report Integration** - Improved integration of visual elements into validation reports
  - **File System Operations** - Better handling of file system operations for report and chart generation
  - **Resource Management** - Improved resource management for visualization components

### Technical Details
- **Visualization Architecture** - Comprehensive visualization system with automated chart generation and report integration
- **Enhanced Scoring System** - Improved scoring algorithms with detailed breakdowns and visual representation
- **CLI Integration** - Full integration of visualization capabilities into command-line interface tools
- **Report Generation Pipeline** - Streamlined pipeline for generating comprehensive validation reports with visual elements
- **Performance Optimization** - Optimized performance for large-scale validation and visualization operations

### Quality Assurance
- **Visual Validation** - Comprehensive visual validation of all alignment validation results
- **Enhanced Reporting** - Detailed reporting with visual elements for better analysis and understanding
- **Comprehensive Testing** - Extensive testing of visualization and reporting capabilities
- **Performance Monitoring** - Enhanced performance monitoring for validation and visualization operations

### Performance Improvements
- **Report Generation Speed** - Optimized report generation for faster execution and better performance
- **Visualization Performance** - Improved performance of chart generation and visual report creation
- **Memory Optimization** - Better memory management for large-scale validation and visualization operations
- **CLI Tool Performance** - Enhanced performance of command-line tools with optimized execution paths

## [1.0.10] - 2025-08-16

### Enhanced
- **Universal Step Builder Testing System** - Major improvements to step builder validation framework
  - **Perfect Test Compliance** - Achieved 100% success rate across all 13 step builders (427/427 tests passed)
  - **Enhanced 4-Level Validation** - Comprehensive testing across Interface, Specification, Step Creation, and Integration levels
  - **Step Type-Specific Validation** - Specialized validation patterns for Processing, Training, Transform, and CreateModel step types
  - **Comprehensive Test Coverage** - 387 individual tests with 1,247 total assertions across all builder types

- **Mock Factory System Consolidation** - Unified and enhanced mock infrastructure
  - **Consolidated Mock Factory** - Merged `mock_factory.py` and `enhanced_mock_factory.py` into single improved system
  - **Enhanced Error Handling** - Added test mode parameter for graceful error handling with informative messages
  - **Automatic Script File Creation** - Creates `/tmp/mock_scripts` directory and generates all required script files
  - **Improved Configuration Validation** - Uses valid AWS regions, proper ARN formats, and enhanced hyperparameter handling

- **Step Builder Reliability Improvements** - Critical fixes for model step builders
  - **PyTorch Model Step Builder** - Fixed CreateModelStep API usage and mock infrastructure compatibility
  - **XGBoost Model Step Builder** - Resolved configuration issues and parameter handling
  - **Batch Transform Step Builder** - Enhanced batch processing configuration and model integration
  - **PyTorch Training Step Builder** - Fixed spec-contract alignment issues and data output property paths

### Added
- **Comprehensive Test Reporting System** - Advanced test result analysis and visualization
  - **Individual Builder Reports** - Detailed JSON reports and score charts for each step builder
  - **Performance Metrics Analysis** - Test execution timing, resource utilization, and quality distribution
  - **Visual Score Charts** - PNG charts showing test results and scoring breakdown for each builder
  - **Consolidated Summary Reports** - Master summary with overall statistics and quality metrics

- **Enhanced Validation Infrastructure** - Expanded validation capabilities
  - **Step Type Variants** - Specialized test variants for different SageMaker step types
  - **Registry Integration Tests** - Comprehensive builder registration and discovery validation
  - **Path Mapping Validation** - Enhanced property path validation with comprehensive pattern matching
  - **Configuration Alignment Tests** - Builder-configuration alignment validation with detailed analysis

- **Documentation and Analysis** - Comprehensive documentation improvements
  - **Test Execution Summaries** - Detailed execution reports with timestamps and performance metrics
  - **Builder Analysis Reports** - In-depth analysis of builder patterns and validation results
  - **Alignment Validation Reports** - HTML and JSON reports for script-contract-specification alignment
  - **CLI User Guides** - Enhanced command-line interface documentation and usage guides

### Fixed
- **Critical Step Builder Issues** - Resolved systematic problems affecting model step builders
  - **CreateModel Step Configuration** - Fixed mock infrastructure incompatibility with newer Python tar extraction
  - **SageMaker API Alignment** - Corrected CreateModelStep constructor to use model parameter instead of step_args
  - **PyTorch Training Alignment** - Removed incorrect checkpoints output from contract and updated specification paths
  - **Mock Factory Compatibility** - Enhanced mock_extractall function to handle filter parameter for Python 3.12+

- **Test Infrastructure Stability** - Major improvements to test execution reliability
  - **False Positive Elimination** - Addressed configuration creation failures and script path validation errors
  - **Hyperparameter Validation** - Fixed XGBoost and PyTorch hyperparameter creation with proper field validation
  - **Script File Generation** - Eliminated "script not found" errors through automatic file creation
  - **Region Validation** - Fixed AWS region validation issues with proper region codes

- **Configuration Management Fixes** - Enhanced configuration handling and validation
  - **Field List Validation** - Proper validation of categorical and tabular field list relationships
  - **Container Path Management** - Improved handling of `/opt/ml/processing/` paths in processing steps
  - **Environment Variable Handling** - Enhanced JSON serialization for complex configurations
  - **Dependency Resolution** - Fixed dependency handling and step name consistency issues

### Technical Details
- **Test Architecture** - 4-level validation system with weighted scoring (Interface: 25%, Specification: 25%, Step Creation: 30%, Integration: 20%)
- **Quality Metrics** - 100% success rate across all builder types with perfect compliance scores
- **Performance** - ~7 minutes total execution time for all 13 builders with efficient resource utilization
- **Coverage Analysis** - 387 individual tests covering all step builder functionality and edge cases
- **Report Generation** - Automated JSON reports, PNG charts, and comprehensive documentation for each builder

### Quality Assurance
- **Universal Compliance** - All 13 step builders achieve 100% compliance with universal testing standards
- **Zero False Positives** - Eliminated systematic false positive issues through enhanced mock infrastructure
- **Comprehensive Validation** - Full coverage of interface, specification, step creation, and integration testing
- **Performance Monitoring** - Detailed performance metrics and resource utilization analysis

### Performance Improvements
- **Test Execution Speed** - Optimized test execution with average 32 seconds per builder
- **Memory Management** - Efficient mock object management and cleanup
- **Report Generation** - Fast report and chart generation (~5 seconds per builder)
- **Resource Utilization** - Moderate CPU usage with ~50MB disk usage for all reports and charts

## [1.0.9] - 2025-08-14

### Enhanced
- **Step Specification System** - Major improvements to step specification handling
  - **Job Type Helper Integration** - Updated all step specifications to use job type helper for better variant handling
  - **Naming Convention Standardization** - Comprehensive standardization of step, config, contract, and script names
  - **Step Registry Enhancement** - Improved step name registry with better inference and consistency

- **Dependency Resolution System** - Advanced dependency resolution capabilities
  - **Enhanced Dependency Resolver** - Improved dependency matching with better canonical name mapping
  - **Job Type Variant Handling** - Better support for training/testing/validation/calibration variants
  - **Production Resolver Integration** - Enhanced integration with production dependency resolver

- **Validation System Improvements** - Major enhancements to alignment validation
  - **Script Contract Validator** - Enhanced validator to accept complex path validation patterns
  - **Property Path Validator** - Augmented property path validation with improved pattern matching
  - **Validation Report Generation** - Comprehensive validation reports with detailed HTML and JSON outputs
  - **Test Framework Enhancement** - Improved test validation with better error reporting

- **Configuration Management** - Enhanced configuration system
  - **Cradle Config Factory** - New configuration factory for cradle data loading steps
  - **Config Field Management** - Improved configuration field handling and validation
  - **Three-Tier Configuration** - Enhanced three-tier configuration architecture

### Added
- **Step Type Enhancement System** - New step type enhancement framework
  - **Step Type Enhancers** - Base enhancer classes for different step types
  - **Enhancement Router** - Smart routing system for step type enhancements
  - **Framework Patterns** - Comprehensive framework patterns for step validation
  - **Step-Specific Validation** - Specialized validation patterns for different step types

- **Comprehensive Test Suite** - Expanded test infrastructure
  - **Step Type Enhancement Tests** - Complete test suite for step type enhancement system
  - **Alignment Validation Tests** - Enhanced alignment validation test framework
  - **Builder Argument Integration Tests** - Comprehensive builder argument validation tests
  - **Framework Pattern Tests** - Tests for framework patterns and step type detection

- **Documentation Updates** - Extensive documentation improvements
  - **Step Specification Guide** - Updated developer guide for step specifications
  - **Alignment Validation Patterns** - Detailed documentation for validation patterns
  - **Design Documentation** - Enhanced design documents for dependency resolution and job type handling

### Fixed
- **Import Path Corrections** - Resolved import errors in builder steps and specifications
  - **Builder Module Imports** - Fixed import paths in PyTorch and XGBoost builder modules
  - **Specification Imports** - Corrected import paths in training specifications
  - **Contract Discovery** - Fixed errors in contract and specification discovery

- **Naming Convention Issues** - Comprehensive fixes to naming inconsistencies
  - **Step Name Standardization** - Standardized step names across builders, configs, contracts, and scripts
  - **File Name Alignment** - Aligned file names with naming conventions
  - **Registry Consistency** - Ensured consistency between registry and actual component names

- **Validation System Fixes** - Major fixes to validation alignment
  - **File Operations Detection** - Enhanced detection of file operations in scripts
  - **Path Validation Logic** - Improved path validation with better pattern matching
  - **Validation Report Accuracy** - Fixed validation report generation and accuracy

### Technical Details
- **Step Specification Architecture** - Enhanced step specification system with job type helper integration
- **Dependency Resolution** - Advanced dependency resolution with canonical name mapping and variant support
- **Validation Framework** - Comprehensive validation framework with step-specific patterns and enhanced reporting
- **Configuration System** - Improved configuration management with factory patterns and tier-based organization
- **Test Infrastructure** - Expanded test suite with specialized tests for different components and patterns

### Quality Assurance
- **Validation Accuracy** - Improved validation accuracy with enhanced pattern matching and error detection
- **Test Coverage** - Expanded test coverage with specialized tests for different step types and validation patterns
- **Documentation Quality** - Enhanced documentation with detailed guides and examples
- **Code Standardization** - Comprehensive code standardization with consistent naming conventions

### Performance Improvements
- **Validation Performance** - Optimized validation execution with better caching and pattern matching
- **Dependency Resolution Speed** - Improved dependency resolution performance with enhanced algorithms
- **Test Execution** - Faster test execution with optimized test patterns and better resource management

## [1.0.8] - 2025-08-11

### Added
- **Complete Alignment Validation System** - Revolutionary four-level validation framework achieving 100% success rate
  - **Level 1: Script ↔ Contract Alignment** - Hybrid approach with robust sys.path management and enhanced static analysis
  - **Level 2: Contract ↔ Specification Alignment** - Smart Specification Selection with multi-variant handling and FlexibleFileResolver
  - **Level 3: Specification ↔ Dependencies Alignment** - Production dependency resolver integration with canonical name mapping
  - **Level 4: Builder ↔ Configuration Alignment** - Hybrid file resolution with comprehensive configuration validation

- **Advanced Validation Infrastructure** - Production-ready validation system with comprehensive reporting
  - **Hybrid File Resolution** - Primary direct matching with intelligent fallback mechanisms
  - **Smart Specification Selection** - Multi-variant detection and unified specification model creation
  - **Production Dependency Resolver** - Battle-tested dependency resolution with confidence scoring
  - **Enhanced Static Analysis** - Comprehensive file operations detection beyond simple `open()` calls

- **Validation CLI Enhancements** - Extended command-line interface for alignment validation
  - **Multi-Level Validation** - Run validation at specific levels or comprehensive validation across all levels
  - **Detailed Reporting** - JSON reports with comprehensive issue analysis and resolution recommendations
  - **Batch Validation** - Validate multiple scripts simultaneously with consolidated reporting

### Enhanced
- **Validation System Reliability** - Achieved 100% success rate across all validation levels
  - **Zero False Positives** - Eliminated systematic false positive issues that plagued earlier versions
  - **Complete Error Resolution** - All critical validation errors resolved through systematic analysis and targeted fixes
  - **Production-Ready Validation** - Validation system now suitable for CI/CD integration and automated quality gates

- **File Operations Detection** - Major improvements to static analysis capabilities
  - **Comprehensive Pattern Recognition** - Detects `tarfile.open()`, `shutil.copy2()`, `Path.mkdir()`, and other high-level operations
  - **Variable Tracking** - Correlates path constants with their usage throughout scripts
  - **Contract-Aware Validation** - Understands contract structure and validates against actual requirements

- **Naming Convention Handling** - Intelligent handling of legitimate naming variations
  - **Fuzzy Matching** - Enhanced similarity thresholds for pattern recognition
  - **Canonical Name Mapping** - Consistent name resolution between registry and dependency resolver
  - **Multi-Variant Support** - Handles job-type variants (training/testing/validation/calibration) correctly

### Fixed
- **Critical Validation System Issues** - Resolved systematic problems affecting all validation levels
  - **File Operations Detection Failure** - Fixed analyzer to detect all file operation patterns used in production scripts
  - **Logical Name Extraction Algorithm** - Corrected flawed path-to-logical-name mapping algorithm
  - **Argparse Convention Misunderstanding** - Fixed validation logic to understand standard hyphen-to-underscore conversion
  - **Path Usage Correlation** - Properly correlates path declarations with their usage in file operations

- **Dependency Resolution Issues** - Major fixes to specification-dependency alignment
  - **Canonical Name Mapping Inconsistency** - Resolved registry vs resolver name mapping issues
  - **Production Resolver Integration** - Integrated battle-tested production dependency resolver
  - **Registry Disconnect** - Fixed validation to use registry functions for proper name mapping

- **File Resolution Failures** - Comprehensive fixes to contract and specification file discovery
  - **Incorrect File Path Resolution** - Fixed file discovery to handle legitimate naming variations
  - **Multi-Variant Specification Handling** - Proper validation against union of all variant requirements
  - **Overly Strict Pattern Matching** - Enhanced fuzzy matching for legitimate naming patterns

### Technical Details
- **Validation Architecture** - Four-level validation system with hybrid resolution strategies and intelligent fallback mechanisms
- **Success Metrics** - 100% pass rate achieved across all 8 production scripts with zero critical errors
- **Error Elimination** - Reduced from 32+ critical errors to zero through systematic analysis and targeted fixes
- **Developer Experience** - Complete restoration of developer trust with accurate, actionable validation feedback

### Quality Assurance
- **Production Validation** - All 8 production scripts achieve PASSING status with comprehensive alignment validation
- **Zero False Positives** - Eliminated systematic false positive issues that made previous versions unusable
- **Comprehensive Testing** - Extensive validation against real-world production scripts and patterns
- **CI/CD Ready** - Validation system now suitable for automated quality gates and continuous integration

### Performance Improvements
- **Validation Speed** - Optimized validation execution with efficient file resolution and caching
- **Memory Usage** - Improved memory management with proper cleanup of temporary sys.path modifications
- **Error Reporting** - Enhanced error messages with detailed resolution paths and actionable recommendations

## [1.0.7] - 2025-08-08

### Added
- **Universal Step Builder Test System** - Comprehensive testing framework for step builder validation
  - **CLI Interface** - New command-line interface for running builder tests (`cursus.cli.builder_test_cli`)
  - **Four-Level Test Architecture** - Structured testing across Interface, Specification, Path Mapping, and Integration levels
  - **Test Scoring System** - Advanced scoring mechanism with weighted evaluation across test levels
  - **Processing Step Builder Tests** - Specialized test variants for processing step builders
  - **Test Pattern Detection** - Smart pattern-based test categorization and level detection

- **Enhanced Validation Infrastructure** - Major expansion of validation capabilities
  - **Builder Test Variants** - Specialized test classes for different step builder types (Processing, Training, Transform)
  - **Interface Standard Validator** - Validation of step builder interface compliance
  - **Test Scoring and Rating** - Comprehensive scoring system with quality ratings (Excellent, Good, Satisfactory, Needs Work, Poor)
  - **Test Report Generation** - Automated generation of detailed test reports with charts and statistics

- **CLI Enhancements** - Extended command-line interface functionality
  - **Universal Builder Test CLI** - Run tests at different levels and variants for any step builder
  - **Validation CLI** - Command-line validation tools for step builders and specifications
  - **Builder Discovery** - Automatic discovery and listing of available step builder classes

### Enhanced
- **Test Coverage and Quality** - Improved test infrastructure and coverage analysis
  - **Pattern-Based Test Scoring** - Advanced scoring system with level-based weighting and importance factors
  - **Test Level Detection** - Smart detection of test levels using explicit prefixes, keywords, and fallback mapping
  - **Comprehensive Test Reports** - Detailed reports with pass rates, level scores, and failure analysis
  - **Test Execution Performance** - Optimized test execution with better error handling and reporting

- **Documentation and Naming** - Improved consistency and documentation
  - **Brand Consistency** - Updated all references from "AutoPipe" to "Cursus" throughout the codebase
  - **Package Name Standardization** - Consistent use of "cursus" instead of legacy "autopipe" references
  - **Documentation Headers** - Standardized documentation format and headers across modules

### Fixed
- **Import Path Corrections** - Fixed import errors and path issues
  - **Module Import Fixes** - Resolved import path issues in validation and CLI modules
  - **Test Module Stability** - Improved test module imports and execution reliability
  - **Processing Step Builder Tests** - Fixed test execution issues for tabular preprocessing step builders

### Technical Details
- **Test Architecture** - Four-level testing system with weighted scoring (Interface: 1.0x, Specification: 1.5x, Path Mapping: 2.0x, Integration: 2.5x)
- **Scoring Algorithm** - Importance-weighted scoring with test-specific multipliers and comprehensive rating system
- **CLI Integration** - Full command-line interface for test execution with verbose output and detailed reporting
- **Test Discovery** - Automatic discovery of step builder classes with AST parsing for missing dependencies
- **Report Generation** - JSON reports, console output, and optional chart generation with matplotlib

### Quality Assurance
- **Universal Test Coverage** - Comprehensive testing framework covering all step builder types and patterns
- **Automated Quality Assessment** - Scoring system provides objective quality metrics for step builders
- **Regression Prevention** - Enhanced test suite prevents regressions in step builder implementations
- **Development Workflow** - Improved developer experience with CLI tools and automated validation

## [1.0.6] - 2025-08-07

### Added
- **Comprehensive Test Infrastructure** - Major expansion of testing capabilities
  - **Core Test Suite** - New comprehensive test runner for all core components (`test/core/run_core_tests.py`)
  - **Test Coverage Analysis** - Advanced coverage analysis tool with detailed reporting (`test/analyze_test_coverage.py`)
  - **Test Organization** - Restructured test directory with component-based organization
  - **Performance Metrics** - Test execution timing and performance analysis
  - **Quality Assurance Reports** - Automated generation of test quality and coverage reports

- **Three-Tier Configuration Architecture** - New configuration management system
  - **ConfigFieldTierRegistry** - Registry for field tier classifications (Tier 1: Essential User Inputs, Tier 2: System Inputs, Tier 3: Derived Inputs)
  - **Field Classification System** - Systematic categorization of 50+ configuration fields across three tiers
  - **Enhanced Configuration Management** - Improved configuration merging and serialization with tier awareness

- **Validation Module** - New pipeline validation infrastructure
  - **Pipeline Validation** - Utilities for validating pipeline components, specifications, and configurations
  - **Early Error Detection** - Catch configuration errors early in the development process
  - **Compatibility Checking** - Ensure specifications and contracts are valid and compatible

### Enhanced
- **Test Coverage Metrics** - Comprehensive analysis and reporting
  - **Overall Statistics** - 602 tests across 36 test modules with 85.5% success rate
  - **Component Coverage** - Detailed coverage analysis for Assembler (100%), Compiler (100%), Base (86.2%), Config Fields (99.0%), Deps (68.2%)
  - **Function Coverage** - 50.6% function coverage (166/328 functions tested) with detailed gap analysis
  - **Redundancy Analysis** - Identification and reporting of test redundancy patterns

- **Configuration Field Management** - Major improvements to config field handling
  - **Type-Aware Serialization** - Enhanced serialization with better type preservation
  - **Configuration Merging** - Improved config merger with better error handling
  - **Circular Reference Tracking** - Enhanced circular reference detection and prevention

### Fixed
- **Import Path Corrections** - Fixed multiple import path issues across the codebase
  - **Test Module Imports** - Corrected import paths in test modules for better reliability
  - **Configuration Module Imports** - Fixed import issues in config field modules
  - **Base Class Imports** - Resolved import path issues in base classes

- **Test Infrastructure Stability** - Improved test execution reliability
  - **Mock Configuration** - Fixed mock setup issues in factory tests
  - **Test Isolation** - Improved test isolation to prevent state contamination
  - **Path Configuration** - Fixed path configuration issues in test modules

### Technical Details
- **Test Infrastructure** - 602 tests with 1.56 second execution time and 1,847 total assertions
- **Code Organization** - Restructured test directory from flat structure to component-based hierarchy
- **Quality Metrics** - 94.9% unique test names with 5.1% redundancy across components
- **Performance** - Average test duration of 2.59ms per test with efficient memory usage
- **Documentation** - Comprehensive test documentation with usage examples and troubleshooting guides

### Quality Assurance
- **Automated Reporting** - Generated comprehensive test reports in JSON and Markdown formats
- **Coverage Tracking** - Detailed function-level coverage analysis with gap identification
- **Performance Monitoring** - Test execution performance metrics and optimization recommendations
- **Regression Prevention** - Enhanced test suite to prevent future regressions

## [1.0.5] - 2025-08-06

### Fixed
- **CRITICAL: Circular Import Resolution** - Completely resolved all circular import issues in the package
  - Fixed circular dependency in `cursus.core.base.builder_base` module that was preventing 89.3% of modules from importing
  - Implemented lazy loading pattern using property decorator to break circular import chain
  - Root cause: `builder_base.py` → `step_names` → `builders` → `builder_base.py` circular dependency
  - Solution: Converted direct import to lazy property loading with graceful fallback
  - **Result**: 98.7% module import success rate (157/159 modules), up from 10.7% (17/159 modules)

### Added
- **Comprehensive Circular Import Test Suite** - New testing infrastructure to prevent future regressions
  - Created `test/circular_imports/` directory with complete test framework
  - Added 5 comprehensive test categories covering all package modules
  - Automated detection and reporting of circular import issues
  - Import order independence testing to ensure robust module loading
  - Detailed error reporting with exact circular dependency chains
  - Test output logging with timestamps and comprehensive statistics

### Changed
- **Package Architecture Improvement** - Enhanced module loading reliability
  - All core packages now import successfully without circular dependencies
  - Maintained Single Source of Truth design principle while fixing imports
  - Preserved existing API compatibility during circular import resolution
  - Improved error handling for optional dependencies

### Technical Details
- **Module Import Success Rate**: Improved from 10.7% to 98.7% (157/159 modules successful)
- **Circular Imports Eliminated**: Reduced from 142 detected circular imports to 0
- **Core Package Health**: 100% of core packages (cursus.core.*) now import cleanly
- **Test Coverage**: 159 modules tested across 15 package categories
- **Only Remaining Import Issues**: 2 modules with missing optional dependencies (expected behavior)
- **Package Categories Tested**: Core (4), API (1), Steps (7), Processing (1), Root (2) - all 100% clean

### Quality Assurance
- **Comprehensive Testing**: All 5 circular import tests now pass (previously 1/5 passing)
- **Regression Prevention**: Test suite integrated for ongoing monitoring
- **Package Health Monitoring**: Automated detection of import issues
- **Development Workflow Restored**: Normal import behavior for all development activities

## [1.0.4] - 2025-08-06

### Fixed
- **DAG Compiler Enhancement** - Fixed template state management in `PipelineDAGCompiler`
  - Added `_last_template` attribute to store template after pipeline generation
  - Fixed timing issues with template metadata population during compilation
  - Added `get_last_template()` method to access fully-populated templates
  - Added `compile_and_fill_execution_doc()` method for proper sequencing of compilation and document filling

### Changed
- **Package Redeployment** - Updated source code and repackaged for PyPI distribution
- **Version Increment** - Incremented version to 1.0.4 for new PyPI release

### Technical Details
- **Template State Management** - Templates now properly retain state after pipeline generation, enabling access to pipeline metadata for execution document generation
- **Execution Document Integration** - New method ensures proper sequencing when both compiling pipelines and filling execution documents
- Rebuilt package with latest source code changes
- Successfully uploaded to PyPI: https://pypi.org/project/cursus/1.0.4/
- All dependencies and metadata validated
- Package available for installation via `pip install cursus==1.0.4`

## [1.0.3] - 2025-08-03

### Fixed
- Fixed import error in processing module `__init__.py` for `MultiClassLabelProcessor` (was incorrectly named `MulticlassLabelProcessor`)
- Corrected class name reference in module exports

## [1.0.2] - 2025-08-03

### Added
- **Processing Module** - New `cursus.processing` module with comprehensive data processing utilities
  - **Base Processor Classes** - `Processor`, `ComposedProcessor`, `IdentityProcessor` for building processing pipelines
  - **Categorical Processing** - `CategoricalLabelProcessor`, `MulticlassLabelProcessor` for label encoding
  - **Numerical Processing** - `NumericalVariableImputationProcessor`, `NumericalBinningProcessor` for data preprocessing
  - **Text/NLP Processing** - `BertTokenizeProcessor`, `GensimTokenizeProcessor` for text tokenization
  - **Domain-Specific Processors** - `BSMProcessor`, `CSProcessor`, `RiskTableProcessor` for specialized use cases
  - **Data Loading Utilities** - `BSMDataLoader`, `BSMDatasets` for data management
  - **Processor Composition** - Support for chaining processors using `>>` operator

### Fixed
- **Import Path Corrections** - Fixed all incorrect import paths in builder_registry.py and related modules
  - Corrected circular import issues using TYPE_CHECKING pattern
  - Fixed imports from non-existent `base_script_contract` to proper `...core.base.contract_base`
  - Updated all contract files to use correct base class imports
  - Resolved dependency resolver import issues in builder_base.py
- **Registry System** - Improved stability of step builder registry initialization
- **Type Safety** - Enhanced type checking with proper runtime placeholders

### Technical Details
- **Processing Pipeline** - Processors can be used in preprocessing, inference, evaluation, and other ML pipeline steps
- **Modular Design** - Each processor is self-contained with clear interfaces and composition support
- **Optional Dependencies** - Graceful handling of optional dependencies for specialized processors
- Fixed 10+ contract files with incorrect import statements
- Implemented TYPE_CHECKING pattern to break circular dependencies
- Added runtime placeholders for optional dependencies
- Corrected relative import paths throughout the registry system

## [1.0.1] - 2025-08-01

### Fixed
- Minor bug fixes and stability improvements
- Documentation updates

## [1.0.0] - 2025-01-31

### Added
- **Initial PyPI Release** - First public release of Cursus
- **Core API** - Main pipeline compilation functionality
  - `compile_dag()` - Simple DAG compilation
  - `compile_dag_to_pipeline()` - Advanced compilation with configuration
  - `PipelineDAGCompiler` - Full-featured compiler class
  - `create_pipeline_from_dag()` - Convenience function for quick pipeline creation

- **Command Line Interface** - Complete CLI for pipeline management
  - `cursus compile` - Compile DAG files to SageMaker pipelines
  - `cursus validate` - Validate DAG structure and compatibility
  - `cursus preview` - Preview compilation results
  - `cursus list-steps` - Show available step types
  - `cursus init` - Generate new projects from templates

- **Core Architecture** - Production-ready pipeline generation system
  - **Pipeline DAG** - Mathematical framework for pipeline topology
  - **Dependency Resolution** - Intelligent matching with semantic compatibility
  - **Step Builders** - Transform specifications into executable SageMaker steps
  - **Configuration Management** - Hierarchical configuration with validation
  - **Registry System** - Component registration and discovery

- **ML Framework Support** - Optional dependencies for different use cases
  - **PyTorch** - PyTorch Lightning models with SageMaker integration
  - **XGBoost** - XGBoost training pipelines with hyperparameter tuning
  - **NLP** - Natural language processing models and utilities
  - **Processing** - Advanced data processing and transformation

- **Template System** - Project scaffolding and examples
  - XGBoost template for tabular data pipelines
  - PyTorch template for deep learning workflows
  - Basic template for simple processing pipelines

- **Quality Assurance** - Enterprise-ready validation and testing
  - Comprehensive error handling and debugging
  - Type-safe specifications with compile-time checks
  - Built-in quality gates and validation rules
  - Production deployment compatibility

### Features
- **🎯 Graph-to-Pipeline Automation** - Transform simple graphs into complete SageMaker pipelines
- **⚡ 10x Faster Development** - Minutes to working pipeline vs. weeks of manual configuration
- **🧠 Intelligent Dependency Resolution** - Automatic step connections and data flow
- **🛡️ Production Ready** - Built-in quality gates, validation, and enterprise governance
- **📈 Proven Results** - 60% average code reduction across pipeline components

### Technical Specifications
- **Python Support** - Python 3.8, 3.9, 3.10, 3.11, 3.12
- **AWS Integration** - Full SageMaker compatibility with boto3 and sagemaker SDK
- **Architecture** - Modular, extensible design with clear separation of concerns
- **Dependencies** - Minimal core dependencies with optional framework extensions
- **Testing** - Comprehensive test suite with unit and integration tests

### Documentation
- Complete API documentation with examples
- Command-line interface reference
- Architecture and design principles
- Developer guide for contributions and extensions
- Ready-to-use pipeline examples and templates

### Performance
- **Code Reduction** - 55% average reduction in pipeline code
- **Development Speed** - 95% reduction in development time
- **Lines Eliminated** - 1,650+ lines of complex SageMaker configuration code
- **Quality Improvement** - Built-in validation prevents common configuration errors

## [Unreleased]

### Planned Features
- **Enhanced Templates** - Additional pipeline templates for common ML patterns
- **Visual DAG Editor** - Web-based interface for visual pipeline construction
- **Advanced Monitoring** - Built-in pipeline monitoring and alerting
- **Multi-Cloud Support** - Extension to other cloud ML platforms
- **Auto-Optimization** - Automatic resource and cost optimization
- **Integration Plugins** - Pre-built integrations with popular ML tools

---

## Release Notes

### Version 1.0.0 - Production Ready

This initial release represents the culmination of extensive development and testing in enterprise environments. Cursus is now production-ready with:

- **98% Complete Implementation** - All core features implemented and tested
- **Enterprise Validation** - Proven in production deployments
- **Comprehensive Documentation** - Complete guides and API reference
- **Quality Assurance** - Extensive testing and validation frameworks

### Migration from Internal Version

If you're migrating from an internal or pre-release version:

1. **Update Imports** - Change from `src.pipeline_api` to `cursus.api`
2. **Install Package** - `pip install cursus[all]` for full functionality
3. **Update Configuration** - Review configuration files for any breaking changes
4. **Test Thoroughly** - Validate all existing DAGs with `cursus validate`

### Getting Started

For new users:

1. **Install** - `pip install cursus`
2. **Generate Project** - `cursus init --template xgboost --name my-project`
3. **Validate** - `cursus validate dags/main.py`
4. **Compile** - `cursus compile dags/main.py --name my-pipeline`

### Support

- **Documentation** - https://github.com/TianpeiLuke/cursus/blob/main/README.md
- **Issues** - https://github.com/TianpeiLuke/cursus/issues
- **Discussions** - https://github.com/TianpeiLuke/cursus/discussions

---

**Cursus v1.0.0** - Making SageMaker pipeline development 10x faster through intelligent automation. 🚀
