# Test Improvement Initiative

> **Status:** Complete — pending merge to `2.x`
> **Branch scope:** 2.x only
> **Branch:** `fix/test-detection-remediation`
> **Date:** 2026-03-12
> **Last updated:** 2026-03-15

---

## 1. Legacy Tests Analysis (`tests/legacy/`)

### Overview

| Metric | Value |
|--------|-------|
| Files | 29 |
| Lines of code | ~6,464 |
| Branch-gating | All files use `IS_2X_BRANCH` skip marker |

### What's in there

- **Integration tests (13 files):** Auto-create target branch, status routing, feature lifecycle, task workflows, missions, research templates. Largest: `test_move_task_delegation.py` (~900 LOC).
- **specify_cli tests (4 files):** Review warnings, workflow auto-moves, event emission, init command.
- **Unit tests (4 files):** Mission schema (1.x format), task commands, workflow instructions, research missions.

### Module coverage

All 12 imported modules (`agent.feature`, `agent.tasks`, `agent.workflow`, `status.migrate`, `status.store`, `status.reducer`, `mission`, `frontmatter`, `tasks_support`, `core.vcs`, `core.context_validation`, `acceptance`) still exist in the 2.x codebase.

### Filename conflicts with current tests

| File | Locations | Risk |
|------|-----------|------|
| `test_event_emission.py` | 3 locations | LOW (branch-gated) |
| `test_mission_schema.py` | Legacy (1.x format) vs current | LOW |
| `test_mission_switching.py` | Legacy stub vs current full | LOW |
| `test_review_warnings.py` | Legacy references current | LOW |

### Verdict

**Safe to keep ignored.** All coverage is either duplicated or superseded by 2.x tests. The branch-gating works correctly. No dead code, no salvageable unique coverage.

**Recommendation:** Keep as-is. Consider renaming files with `_legacy` suffix to avoid IDE confusion if it becomes annoying.

---

## 2. Final Test Suite Profile (post-migration)

### Metrics

| Metric | Value |
|--------|-------|
| Test files (non-legacy) | 341 |
| Total test cases | 5,608 |
| conftest.py files | 8 |
| Full suite (fast+slow+unmarked) | ~5 min (local), ~20 min (CI) |

### Directory breakdown (vertical-slice layout)

| Directory | Type | Notes |
|-----------|------|-------|
| `missions/` | Unit + integration | Mission lifecycle, guards, schema |
| `merge/` | Unit + integration | Merge plans, preflight, multi-parent |
| `agent/` | Unit + integration | Context, validators, review, orchestrator API |
| `tasks/` | Unit + integration | Frontmatter, lane ops, planning workflow |
| `git_ops/` | Unit + integration | Branch tracking, stale detection, safe commit |
| `status/` | Unit + integration | Resolver, E2E status routing |
| `upgrade/` | Unit + integration | Migrations, version update |
| `init/` | Integration | Charter runtime, init flow |
| `sync/` | Async unit | Dual-write, staleness detection |
| `runtime/` | Unit + integration | Bootstrap, doctor, paths, show-origin |
| `research/` | Unit + integration | Deliverables, plan missions, workflow |
| `next/` | Unit + integration | Decision, prompt builder, runtime bridge |
| `cross_cutting/` | Mixed | Dashboard, encoding, packaging, versioning, misc |
| `doctrine/` | Unit | Schema compliance |
| `adversarial/` | Security | Attack/fuzzing tests |
| `e2e/` | E2E | CLI workflow |
| `docs/` | Unit | Doc integrity |
| `release/` | Low-cost | Release validation |
| `concurrency/` | Mixed | Concurrency tests |
| `cross_branch/` | Integration | Branch compatibility |
| `contract/` | Integration | Handoff verification |
| `charter/` | Unit | Charter compliance |
| `legacy/` | Gated | 0 items collected on `2.x`; immutable snapshot |

### Marker summary

| Marker | Tests | Runtime (local) |
|--------|-------|-----------------|
| `fast` | ~1,658 | ~5s |
| `git_repo` | ~898 | ~varies |
| `slow` | ~684 | ~26s |
| `legacy` | 0 (gated) | — |
| unmarked | ~2,368 | ~4 min |
| **Total** | **5,608** | **~5 min** |

---

## 3. Work Completed

### 3.1 Pytest Marker Annotations (commit `a1ad4367`)

Added `fast` and `slow` pytest markers to all profiled `specify_cli/` subdirectories using `pytest_collection_modifyitems` hooks in conftest.py files. This enables `pytest -m fast` for rapid dev-loop feedback.

**Marker assignment (based on measured timing):**

| Directory | Tests | Time | Marker |
|-----------|-------|------|--------|
| `status/` | 632 | 1.25s | `fast` |
| `glossary/` | 689 | 0.64s | `fast` |
| `upgrade/` | 168 | 1.73s | `fast` |
| `charter/` | 114 | 0.62s | `fast` |
| `cli/` | 190 | 12.88s | `slow` |
| `core/` | 153+57skip | 16.05s | `slow` |
| `test_cli/` | 101 | 29.42s | `slow` |
| `test_core/` | 120 | 25.04s | `slow` |

**Implementation note:** `pytestmark` in conftest.py does NOT propagate in pytest 9.0.2. Used `pytest_collection_modifyitems` hook scoped with `Path(__file__).parent` instead.

**Current marker stats (2026-03-14):**

| Marker | Tests | Runtime (local) | Notes |
|--------|-------|-----------------|-------|
| `fast` | 1,659 | 5.2s | Pure logic, no subprocess/git |
| `slow` | 684 | 25.9s | subprocess/git, specify_cli/test_cli + test_core |
| unmarked | 2,992 | ~4 min | integration, sync, legacy, root, e2e |
| **Total** | **5,335** | **~5 min** | Local machine ~3× faster than CI reference |

> **Note:** The reference machine times in the original spec (73.51s for slow) were measured on a slower CI-equivalent machine. Local timings are ~3× faster. Use the reference times for CI budgeting.

### 3.2 Merge Test Unit Extraction (commit `c816d247`)

**Target:** `tests/specify_cli/test_cli/test_merge_workspace_per_wp.py` — the single slowest file (101 tests, 29.42s). Every test created real git repos + worktrees (1.3–1.5s fixture cost per test).

**Action:** Extracted pure-logic and mock-boundary tests into `test_merge_workspace_per_wp_unit.py`. Removed 13 integration tests now covered by unit mocks.

| File | Tests | Runtime | Change |
|------|-------|---------|--------|
| `test_merge_workspace_per_wp.py` (before) | 101 | 29.42s | — |
| `test_merge_workspace_per_wp.py` (after) | 17 + 3 xfail | 23.64s | −6s |
| `test_merge_workspace_per_wp_unit.py` (new) | 29 | 0.33s | — |
| **Net** | **49** | **23.97s** | **−5.5s, +29 fast tests** |

**Unit test coverage (29 tests, 0.33s):**

| Function | Unit tests | Approach |
|----------|-----------|----------|
| `extract_feature_slug` | 3 | Pure string parsing |
| `extract_wp_id` | 3 | Pure Path parsing |
| `detect_worktree_structure` | 7 | Mocked `get_main_repo_root` + `_list_wp_branches` |
| `find_wp_worktrees` | 4 | Mocked filesystem + branch listing |
| `validate_wp_ready_for_merge` | 4 | Mocked `subprocess.run` |
| `_build_workspace_per_wp_merge_plan` | 4 | Mocked `_branch_is_ancestor` + `_order_wp_workspaces` |
| `merge_workspace_per_wp` (dry-run) | 2 | Fully mocked internals |
| VCS detection | 2 | Mocked availability checks |

**Integration tests kept (17):** Tests requiring real git repos — worktree detection from within worktrees, full merge workflows, ancestry chain planning.

### 3.3 test_core/ Unit Extraction (commits `3ed22b5a`, `b402d17f`)

Extracted pure-logic and mock-boundary tests from the two most expensive `test_core/` files.

**`test_git_ops.py` → `test_git_ops_unit.py` (17 new fast tests, 0.29s):**

| Function | Unit tests | Approach |
|----------|-----------|----------|
| `run_command` | 3 | Pure subprocess mock |
| `resolve_target_branch` | 8 | Mocked `resolve_primary_branch` |
| `resolve_primary_branch` | 6 | Mocked `subprocess.run` heuristics |

**`test_create_feature_branch.py` → `test_create_feature_branch_unit.py` (9 new fast tests, 0.57s):**

| Function | Unit tests | Approach |
|----------|-----------|----------|
| `create_feature` | 9 | Mocked `locate_project_root`, `is_git_repo`, `is_worktree_context`, `get_current_branch`, `get_next_feature_number`, `safe_commit` |

**Discovery:** `create_feature()` calls `safe_commit()` after writing `meta.json` — mocking git I/O requires patching `specify_cli.cli.commands.agent.feature.safe_commit` in addition to the root/branch guard mocks.

---

### 3.4 Fixture Consolidation, Lint + Type Cleanup (2026-03-14)

#### Fixture consolidation (commit `969b5746`)

- Moved `isolated_env` and `run_cli` from `tests/e2e/conftest.py` and `tests/integration/conftest.py` to root `tests/conftest.py`
- Restored accidentally-removed `temp_repo` fixture in root conftest
- Re-added missing `import tomllib` to `tests/integration/conftest.py`
- Removed 9 stale `@pytest.mark.xfail` markers across 4 files (`test_distribution.py`, `test_implement_multi_parent.py`, `test_merge_workflow_complete.py`, `test_multi_parent_merge.py`) — all now pass unconditionally

#### Ruff lint sweep (commit `e7a3781c`)

Applied `ruff check --fix` and `--unsafe-fixes` across the full `tests/` tree. **687 violations → 0.**

Key rules resolved: F401 (249 unused imports), UP017/UP006/UP035 (deprecated typing), SIM117 (51 nested `with`), W293 (trailing whitespace), F841 (60 unused variables), E501 (38 long lines), E741 (ambiguous names), E402 (imports not at top), F811 (redefined names), SIM105/SIM102 (simplifications), E722 (bare except), B018/SIM115 (misc).

#### Mypy type annotations (commit `a9d77b3d`)

Added strict-mode type annotations to all test files authored during this initiative:
- `test_git_ops_unit.py`, `test_create_feature_branch_unit.py`: `-> None` on all test functions, typed helper closures
- `tests/conftest.py`: `-> None` on `ensure_imports`, corrected `git_stale_workspace` return type to `dict[str, Path | str]`
- `tests/adversarial/test_distribution.py`: `-> None` on all test methods, typed `tmp_path_factory` parameters

---

## 4. Test Structure Redesign — Vertical Slices

**Status:** Complete (all directories migrated) ✓  
**ADR:** `architecture/2.x/adr/2026-03-15-1-vertical-slice-test-organisation.md`

### Problem

The test tree has two competing top-level axes that are never reconciled:

- `unit/` and `integration/` are organised by **test type**, but their internals are flat bags of files with no relation to system capabilities. You cannot look at either directory and understand what the system does.
- `specify_cli/` mirrors the **source package tree**, but mixes unit and integration tests indiscriminately. Test type is invisible.

Neither axis is consistently applied. Neither gives a readable picture of the system.

### Decision

**Top level = vertical slices (system capabilities).** Test type is expressed orthogonally via pytest markers (`fast`, `git_repo`, `slow`) and filename suffixes (`*_unit.py` / `*_integration.py`).

Target structure:

```
tests/
  missions/       merge/        agent/        tasks/
  git_ops/        status/       upgrade/      init/
  sync/           runtime/      research/     next/
  cross_cutting/  doctrine/     adversarial/  e2e/
  docs/           release/      legacy/
```

`tests/README.md` has been rewritten to document this intent, the desiderata, and
the conventions for writing new tests.

### Stale work-tracking artefacts

The following files were written during earlier ad-hoc test sprints. They describe
work that is now complete or superseded. They have been moved here for archival:

- `IMPLEMENTATION_COMPLETE.md` — encoding & plan validation test suite completion report (Nov 2025)
- `TESTING_PROGRESS.md` — per-suite progress tracker for the same sprint
- `TESTING_REQUIREMENTS_ENCODING_AND_PLAN_VALIDATION.md` — original requirements spec for that sprint

### Migration completed (2026-03-15)

| Source | Target slice(s) | Status |
|--------|----------------|--------|
| `unit/` (54 files) | Various slices | ✓ Done — `_unit` suffix, git mv |
| `integration/` (39 files) | Various slices | ✓ Done — `_integration` suffix, `pytestmark` added |
| `specify_cli/` (141 files) | Various slices | ✓ Done — migrated in final phase |
| Empty shell dirs (`unit/`, `integration/`, `specify_cli/`) | — | ✓ Removed |

Fixtures `test_project`, `clean_project`, `dirty_project`, `project_with_worktree`,
`dual_branch_repo` were promoted from `integration/conftest.py` to the root
`tests/conftest.py` so they are visible across all slice directories.

**Final state (2026-03-15):**
- Collection: **5,608 tests**, 0 errors (up from 5,335 before this initiative)
- `fast` suite: ~1,658 passed, 1 skipped
- `git_repo` suite: ~898 passed — 10 pre-existing failures unchanged
- All three legacy top-level directories (`unit/`, `integration/`, `specify_cli/`) removed
- `tests/legacy/` formalised: `conftest.py` added, `legacy` marker registered, `IS_2X_BRANCH` gating confirmed working
- 26 new unit tests added across 3 new files (easy-wins coverage pass)
- CI redesigned: fast-tests / integration-tests / slow-and-e2e job topology

**Per-file slice classification for `unit/` and `integration/`:**

| File | Slice |
|------|-------|
| `unit/test_agent_config.py` | `agent/` |
| `unit/test_atomic_status_commits.py` | `git_ops/` |
| `unit/test_base_branch_tracking.py` | `git_ops/` |
| `unit/test_branch_contract.py` | `git_ops/` |
| `unit/test_context_validation.py` | `agent/` |
| `unit/test_doctrine_curation.py` | `cross_cutting/` (doctrine) |
| `unit/test_dossier_timezone_defaults.py` | `sync/` |
| `unit/test_gitignore_manager.py` | `cross_cutting/` |
| `unit/test_guards.py` | `missions/` |
| `unit/test_implement_merged_deps.py` | `merge/` |
| `unit/test_lane_directory_removal.py` | `tasks/` |
| `unit/test_m_0_12_0_documentation_mission.py` | `upgrade/` |
| `unit/test_merge_forecast.py` | `merge/` |
| `unit/test_merge_preflight.py` | `merge/` |
| `unit/test_merge_state.py` | `merge/` |
| `unit/test_migration_charter_cleanup.py` | `upgrade/` |
| `unit/test_migration_python_only.py` | `upgrade/` |
| `unit/test_mission_schema.py` | `missions/` |
| `unit/test_move_task_git_validation.py` | `tasks/` |
| `unit/test_multi_parent_merge.py` | `merge/` |
| `unit/test_multi_parent_merge_adversarial.py` | `merge/` |
| `unit/test_multi_parent_merge_empty_branches.py` | `merge/` |
| `unit/test_paths.py` | `runtime/` |
| `unit/test_pre_commit_wp_guard.py` | `tasks/` |
| `unit/test_research_deliverables.py` | `research/` |
| `unit/test_stale_detection.py` | `git_ops/` |
| `unit/test_status_resolver.py` | `status/` |
| `unit/test_upgrade_auto_commit.py` | `upgrade/` |
| `unit/test_validators.py` | `agent/` |
| `unit/test_workspace_context.py` | `runtime/` |
| `unit/agent/test_context.py` | `agent/` |
| `unit/agent/test_context_resolve.py` | `agent/` |
| `unit/agent/test_feature_lifecycle.py` | `missions/` |
| `unit/agent/test_git_state_detection.py` | `git_ops/` |
| `unit/agent/test_review_feedback_pointer_2x.py` | `agent/` |
| `unit/agent/test_review_validation.py` | `agent/` |
| `unit/agent/test_tasks_2x.py` | `tasks/` |
| `unit/agent/test_workflow_feedback_pointer_2x.py` | `agent/` |
| `unit/mission_v1/test_compat.py` | `missions/` |
| `unit/mission_v1/test_events.py` | `missions/` |
| `unit/mission_v1/test_guards.py` | `missions/` |
| `unit/mission_v1/test_runner.py` | `missions/` |
| `unit/mission_v1/test_schema.py` | `missions/` |
| `unit/next/test_decision.py` | `next/` |
| `unit/next/test_prompt_builder.py` | `next/` |
| `unit/next/test_runtime_bridge.py` | `next/` |
| `unit/orchestrator_api/test_envelope.py` | `agent/` |
| `unit/runtime/test_bootstrap.py` | `runtime/` |
| `unit/runtime/test_doctor.py` | `runtime/` |
| `unit/runtime/test_global_runtime_convergence.py` | `runtime/` |
| `unit/runtime/test_home.py` | `runtime/` |
| `unit/runtime/test_merge.py` | `merge/` |
| `unit/runtime/test_resolver.py` | `runtime/` |
| `unit/runtime/test_show_origin.py` | `runtime/` |
| `integration/test_auto_merge_dependencies.py` | `merge/` |
| `integration/test_bug_124_branch_routing.py` | `git_ops/` |
| `integration/test_config_show_origin.py` | `runtime/` |
| `integration/test_conflict_resolution.py` | `merge/` |
| `integration/test_charter_runtime.py` | `init/` |
| `integration/test_dual_write.py` | `sync/` |
| `integration/test_e2e_mission_v1.py` | `missions/` |
| `integration/test_e2e_runtime.py` | `runtime/` |
| `integration/test_gitignore_isolation.py` | `cross_cutting/` |
| `integration/test_implement_multi_parent.py` | `merge/` |
| `integration/test_init_flow.py` | `init/` |
| `integration/test_init_minimal.py` | `init/` |
| `integration/test_merge_no_remote.py` | `merge/` |
| `integration/test_merge_workflow_complete.py` | `merge/` |
| `integration/test_merged_dependency_workflow.py` | `merge/` |
| `integration/test_migrate.py` | `upgrade/` |
| `integration/test_migration_e2e.py` | `upgrade/` |
| `integration/test_mission_guards.py` | `missions/` |
| `integration/test_mission_loading.py` | `missions/` |
| `integration/test_mission_software_dev.py` | `missions/` |
| `integration/test_mission_switching.py` | `missions/` |
| `integration/test_next_command.py` | `next/` |
| `integration/test_next_replay_parity.py` | `next/` |
| `integration/test_planning_workflow.py` | `tasks/` |
| `integration/test_read_cutover.py` | `upgrade/` |
| `integration/test_research_plan_missions.py` | `research/` |
| `integration/test_research_workflow.py` | `research/` |
| `integration/test_safe_commit_helper.py` | `git_ops/` |
| `integration/test_specify_metadata_explicit.py` | `missions/` |
| `integration/test_status_e2e.py` | `status/` |
| `integration/test_sync_e2e.py` | `sync/` |
| `integration/test_sync_staleness.py` | `git_ops/` |
| `integration/test_two_branch_strategy.py` | `git_ops/` |
| `integration/test_version_isolation.py` | `cross_cutting/` |
| `integration/test_workspace_per_wp_workflow.py` | `tasks/` |
| `integration/test_worktree_exclusion.py` | `git_ops/` |
| `integration/orchestrator_api/test_commands.py` | `agent/` |
| `integration/orchestrator_api/test_json_envelope_contract.py` | `agent/` |
| `integration/test_agent_command_wrappers.py` | `agent/` |

### Legacy directory (complete)

`tests/legacy/` is formalised as an immutable frozen snapshot:
- `tests/legacy/conftest.py` added: auto-applies `legacy` + `slow` markers to all files; documents deprecation roadmap
- `legacy` marker registered in `pyproject.toml`
- `pytest_ignore_collect` in root `conftest.py` ensures 0 items collected on `2.x`
- Files are branch-gated with `IS_2X_BRANCH` skip markers (unchanged)
- No test files in `legacy/` were modified — purely additive infrastructure
