Metadata-Version: 2.4
Name: harness-test
Version: 0.1.0
Summary: A CLI + TUI testing framework for Claude Code harness components
Project-URL: Homepage, https://github.com/hgflima/harness-test
Project-URL: Repository, https://github.com/hgflima/harness-test
Project-URL: Issues, https://github.com/hgflima/harness-test/issues
Author-email: Henrique Lima <hgflima@gmail.com>
License-Expression: MIT
Keywords: agent-sdk,claude-code,cli,harness,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: claude-agent-sdk>=0.1.55
Requires-Dist: pydantic-settings>=2.7
Requires-Dist: pydantic>=2.12
Requires-Dist: rich>=14.0
Requires-Dist: ruamel-yaml>=0.19
Requires-Dist: typer>=0.24
Description-Content-Type: text/markdown

<div align="center">

# Harness Test

[![Python 3.12+](https://img.shields.io/badge/python-3.12+-3776AB?logo=python&logoColor=white)](https://python.org)
[![Typer](https://img.shields.io/badge/CLI-Typer-009688?logo=fastapi&logoColor=white)](https://typer.tiangolo.com)
[![Pydantic v2](https://img.shields.io/badge/models-Pydantic_v2-E92063?logo=pydantic&logoColor=white)](https://docs.pydantic.dev)
[![Rich](https://img.shields.io/badge/output-Rich-F9A825)](https://rich.readthedocs.io)
[![Claude Agent SDK](https://img.shields.io/badge/executor-Claude_Agent_SDK-D97757?logo=anthropic&logoColor=white)](https://docs.anthropic.com)
[![uv](https://img.shields.io/badge/pkg-uv-DE5FE9?logo=uv&logoColor=white)](https://docs.astral.sh/uv/)

**Test your Claude Code harness the same way you test code — run a command, get a report, know what works.**

</div>

---

## About

Harness Test is a CLI testing framework for **Claude Code harness components** — CLAUDE.md, skills, hooks, subagents, and AGENTS.md.

It provides three testing layers with increasing depth and cost:

| Layer | What it does | Token cost |
|-------|-------------|------------|
| **Static validation** | Syntax, structure, broken references, rule conflicts | Zero |
| **Behavioral testing** | Runs prompts against Agent SDK, evaluates assertions | Per-test |
| **Statistical evaluation** | Multi-run consistency and variance analysis | Multi-run |

## Tech Stack

| Category | Technology |
|----------|-----------|
| **Language** | Python 3.12+ |
| **CLI** | Typer |
| **Data Models** | Pydantic v2 |
| **Terminal Output** | Rich (tables, Live streaming) |
| **YAML Parsing** | ruamel.yaml (round-trip fidelity) |
| **Test Executor** | Claude Agent SDK |
| **Package Manager** | uv |
| **Linter / Formatter** | Ruff |
| **Type Checker** | Pyright |
| **Tests** | pytest + pytest-asyncio |

## Architecture

```
┌─────────────────────────────────────────────────────┐
│                     CLI (Typer)                      │
│              init  ·  run  ·  report                 │
├──────────┬──────────┬──────────────┬────────────────┤
│ Scanner  │Validator │   Runner     │   Reporter     │
│          │          │              │                │
│ claude_md│ claude_md│  executor    │  formatter     │
│ skills   │ skills   │  (Agent SDK) │  (Rich tables) │
│ hooks    │ hooks    │  assertions  │                │
│ subagents│ subagents│  collector   │                │
│ agents_md│references│              │                │
│ memory   │ rules    │              │                │
├──────────┴──────────┴──────────────┴────────────────┤
│                 Pydantic Models                      │
│   component · config · test_spec · results · valid. │
└─────────────────────────────────────────────────────┘
```

**Data Flow:**

```
init:     scan_project() → detect_auth() → write_config()
static:   load_config() → scan_project() → validate_all() → Rich table
behavior: load_config() → load_all_specs() → run_tests() → save_results() → Rich Live
report:   load_results() → scan_project() → render_report() → Rich tables / JSON
```

## Project Structure

```
src/harness_test/
  cli.py            # All CLI commands (init, run, report)
  config.py         # Auth detection + config management
  spec_loader.py    # YAML test spec loader
  main.py           # Typer app entry point
  models/           # Pydantic models (component, config, test_spec, results, validation)
  scanner/          # One scanner per harness type + scan_project() orchestrator
  validator/        # One validator per concern + validate_all() orchestrator
  runner/           # Agent SDK executor, assertions, result collector, run_tests()
  reporter/         # Rich formatter + render_report()
tests/              # Mirrors src/ structure
.harness-test/      # Runtime config (config.yaml, test results)
```

## Getting Started

### Prerequisites

- **Python 3.12+**
- **[uv](https://docs.astral.sh/uv/)** package manager
- **Claude Code subscription** (for behavioral tests) or **Anthropic API key**

### Installation

```bash
git clone https://github.com/hgflima/harness-test.git
cd harness-test
uv sync
```

### Usage

#### 1. Initialize

Discover harness components, detect auth method, and create config:

```bash
uv run harness-test init
```

#### 2. Run Tests

```bash
# Static validation (zero tokens)
uv run harness-test run --layer static

# Behavioral tests via Agent SDK
uv run harness-test run

# Re-run only failed tests
uv run harness-test run --failed
```

#### 3. View Report

```bash
# Rich terminal output
uv run harness-test report

# JSON for CI pipelines
uv run harness-test report --json
```

### Development

```bash
uv run pytest                     # Run test suite (218 tests)
uv run ruff check src/ tests/     # Lint
uv run ruff format src/ tests/    # Format
```

## Key Design Decisions

- **Pydantic everywhere** — All data crosses module boundaries as Pydantic models, never raw dicts
- **Scanner/Validator symmetry** — Each harness component type has a matched scanner + validator pair
- **ruamel.yaml over PyYAML** — Round-trip fidelity for YAML specs
- **Executor isolation** — Each behavioral test runs in its own Agent SDK session
- **Exit codes** — `0` success, `1` failures/errors, `2` runtime/config errors

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Run tests and linting (`uv run pytest && uv run ruff check src/ tests/`)
4. Commit your changes
5. Open a Pull Request

## Acknowledgements

- [Claude Code](https://claude.ai/claude-code) by Anthropic
- [Claude Agent SDK](https://docs.anthropic.com) for behavioral test execution
- [Typer](https://typer.tiangolo.com) for CLI ergonomics
- [Rich](https://rich.readthedocs.io) for terminal output
- [Pydantic](https://docs.pydantic.dev) for data validation

---

<div align="center">

If this project helps you test your Claude Code harness, consider giving it a star.

</div>
