Metadata-Version: 2.4
Name: cli-test-framework
Version: 0.5.2
Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
Home-page: https://github.com/ozil111/cli-test-framework
Author: Xiaotong Wang
Author-email: xiaotongwang98@gmail.com
Project-URL: Documentation, https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md
Project-URL: Source, https://github.com/ozil111/cli-test-framework
Project-URL: Tracker, https://github.com/ozil111/cli-test-framework/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: dukpy==0.5.0
Requires-Dist: h5py<4.0.0,>=3.8.0; python_version < "3.12"
Requires-Dist: h5py<4.0.0,>=3.10.0; python_version >= "3.12"
Requires-Dist: numpy<2.0.0,>=1.21.0; python_version < "3.12"
Requires-Dist: numpy<2.0.0,>=1.26.0; python_version >= "3.12"
Requires-Dist: setuptools>=75.8.0
Requires-Dist: wheel>=0.45.1
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# CLI Test Framework

A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.

Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.

## Highlights

- **Parallel Execution** — Multi-thread / multi-process, 3-5x speedup
- **Resource-Aware Scheduling** — Automatic CPU core management, prevents solver thread runaway
- **Sequence Steps** — Multi-step execution within a single test case, fail-fast
- **Setup Module** — Auto-configure environment variables before tests, auto-cleanup after
- **File Comparison** — Text / JSON / HDF5 / Binary, ready to use from command line
- **Filtered Execution** — Run specific test cases by name

## Quick Start

```bash
pip install cli-test-framework
```

### 30-Second Setup

1. Create `test_cases.json`:

```json
{
    "test_cases": [
        {
            "name": "hello",
            "command": "echo",
            "args": ["Hello World"],
            "expected": {
                "return_code": 0,
                "output_contains": ["Hello World"]
            }
        }
    ]
}
```

2. Run:

```bash
cli-test run test_cases.json
```

### Parallel Execution

```bash
cli-test run test_cases.json --parallel --workers 4
```

### Python API

```python
from cli_test_framework.runners import JSONRunner, ParallelJSONRunner

# Sequential
runner = JSONRunner(config_file="test_cases.json")
success = runner.run_tests()

# Parallel
runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
success = runner.run_tests()
```

### File Comparison

```bash
compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
```

📖 **Full Documentation**: [docs/user_manual.md](docs/user_manual_en.md)

## Changelog

### 0.5.2

- Runtime history tracking (`--history-dir`): persist per-case execution time in `.symtest`, enable smart scheduling & regression detection
- Regression warning: alert when a case runs slower than historical average × threshold (`--regression-threshold`, default 1.5)
- Smart scheduling: parallel runner prioritizes historical `avg_duration` over config `estimated_time` for task ordering
- Per-case duration now shown in test result output

### 0.5.1
- Run specific test cases by name (`-t` / `test_case_filter`)

### 0.5.0
- Steps feature: sequential multi-command execution within a single test case, fail-fast

### 0.4.2
- Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
- Auto-inject `OMP_NUM_THREADS` / `MKL_NUM_THREADS` / `NPROC` to prevent solver thread runaway
- Per-test `timeout` support to prevent hanging

### 0.4.1
- Multi-thread / multi-process parallel execution, 3-5x speedup

## Contributing

Before submitting a PR, please make sure all tests pass:

```bash
python tests\run_all.py
```

## License

MIT
