Metadata-Version: 2.4
Name: cli-test-framework
Version: 0.8.0
Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
Home-page: https://github.com/ozil111/cli-test-framework
Author: Xiaotong Wang
Author-email: xiaotongwang98@gmail.com
Project-URL: Documentation, https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md
Project-URL: Source, https://github.com/ozil111/cli-test-framework
Project-URL: Tracker, https://github.com/ozil111/cli-test-framework/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: dukpy==0.5.0
Requires-Dist: h5py<4.0.0,>=3.11.0
Requires-Dist: numpy>=1.21.0; python_version < "3.12"
Requires-Dist: numpy>=1.26.0; python_version >= "3.12"
Requires-Dist: setuptools>=75.8.0
Requires-Dist: wheel>=0.45.1
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# CLI Test Framework

A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.

Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.

## Why this exists

This project was created from a real need: regression testing finite-element software.

In solver-style projects, checking the exit code is not enough. A test often needs to run multiple commands, generate numerical result files, compare HDF5/CSV outputs with tolerances, and make sure old cases do not become slower over time.

CLI Test Framework was built for that workflow.

## Highlights

- **Golden File Assertion** — `compare_files` embedded in test `expected`, compares output files against baselines with tolerance
- **Parallel Execution** — Multi-thread / multi-process, 3-5x speedup
- **Resource-Aware Scheduling** — Automatic CPU core management, prevents solver thread runaway
- **Sequence Steps** — Multi-step execution within a single test case, fail-fast
- **Setup Module** — Auto-configure environment variables before tests, auto-cleanup after
- **File Comparison** — Text / JSON / CSV / XML / HDF5 / Binary, with CLI and embedded assertion support
- **Filtered Execution** — Run specific test cases by name
- **JUnit XML Output** — `--junit-xml` for GitLab CI / Jenkins / CircleCI test report panels

## Quick Start

```bash
pip install cli-test-framework
```

### 30-Second Setup

1. Create `test_cases.json`:

```json
{
    "test_cases": [
        {
            "name": "hello",
            "command": "echo",
            "args": ["Hello World"],
            "expected": {
                "return_code": 0,
                "output_contains": ["Hello World"]
            }
        }
    ]
}
```

2. Run:

```bash
cli-test run test_cases.json
```

### Golden File Comparison in Tests

Run a simulation, then compare its output file against a reference:

```json
{
    "test_cases": [
        {
            "name": "FEA displacement check",
            "command": "my_solver",
            "args": ["--input", "case1.dat", "--output", "out.h5"],
            "expected": {
                "return_code": 0,
                "compare_files": [
                    {
                        "actual": "out.h5",
                        "baseline": "ref/golden.h5",
                        "rtol": 1e-5,
                        "atol": 1e-8,
                        "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
                    }
                ]
            }
        }
    ]
}
```

- `actual` – file produced by the command
- `baseline` – reference file to compare against
- `type` – comparator type (auto-detected from extension if omitted: `.h5`→h5, `.json`→json, `.csv`→csv, `.xml`→xml, `.txt`→text)
- all other keys are forwarded as comparator parameters (`rtol`, `atol`, `tables`, `table_regex`, `data_filter`, `encoding`, `structure_only`, `delimiter`, `compare_mode`, `key_field`, etc.)

Multiple files and mixed assertion types coexist naturally:

```json
{
    "expected": {
        "return_code": 0,
        "output_contains": ["simulation finished"],
        "compare_files": [
            {"actual": "out.h5", "baseline": "ref/disp.h5", "rtol": 1e-5},
            {"actual": "report.csv", "baseline": "ref/expected.csv", "rtol": 1e-6}
        ]
    }
}
```

### Parallel Execution

```bash
cli-test run test_cases.json --parallel --workers 4
```

### JUnit XML (CI Integration)

```bash
cli-test run test_cases.json --junit-xml report.xml
```

In GitLab CI:

```yaml
# .gitlab-ci.yml
artifacts:
  reports:
    junit: report.xml
```

### Python API

```python
from cli_test_framework.runners import JSONRunner, ParallelJSONRunner

# Sequential
runner = JSONRunner(config_file="test_cases.json")
success = runner.run_tests()

# Parallel
runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
success = runner.run_tests()
```

### File Comparison (Standalone CLI)

```bash
compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
```

📖 **Full Documentation**: [docs/user_manual.md](docs/user_manual_en.md)

## Changelog

### 0.8.0

- **JUnit XML output**: New `--junit-xml <filepath>` CLI option writes JUnit-format XML reports that integrate directly with GitLab CI, Jenkins, CircleCI, and other CI tools. Supports `passed` / `failed` / `timeout` / `error` status mapping. Zero extra dependencies — built with `xml.etree.ElementTree`.

### 0.7.0

- **Unified logging system**: All diagnostic output (executor, runner, scheduler, setup) now uses Python's standard `logging` module under the `cli_test_framework` namespace. Library users can suppress output entirely via `logging.getLogger("cli_test_framework").setLevel(logging.WARNING)`. Removes the previous ad-hoc `print()` + `_print_lock` pattern — `logging` is inherently thread-safe.
- **Default handler**: A `StreamHandler` at INFO level is installed on first import, so CLI behavior is unchanged. Use `--verbose` / `--debug` to enable DEBUG-level output.
- **Public API**: `get_logger(name)` exposed via `cli_test_framework.get_logger` for consistent logger creation in extensions.

### 0.6.0

- **Golden file assertion**: `compare_files` is now a first-class assertion in test `expected` — compare output files against baselines directly in your test definitions, with full tolerance and parameter support. The `file_comparator` subsystem is now integrated into the assertion pipeline, completing the closed loop from command execution to result verification.

### 0.5.2

- Runtime history tracking (`--history-dir`): persist per-case execution time in `.symtest`, enable smart scheduling & regression detection
- Regression warning: alert when a case runs slower than historical average × threshold (`--regression-threshold`, default 1.5)
- Smart scheduling: parallel runner prioritizes historical `avg_duration` over config `estimated_time` for task ordering
- Per-case duration now shown in test result output

### 0.5.1
- Run specific test cases by name (`-t` / `test_case_filter`)

### 0.5.0
- Steps feature: sequential multi-command execution within a single test case, fail-fast

### 0.4.2
- Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
- Auto-inject `OMP_NUM_THREADS` / `MKL_NUM_THREADS` / `NPROC` to prevent solver thread runaway
- Per-test `timeout` support to prevent hanging

### 0.4.1
- Multi-thread / multi-process parallel execution, 3-5x speedup

## Contributing

Before submitting a PR, please make sure all tests pass:

```bash
python tests\run_all.py
```

## License

MIT
