Metadata-Version: 2.4
Name: mlstatreport
Version: 0.1.2
Summary: Statistical reporting engine for machine-learning benchmark experiments.
Author: MLStatReport contributors
Maintainer: MLStatReport contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/jmartinezot/MLStatReport
Project-URL: Documentation, https://github.com/jmartinezot/MLStatReport#readme
Project-URL: Issues, https://github.com/jmartinezot/MLStatReport/issues
Project-URL: Source, https://github.com/jmartinezot/MLStatReport
Keywords: benchmarking,critical-difference,machine learning,nonparametric statistics,reporting,statistics
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<2.3,>=1.26
Requires-Dist: scipy<1.16,>=1.11
Requires-Dist: pandas<2.4,>=2.1
Requires-Dist: matplotlib<3.11,>=3.8
Requires-Dist: openpyxl>=3.1
Requires-Dist: jinja2>=3.1
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.27
Requires-Dist: typer>=0.12
Requires-Dist: pydantic>=2.6
Requires-Dist: python-multipart>=0.0.9
Provides-Extra: dev
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: httpx2>=2.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "dev"
Requires-Dist: twine>=5.1; extra == "dev"
Dynamic: license-file

# MLStatReport

MLStatReport turns machine-learning benchmark tables into traceable statistical
reports. It provides a reusable Python library, a command-line interface, and a
REST API that all execute the same validated analysis workflow.

The current release provides a focused frequentist workflow for multiple
algorithms evaluated on multiple datasets. It produces deterministic JSON,
CSV, Markdown, LaTeX, figures, a local bibliography, and an optional PDF when
a LaTeX compiler is available.

## Installation

MLStatReport requires Python 3.10 or newer.

```bash
python -m pip install mlstatreport
```

For a development checkout:

```bash
python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
```

For a runtime-only installation, use `python -m pip install .` instead.

## Dev Container

The recommended development environment is Visual Studio Code Dev Containers.

1. Open the repository in VS Code.
2. Run **Dev Containers: Reopen in Container**.
3. In the container terminal, run `make install`.

The container uses `bash` with Python 3.12 as the default development
interpreter and includes the tools needed for formatting, type checking, tests,
and the API service.

## CLI quickstart

The example table is a wide benchmark matrix: one dataset column followed by
one score column per algorithm.

```bash
mlstatreport --version

mlstatreport analyze examples/results.csv \
    --control Proposed \
    --metric accuracy \
    --higher-is-better \
    --posthoc holm \
    --out report/
```

Use `mlstatreport analyze --help` to see supported input formats and all
options. Invalid configurations, such as `--posthoc holm` without `--control`,
produce clear validation errors and do not write a report.

## Expected output

The command creates a self-contained report directory:

```text
report/
├── analysis.json
├── bibliography.bib
├── input_summary.md
├── interpretation.md
├── references.md
├── report.md
├── report.tex
├── report.pdf                  # only when LaTeX compilation succeeds
├── report_manifest.json
├── statistical_summary.md
├── figures/
│   ├── average_ranks.png
│   ├── critical_difference.png # only with more than two algorithms
│   ├── performance_heatmap.png
│   └── pvalue_heatmap.png      # only with control post-hoc comparisons
└── tables/
    ├── average_ranks.csv
    ├── descriptive_statistics.csv
    ├── effect_sizes.csv        # when effect sizes are present
    ├── posthoc_holm.csv        # only with control post-hoc comparisons
    └── posthoc_nemenyi.csv     # only without a control, k > 2
```

`analysis.json` is the deterministic machine-readable output. The report
manifest records generated and skipped formats. No-control multi-algorithm
critical-difference visual groups are exposed as
`statistical_analysis.critical_difference.cd_groups`; control/Holm workflows
leave that list empty because formal post-hoc decisions are Holm
control-vs-rest comparisons. See
[docs/EXAMPLE_OUTPUT.md](https://github.com/jmartinezot/MLStatReport/blob/main/docs/EXAMPLE_OUTPUT.md)
for the purpose of each file.
Generated `report/` directories are intentionally ignored by Git and excluded
from Docker build contexts.

## REST API quickstart

Start the local API in development mode:

```bash
make api
```

In another terminal:

```bash
curl http://localhost:8000/health

curl -X POST http://localhost:8000/api/v1/analyze \
    -F "file=@examples/results.csv" \
    -F "control_algorithm=Proposed" \
    -F "metric=accuracy" \
    -F "higher_is_better=true" \
    -F "posthoc=holm" \
    --output analysis.zip
```

The analysis endpoint accepts CSV, XLSX, and JSON benchmark files and returns
the report directory as a ZIP archive. Uploads and generated files use
request-scoped temporary storage. Validation failures return structured JSON
errors; unexpected failures return a safe HTTP 500 response without a
traceback. See
[docs/API.md](https://github.com/jmartinezot/MLStatReport/blob/main/docs/API.md)
for endpoint fields and response formats.

## Docker and Docker Compose

Build and run the production image:

```bash
docker build -t mlstatreport .
docker run --rm -p 8000:8000 mlstatreport
```

`make docker` is a convenience alias for the build command. For a live-reload
development API using the checked-out source, run:

```bash
docker compose up --build
```

Stop the Compose service with `docker compose down`.

## Statistical methods currently implemented

- Per-algorithm descriptive statistics: mean, standard deviation, median,
  minimum, maximum, and sample count.
- Per-dataset average ranks with average-rank tie handling for both
  higher-is-better and lower-is-better metrics.
- Wilcoxon signed-rank test for exactly two algorithms, using
  `zero_method="wilcox"` to omit exact zero paired differences and a
  two-sided alternative by default.
- Paired row-wise win rate for the two-algorithm Wilcoxon workflow, computed
  over matched dataset rows with metric direction respected.
- Friedman omnibus test and Iman-Davenport correction for more than two
  algorithms.
- Kendall's W global effect size for the Friedman workflow, using the same
  tie-correction convention selected for the Friedman statistic.
- Nemenyi critical-difference value for more than two algorithms, used to
  render a Demšar-style CD diagram. No-control Nemenyi workflows include
  Nemenyi-style all-pairwise CD group bars when available; control/Holm
  workflows use the CD diagram as a rank-scale reference only.
- Control-versus-rest average-rank comparisons with two-sided normal p-values
  and Holm step-down adjustment for the multi-algorithm workflow.
- Nemenyi all-pairwise average-rank comparisons for the multi-algorithm
  workflow when no control algorithm is supplied.
- Pairwise paired win-rate values in Holm and Nemenyi post-hoc tables,
  computed over matched dataset rows and reported separately from significance
  decisions.
- Deterministic matplotlib average-rank, critical-difference, performance, and
  post-hoc p-value heatmap figures.
- Rule-based, evidence-linked interpretation and local BibTeX/Markdown
  references.

For exactly two algorithms, configure the pairwise workflow with
`--pairwise-test wilcoxon` and `--pairwise-alternative two-sided|less|greater`.
For more than two algorithms, pairwise settings are accepted as provenance but
ignored by the Friedman/Iman-Davenport workflow. Multi-algorithm runs use Holm
control-versus-rest comparisons when `--control` is provided, and Nemenyi
all-pairwise comparisons when no control is provided.

## Not implemented yet

- Finner, Bergmann-Hommel, Quade, aligned-rank, Bayesian methods, effect-size
  confidence intervals, or effect-size families beyond Kendall's W and paired
  win rate.
- Full journal-template support, bibliography retrieval, authentication,
  background jobs, report upload storage, and a web frontend.

## Citations

Each generated report selects and writes the methodological citations it uses
to `bibliography.bib` and `references.md`. The built-in registry includes the
Wilcoxon (1945), Friedman (1937), Iman-Davenport (1980), Holm (1979),
Nemenyi (1963), Kendall and Babington Smith (1939), Demšar (2006),
García et al. (2010), García and Herrera (2008), and Santafé et al. (2015)
references.
Verify the generated bibliography against the original sources before
publication, particularly entries intentionally stored with limited metadata.

## Development commands

```bash
make install      # install editable package with development tools
make format       # apply Ruff fixes and Black formatting
make lint         # run Ruff and Black checks
make typecheck    # run mypy
make test         # run pytest
make ci           # lint, type-check, test, build, and check distributions
make build        # build source and wheel distributions
make check-dist   # build and validate distributions with twine
make clean        # remove local build and test artifacts
make api          # run the live-reload API
make docker       # build the production Docker image
```

## Project layout

```text
mlstatreport/      Reusable library, CLI, API, and rendering packages
examples/          Small benchmark inputs used by the quickstart
tests/             Automated tests
docs/              Architecture and output documentation
.devcontainer/     Development container configuration
```

See
[docs/ARCHITECTURE.md](https://github.com/jmartinezot/MLStatReport/blob/main/docs/ARCHITECTURE.md)
for package boundaries and extension guidance, and
[RELEASE.md](https://github.com/jmartinezot/MLStatReport/blob/main/RELEASE.md)
for the release checklist. Release history is tracked in
[CHANGELOG.md](https://github.com/jmartinezot/MLStatReport/blob/main/CHANGELOG.md),
and contribution guidance is in
[CONTRIBUTING.md](https://github.com/jmartinezot/MLStatReport/blob/main/CONTRIBUTING.md).

## License

MLStatReport is released under the
[MIT License](https://github.com/jmartinezot/MLStatReport/blob/main/LICENSE).
