Metadata-Version: 2.4
Name: nirs4all-cluster
Version: 0.1.2
Summary: Distributed job queue for nirs4all.run() — FastAPI server + polling workers.
Project-URL: Homepage, https://nirs4all.org
Project-URL: Documentation, https://nirs4all-cluster.readthedocs.io
Project-URL: Repository, https://github.com/GBeurier/nirs4all-cluster
Project-URL: Issues, https://github.com/GBeurier/nirs4all-cluster/issues
Project-URL: Changelog, https://github.com/GBeurier/nirs4all-cluster/blob/main/CHANGELOG.md
Author: nirs4all
License-Expression: CECILL-2.1 OR AGPL-3.0-or-later
License-File: LICENSE
License-File: LICENSES/AGPL-3.0-or-later.txt
License-File: LICENSES/Apache-2.0.txt
License-File: LICENSES/BSD-2-Clause.txt
License-File: LICENSES/BSD-3-Clause.txt
License-File: LICENSES/COMMERCIAL-LICENSE.md
License-File: LICENSES/COMMERCIAL-LICENSE_FR.md
License-File: LICENSES/CeCILL-2.1.txt
License-File: LICENSES/MIT.txt
Keywords: cluster,distributed,job-queue,nirs,nirs4all
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: packaging>=23
Requires-Dist: pydantic>=2.6
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: uvicorn[standard]>=0.27
Requires-Dist: websockets>=12
Provides-Extra: dev
Requires-Dist: mypy>=1.9; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: docs
Requires-Dist: furo>=2024.8.6; extra == 'docs'
Requires-Dist: linkify-it-py<3,>=2.0; extra == 'docs'
Requires-Dist: myst-parser<5,>=4.0; extra == 'docs'
Requires-Dist: sphinx-copybutton<0.6,>=0.5.2; extra == 'docs'
Requires-Dist: sphinx<9,>=8.1; extra == 'docs'
Requires-Dist: sphinxcontrib-mermaid<2,>=1.0; extra == 'docs'
Provides-Extra: worker
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/brand/horizontal-dark.svg">
    <img alt="nirs4all-cluster" src="assets/brand/horizontal.svg" width="440">
  </picture>
</p>

# nirs4all-cluster

> **Status: beta · trusted-LAN.** A small distributed job queue for `nirs4all.run()`,
> built for a **trusted local network** — not the open internet or untrusted multi-tenant
> use. The documented non-goals and security posture still hold (a single static token, no
> sandbox, a single SQLite server). See [`SECURITY.md`](SECURITY.md),
> [`PROTOTYPE_DESIGN.md`](PROTOTYPE_DESIGN.md) (the design source of truth) and
> [`PROTOTYPE_TO_PRODUCTION.md`](PROTOTYPE_TO_PRODUCTION.md). The broader question of a
> native cluster vs. an opt-in Dask backend in `nirs4all` remains open.
>
> 📖 **Docs:** <https://nirs4all-cluster.readthedocs.io> · 🖥️ **Dashboard:** `http://HOST:8765/ui`

**Distributed** execution of`nirs4all`pipelines (client / server / workers): a coordinator
receives jobs and dispatches the work to workers that poll the server. The prototype
**does not modify any other library** in the ecosystem:`nirs4all`is only imported by the
runner subprocess, and the server/client works without it.

## What it does

- Submission of a`nirs4all.run()`job via Python SDK or CLI. - FastAPI server + SQLite file + local object store addressed by SHA-256. - Workers polling (long-polling HTTP + heartbeat), with a task sandbox per folder. - Atomic job (Level 0) and`pipelines × datasets`decomposition (Level 1) with aggregation/ranking. - Download artifacts: JSON summary, logs, best`.n4a`model. - Recovery after worker crash (lease + retry), cooperative cancellation, idempotence. - Routing by capabilities: labels, memory, **package versions (PEP 440)**, **GPU/CUDA** (auto-detected,`requirements.min_gpu_count`or`cuda=true`label). A`nirs4all.run`job requires`nirs4all`by default. - **Live web dashboard** at `/ui` (jobs, workers, events; cancel) over a global WebSocket stream. - **Version compatibility tracking**: client/server/worker advertise their `nirs4all-cluster` + protocol version; incompatible protocol majors are rejected, compatible drift is logged/traced. **Pipeline content fingerprints** recorded end-to-end. - **Richer job listing/filtering** (`n4cluster jobs --status … --name …`, `GET /v1/stats`) and **structured logging** (`--log-level`, `--log-file`).

## Installation

```bash
# Worker environment = an existing nirs4all environment + this package:
uv pip install -e .            # serveur + client + transport worker
# (workers provide nirs4all themselves; it is not a hard dependency)
```

Python ≥ 3.11. The server and client only need FastAPI/uvicorn/httpx/pydantic; only the
worker needs a provisioned`nirs4all`environment.

## Quickstart (LAN de confiance)

```bash
# 1) serveur
n4cluster server --host 0.0.0.0 --port 8765 --state ./cluster-state

# 2) one or more workers (on machines that can see nirs4all and the dataset)
#    The worker auto-detects GPUs (nvidia-smi) and declares the cuda + gpu_count labels;
#    force with --gpus N (0 to hide GPUs).
n4cluster worker --server http://HOST:8765 --labels site=lab --slots 1

# 3) submit a job and wait for the result
n4cluster submit examples/job.shared-path.yaml --wait --out ./results
n4cluster status   <job_id>
n4cluster jobs     --status running        # filter the job list
n4cluster logs     <job_id>
n4cluster cancel   <job_id>
n4cluster artifacts <job_id> --out ./results

# 4) watch everything live in the browser
#    open http://HOST:8765/ui
```

SDK Python :

```python
from nirs4all_cluster import ClusterClient

client = ClusterClient("http://host:8765", token=None)
job = client.submit_run(
    pipeline="/shared/pipelines/pls.yaml",                 # kind=path
    dataset="/shared/datasets/corn",                       # kind=shared_path
    params={"random_state": 42, "refit": True},
)
job = client.wait(job.id)
print(job.aggregate.best_metric, job.aggregate.ranking)
client.download_best_model(job.id, "best_model.n4a")
```

## Architecture

```
submitter (SDK/CLI/Studio) ──REST + WS──► serveur (FastAPI + SQLite + object store + scheduler + events)
                                              ▲
                          long-polling HTTP + heartbeat
                                              │
                                          workers ──► subprocess runner ──► nirs4all.run(workspace=task_ws)
```

- **`nirs4all_cluster/server/`** —`app.py`(API),`db.py`(SQLite file, atomic leasing, reaper),`scheduler.py`(state machines + matching),`artifacts.py`(SHA-256 store),`events.py`(broker). - **`nirs4all_cluster/worker/`** —`agent.py`(polling loop),`materialize.py`(resolution of
  references → local paths),`executor.py`(subprocess + capture + undo). - **`nirs4all_cluster/runners/nirs4all_run.py`** — **only** module that imports`nirs4all`. - **`client.py`** (SDK), **`cli.py`** (`n4cluster`), **`schemas.py`** (Pydantic contract).

## Tests and validation

```bash
pytest -q                                   # unit/API tests without nirs4all + skipped integration if unavailable
pytest -q tests/test_release_smoke.py       # wheel + installed CLI/server/worker smoke, no nirs4all
python scripts/validation.py                # end-to-end harness on nirs4all-data (8/8)
```

Results measured on`nirs4all-data`(see [`WORKLOG.md`](WORKLOG.md)): atomic job →`.n4a`,
2 workers in parallel, **kill worker → retry**, cancellation not restarted,`pipeline ×
dataset`aggregation, and **exact metric parity vs local`nirs4all.run()`(diff = 0.0)** — beyond the criterion
go/no-go ≤ 1e-10.

## Go/no-go criteria to upgrade to product

The go remains conditional on **all** of these conditions:

1. ≥ 2 labs/partners explicitly request distributed execution. *(not measurable here)*
2. Speedup ≥ 3× on a real workload (grid search AOM / HPO on ≥ 32 datasets). *(to be measured)*
3. Metric-identical results (≤ 1e-10) with single-machine. → **reached: diff = 0.0** on the atomic job. 4. Data + security + recovery model written **before** the code. → done in`PROTOTYPE_DESIGN.md`. 5. Framing topics covered from the start (mTLS, secrets, third-party sandboxing, IP/GDPR datasets,
   heavy TF/Torch/JAX environments, transfer costs, idempotence/resumption, quotas/fairness,
   heterogeneous scheduling). → listed in`PROTOTYPE_TO_PRODUCTION.md`.

These criteria gate a full **production** commitment (mTLS, sandboxing, Postgres, network
object storage — see [`PROTOTYPE_TO_PRODUCTION.md`](PROTOTYPE_TO_PRODUCTION.md)). Until they
are met, the ecosystem's default option remains an opt-in Dask backend in `nirs4all`; this
repository is published as a usable **trusted-LAN beta** and an auditable reference for the
design, not as a production roadmap commitment.

## Non-objectifs (rappel)

No modification of other libs, no open multi-tenant, no sandbox for Python code
arbitrary, no K8s/Ray/Dask type scheduler, no concurrent writing in a workspace`nirs4all`shared, no distribution of folds. See`PROTOTYPE_DESIGN.md`§ Non-objectives.

## References

`PROTOTYPE_DESIGN.md`,`PROTOTYPE_TO_PRODUCTION.md`,`WORKLOG.md`, and`nirs4all-ecosystem/NIRS4ALL-ECOSYSTEM_VISION.md`(annex *Perspective: distributed execution*, risk R13).

## License

`nirs4all-cluster` is dual-licensed open-source — **`CeCILL-2.1 OR AGPL-3.0-or-later`** (your choice) —
with an optional **commercial license** for closed-source / SaaS use. For any commercial use, contact
<nirs4all-admin@cirad.fr>. See [`LICENSING.md`](LICENSING.md), the texts under [`LICENSES/`](LICENSES/),
and [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md).
