Metadata-Version: 2.5
Name: dispcraft
Version: 0.1.0
Summary: NISP dispersion calibration — pedagogical project
Project-URL: Repository, https://gitlab.in2p3.fr/dispers/dispcraft
Author-email: Julien Zoubian <zoubian@cppm.in2p3.fr>
License-Expression: GPL-3.0-only
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: mlflow>=3.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: pytorch-lightning<3,>=2.6.5
Requires-Dist: requests>=2.28
Requires-Dist: scipy>=1.10
Requires-Dist: torch<3,>=2.12
Description-Content-Type: text/markdown

# dispcraft

Physics-first, then hybrid physics+ML modeling of the Euclid/NISP grism
spectral dispersion — from a geometrical-optics instrument model, through
per-dataset and joint multi-dataset calibration against ground-test data, to
an ML residual corrector and a field-dependent extension, benchmarked
against a published reference result (arXiv:2506.08378).

> **Project status: exploratory research prototype, ongoing — not
> production-ready.** This is a proof-of-concept pipeline, not a finished
> deliverable. The best model produced so far (hybrid physical + ML
> residual) is **1.7x worse on `y` and 8.4x worse on `z`** than the
> reference paper's own held-out accuracy (see Results below and
> `docs/Status_Report.md` §1) — it does **not** currently reach the
> accuracy this instrument needs. Development is ongoing; open gaps and
> next steps are tracked in `docs/Status_Report.md` §9-10.

**If you just want to use the calibrated models (predict where a spectrum
lands, given field position and wavelength), see the
[User Guide](docs/User_Guide.md)** — but read the status note above first:
these are the best models this exploratory project has produced to date,
not models validated as meeting an accuracy requirement. This README covers
the project itself.

## What's here

- **`dispcraft/`** — the library: instrument optics (`optics/`), the
  ground-test forward model and joint-fit machinery (`calibration.py`),
  the field-dependent correction tier (`field_calibration.py`), the
  0th-order dispersion model (`zeroth_dispersion.py`), the Chebyshev
  polynomial residual alternative (`chebyshev_residual.py`), data loading
  (`measurement.py`), the residual-MLP building block (`ml.py`), GitLab
  model-registry helpers (`model_registry.py`), and the unified
  1st-/0th-order prediction entry point (`prediction.py`).
- **`scripts/`** — `register_models.py` trains and publishes the recommended
  residual/NN models to GitLab's model registry (see the User Guide);
  `freeze_stage5_fits.py`, `freeze_chebyshev_fits.py`, and
  `freeze_best_models.py` replay already-validated fits to produce the
  corresponding `models/*.toml` files.
- **`models/*.toml`** — frozen, versioned calibration results (one file per
  fit). Every tunable physical parameter used in a real result is recorded
  here, not hardcoded in code or notebooks.
- **`notebooks/`** — one notebook per project stage/phase, prototyping and
  producing every result before (validated bits) get promoted to
  `dispcraft/`.
- **`data/`** — ground-test PSF datasets (`docs/internship/2-Intro_data/index.html`
  is the authoritative description of what's in here and how it was built;
  see also `data/README.md`).
- **`docs/`** — `User_Guide.md` (how to use the calibrated models),
  `Status_Report.md` (full write-up of methodology and results),
  reference material (`Euclid-NISP-Specs.md`, the benchmark paper summary),
  `internship/` (the slide decks, see below), and the source for the
  published documentation site (`mkdocs.yml` at the repo root).
- **`docs/internship/0-Intro_generale/` … `4-Projet/`** — the slide decks
  that define this project's scope and order (see Roadmap below).
- **`webapp/`** — a self-contained, static JS visualization of the physical
  grism model (no build step, no data dependency) — an interactive,
  outreach-friendly complement to the notebooks.
- **`tests/`** — one test module per `dispcraft/` module.

## Roadmap

This project follows a fixed sequence of stages, each driven by a slide
deck, plus three further stages added by explicit request once the deck's
own scope closed. Status and detailed findings for every stage are tracked
in `CLAUDE.md`; the short version:

| Stage | Topic | Status |
|---|---|---|
| 0 | General introduction | done |
| 1 | Subject/problem, instrument model | done |
| 2 | Data introduction | done |
| 3 | ML introduction, per-dataset physical calibration | done |
| 4 | Project integration: ML residual, joint fitting, literature comparison | done (deck's own scope, one phase skipped by decision) |
| 5 | Extension: field-dependent parameters, 0th-order dispersion, BGS model | done |
| 6 | Status report | done — see `docs/Status_Report.md` |
| 7 | Model distribution: GitLab model registry, unified prediction entry point | done |
| 8 | Chebyshev polynomial residual model (interpretable alternative to the MLP) | done |

## Getting started

<!-- --8<-- [start:getting-started] -->
This project uses [pixi](https://pixi.sh) for environment management —
don't use a bare `venv`/`pip install`.

```bash
pixi install        # create/sync the environment from pixi.toml/pixi.lock
pixi run test        # run the test suite (pytest tests)
pixi run mlflow-ui    # browse logged experiment runs (sqlite:///mlflow.db)
```

To work in a notebook or script interactively:

```bash
pixi shell
jupyter lab notebooks/
```

Data files (`data/*.csv`) are already provided; `data/README.md` documents
how they were extracted from the ground-test database, if you need to
regenerate them.
<!-- --8<-- [end:getting-started] -->

## Results at a glance

<!-- --8<-- [start:results-glance] -->
Final held-out accuracy (mean over the 6 RGS ground-test configs), compared
to the reference paper's own held-out (Argon) figure — full detail in
`docs/Status_Report.md`:

| Model | RMSE y (mm) | RMSE z (mm) |
|---|---|---|
| Physical model only (joint 3-tier fit) | 0.194 | 0.300 |
| **Hybrid: physical + ML residual (recommended default)** | **0.019** | **0.075** |
| Reference paper, held-out | 0.009 | 0.009 |

**This does not yet reach the reference's accuracy** (1.7x worse on `y`,
8.4x worse on `z`) — closing that gap is ongoing work, not a solved
problem. See `docs/Status_Report.md` §9-10 for the open items.
<!-- --8<-- [end:results-glance] -->

See the [User Guide](docs/User_Guide.md) for how to reproduce and use this
(and the BGS / 0th-order counterparts) yourself — as a snapshot of current,
unfinished work, not a validated production model.

## License

GNU General Public License v3.0 — see [LICENSE](LICENSE).
