Metadata-Version: 2.4
Name: svc-processing
Version: 0.1.5
Summary: Pure-Python SVC HR-1024i hyperspectral .sig processing pipeline
Author: Gold Lab (GrapeSPEC project), Cornell University
Author-email: Cole Regnier <nr466@cornell.edu>
License-Expression: GPL-3.0-only
Project-URL: Repository, https://github.com/regs08/SVCProcessingPipeline
Keywords: hyperspectral,spectroscopy,SVC,HR-1024i,remote-sensing,reflectance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scipy
Provides-Extra: demo
Requires-Dist: ipykernel; extra == "demo"
Requires-Dist: matplotlib; extra == "demo"
Requires-Dist: nbclient; extra == "demo"
Requires-Dist: nbconvert; extra == "demo"
Requires-Dist: nbformat; extra == "demo"
Provides-Extra: dev
Requires-Dist: pyflakes; extra == "dev"
Requires-Dist: pytest>=8.3.0; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: vulture; extra == "dev"
Dynamic: license-file

# SIG Processing Pipeline

A pure-Python pipeline for processing SVC HR-1024i field hyperspectral `.sig` files. Reads raw multi-detector spectra, performs sensor stitching and radiometric correction, applies resolution-matched Gaussian smoothing, and resamples onto a uniform 1 nm grid from 400–2500 nm. Numerically verified against the legacy R/`spectrolab` reference to better than 1 × 10⁻⁶ absolute reflectance.

> **This README is the canonical entry point for both humans and LLMs.** Every directory in the repo has its own `README.md` describing its contents in detail. Follow the links below — do not assume anything that is not stated here or in the linked docs.

---

## Repository map

| Path | Purpose | Read me first |
|---|---|---|
| [`TUTORIAL.md`](TUTORIAL.md) | Beginner walkthrough — starts in the Jupyter notebook (gentle path), with an advanced CLI track. | Start here if new |
| [`pyproject.toml`](pyproject.toml) | Package metadata and `svc-pipeline` console script. | This file ↓ |
| [`pipeline/`](pipeline/) | Core Python package: CLI, `SigFileProcessor`, `resample_spectra`, `SVCDataProcessor`. | [`pipeline/README.md`](pipeline/README.md) |
| [`tests/`](tests/) | Pytest suite, including the R-vs-Python parity test. | [`tests/README.md`](tests/README.md) |
| [`config/`](config/) | Run configs + instrument calibration JSONs. | [`config/README.md`](config/README.md) |
| [`docs/`](docs/) | Manuscript-grade methods, parity reports, LLM re-test prompt. | [`docs/README.md`](docs/README.md) |
| [`archived_r_scripts/`](archived_r_scripts/) | Frozen R/`spectrolab` reference (Pipeline A) — kept only for parity verification. | [`archived_r_scripts/README.md`](archived_r_scripts/README.md) |
| [`notebooks/`](notebooks/) | Demo and visualization notebooks (not on the production path). | [`notebooks/README.md`](notebooks/README.md) |
| [`naming_ids/`](naming_ids/) | Private CSV lookup tables for grouping scans into samples; only the schema README is tracked. | [`naming_ids/README.md`](naming_ids/README.md) |
| [`FOLDER_STRUCTURE.md`](FOLDER_STRUCTURE.md) | Authoritative tree + reading order. | — |
| [`ACKNOWLEDGMENTS.md`](ACKNOWLEDGMENTS.md) | Attribution, third-party licenses, and citation. | — |
| [`LICENSE`](LICENSE) | GNU General Public License v3.0. | — |

Generated outputs (gitignored): `pipeline_outputs/sig_processed/<run>/`, `pipeline_outputs/sig_resampled/<run>/`.

---

## Quick start

> New to the pipeline? [`TUTORIAL.md`](TUTORIAL.md) starts with the interactive
> Jupyter notebook (the gentle path) and has an advanced track for the CLI below.

**Installed from PyPI, in a new project folder** (no clone of this repo needed):

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install svc-processing   # package name

svc-pipeline --init-config   # command name — different from the package name above
# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline
```

**Working from a clone of this repo** (config/config.json already exists) — run the
setup script once, then activate the environment it created:

```bash
./setup.sh            # creates .venv and installs the project + notebook tools
# ./setup.sh --dev    # add the test & lint tools for development

source .venv/bin/activate          # macOS / Linux
# .venv\Scripts\activate           # Windows PowerShell

# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline config.json
```

The script always installs into *this* repo's `.venv` no matter which folder you run
it from, so it avoids the "Directory '.' is not installable" and "No module named …"
errors that come from being in the wrong directory. Prefer to do it by hand?

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev,demo]"
svc-pipeline config.json
```

Outputs land under `pipeline_outputs/` by default. See [`config/README.md`](config/README.md) for every supported key.

---

## Pipeline architecture

```
   raw *.sig files (sig_input_dir)
            │
            ▼
   ┌────────────────────────────────────┐
   │ Stage 1: SigFileProcessor          │  pipeline/sig_processor.py
   │   - instrument-consistency check   │
   │   - truncate at calibration        │
   │     end-line wavelength            │
   │   - write summary CSV              │
   └────────────────────────────────────┘
            │
            ▼  processed *.sig + *_processed_sig_summary.csv
   ┌────────────────────────────────────┐
   │ Stage 2: resample_spectra()        │  pipeline/resampler.py
   │   - detect sensor segments         │
   │   - guess_splice_at + trim         │
   │   - match_sensors (iter = 1)       │
   │   - smooth_fwhm (k=3 kmeans)       │
   │   - Gaussian resample fwhm=10      │
   │     onto 400–2500 nm @ 1 nm        │
   └────────────────────────────────────┘
            │
            ▼  <run>_merged_spectra.csv
   ┌────────────────────────────────────┐
   │ Stage 3 (post-hoc, optional):      │  pipeline/processor.py
   │   SVCDataProcessor /               │  + notebooks/
   │   SigSpectraAverager — group &     │
   │   average scans into samples.      │
   └────────────────────────────────────┘
```

The `svc-pipeline` console script runs Stages 1 and 2 through `pipeline.cli`,
and Stage 3 too once a `groups_csv` is configured (or passed via
`--groups-csv`) — see [`config/README.md`](config/README.md#grouping--stage-3-optional).
Stage 3 is also available directly from notebooks against the Stage 2 output.
The pipeline is **single-pass and idempotent per input directory** — previous
processed `.sig` files in the target directory are deleted at the start of each
run.

For the formal algorithmic spec (every constant, every formula), read [`docs/supplementary_methods.md`](docs/supplementary_methods.md). That document — not this README — is the source of truth for *what* the code does.

---

## Running the pipeline

```bash
svc-pipeline [config] [options]
```

| Argument | Meaning |
|---|---|
| `config` | Run-config JSON (positional, optional; default `config/config.json`). Bare names resolve under `config/`, so `config.json`, `config`, and `config/config.json` are equivalent. See [`config/README.md`](config/README.md) for schema. |
| `--input-dir <path>` | Override `sig_input_dir` and process only this directory. |
| `--step {1,2,3,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1); `3` = group + average only (requires Stage 2 and `groups_csv`); `all` = every stage that's configured (default). |
| `--groups-csv <path>` | Override the config's `groups_csv` to group repeat scans and average them. |
| `--group-method {mean,median,sum,min,max}` | Override the config's `group_agg_method` for Stage 3 (default `mean`). |
| `--verbose` | Print INFO/DEBUG logs before and after each stage. |

### Expected output layout

```
pipeline_outputs/
├── sig_processed/<input_dir_name>/
│   ├── <truncated *.sig files>
│   └── <input_dir_name>_processed_sig_summary.csv
└── sig_resampled/<input_dir_name>/
    ├── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
    └── <input_dir_name>_grouped_spectra.csv    # only if groups_csv is configured
```

`summary_csv_name`, `merged_csv_name`, and `grouped_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).

---

## Verification (R-vs-Python parity)

The Python resampler is verified against the legacy R/`spectrolab` script ([`archived_r_scripts/merge_resample_sig.R`](archived_r_scripts/merge_resample_sig.R)). Acceptance threshold: **1 × 10⁻³ absolute reflectance** (0.1 %, well below the HR-1024i radiometric noise floor). Historical results:

| Dataset | Samples | Max abs diff | Mean abs diff |
|---|---|---|---|
| Silver instrument (Serial 1202103) | 66 | 1.10 × 10⁻⁶ | 4.0 × 10⁻⁸ |
| Bronze instrument (Serial 2212118), `a4any_sb_2025-cn_ch-svc-aviris_bottom` | 15 | 9.4 × 10⁻⁷ | 3.4 × 10⁻⁸ |

Run the parity test (skipped automatically without reference data):

```bash
pytest tests/test_resampler_parity.py \
    --r-reference-csv=/path/to/r_output/merged_spectra.csv \
    --r-input-dir=/path/to/processed_sig_files/
```

To re-run parity on a new dataset, hand the prompt at [`docs/parity_retest_prompt.md`](docs/parity_retest_prompt.md) to any capable coding LLM. Full details in [`tests/README.md`](tests/README.md) and [`docs/README.md`](docs/README.md).

---

## Public Python API

The most-used entry points (all importable from their concrete modules — `pipeline/__init__.py` is intentionally empty):

```python
from pipeline.sig_processor import SigFileProcessor   # truncation + instrument inspection
from pipeline.resampler      import process_sig_file, resample_spectra
from pipeline.notebook       import Spectrum, SpectraCollection, build_config
from pipeline.processor      import (
    SVCDataProcessor,      # chainable load/group/average
    SigSpectraAverager,    # facade — pass a DataFrame, get aggregated DataFrame back
    GroupSpec,             # GroupSpec.from_csv("naming_ids/<file>.csv")
    find_spectra_by_name,  # cross-DataFrame name search
)
```

Detailed signatures and behavioural notes in [`pipeline/README.md`](pipeline/README.md).

---

## For LLMs working in this repo

1. **Read [`FOLDER_STRUCTURE.md`](FOLDER_STRUCTURE.md) first** — it lists every directory and the reading order.
2. **Before modifying [`pipeline/resampler.py`](pipeline/resampler.py), re-read [`docs/supplementary_methods.md`](docs/supplementary_methods.md).** The constants `_FWHM_NM`, `_SIGMA_NM`, `_INTERP_WVL`, `_FIXED_SENSOR`, `_BAND_MIN`, `_BAND_MAX`, and the algorithm steps are load-bearing for the parity claim. Changing any of them requires re-running the parity test and writing a new `docs/parity_<dataset>_<date>.md`.
3. **The R script at [`archived_r_scripts/merge_resample_sig.R`](archived_r_scripts/merge_resample_sig.R) is frozen.** Treat it as a behavioural reference, not as live code to edit.
4. **Never commit machine paths or private data.** Use the placeholders already present in [`config/config.json`](config/config.json) and the notebooks.
5. **Run `pytest` after any change to `pipeline/`.** The parity test will skip if reference data is unavailable; the rest of the suite still runs.

---

## Demo Notebook

[`notebooks/pipeline_demo.ipynb`](notebooks/pipeline_demo.ipynb) is designed to
be shared as a standalone file: its first code cell checks for the notebook
helpers, installs `svc-processing[demo]>=0.1.5` into the current kernel when
needed, and falls back to the public GitHub source archive while that release is
not yet on PyPI. The tutorial assumes you have an authorized folder of raw
`.sig` scans; set `DATA_FOLDER`, `OUTPUT_FOLDER`, `INSTRUMENT`, and `END_LINE` in
the opening cells. Raw field scans are not shipped in the public repository
because their headers can contain time and GPS metadata.
See
[`notebooks/pipeline_demo/README.md`](notebooks/pipeline_demo/README.md).

Authorized parity work can still stage the separate external field dataset with
[`scripts/prepare_demo_data.py`](scripts/prepare_demo_data.py); those raw files
never enter the package or default notebook path.

---

## Requirements

Python 3.11 is the supported runtime. Runtime, demo, and development
dependencies are declared in [`pyproject.toml`](pyproject.toml).

- `numpy`, `scipy`, `pandas` — numerical core.
- `matplotlib`, `ipykernel` — demo notebook plotting and execution.
- `pytest>=8.3.0` — test runner for local verification.

Install with `python -m pip install -e ".[dev,demo]"` for local development and
use `svc-pipeline = "pipeline.cli:main"` as the command-line entry point.

R is **not** required for the production pipeline. It is needed only to regenerate Pipeline A parity references; see [`archived_r_scripts/README.md`](archived_r_scripts/README.md).

---

## License

Released under the [GNU General Public License v3.0](LICENSE) © 2026 Cole
Regnier ([regs08](https://github.com/regs08), nr466@cornell.edu), Gold Lab
(GrapeSPEC project), Cornell University. The package uses the SPDX expression
`GPL-3.0-only`, chosen for compatibility with the GPL-3 `spectrolab` reference
implementation.

---

## Acknowledgments & Citation

This project is an independent reimplementation of the **spectrolab** algorithm
(no source copied) and builds on prior field-spectroscopy work. See
[`ACKNOWLEDGMENTS.md`](ACKNOWLEDGMENTS.md) for full attribution, third-party
licenses, and how to cite this software and spectrolab.
