Metadata-Version: 2.4
Name: svc-processing
Version: 0.1.2
Summary: Pure-Python SVC HR-1024i hyperspectral .sig processing pipeline
Author: Gold Lab (GrapeSPEC project), Cornell University
Author-email: Cole Regnier <nr466@cornell.edu>
License-Expression: GPL-3.0-only
Project-URL: Repository, https://github.com/regs08/SVCProcessingPipeline
Keywords: hyperspectral,spectroscopy,SVC,HR-1024i,remote-sensing,reflectance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scipy
Provides-Extra: demo
Requires-Dist: matplotlib; extra == "demo"
Requires-Dist: nbclient; extra == "demo"
Requires-Dist: nbconvert; extra == "demo"
Requires-Dist: nbformat; extra == "demo"
Provides-Extra: dev
Requires-Dist: pyflakes; extra == "dev"
Requires-Dist: pytest>=8.3.0; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: vulture; extra == "dev"
Dynamic: license-file

# SIG Processing Pipeline

A pure-Python pipeline for processing SVC HR-1024i field hyperspectral `.sig` files. Reads raw multi-detector spectra, performs sensor stitching and radiometric correction, applies resolution-matched Gaussian smoothing, and resamples onto a uniform 1 nm grid from 400–2500 nm. Numerically verified against the legacy R/`spectrolab` reference to better than 1 × 10⁻⁶ absolute reflectance.

> **This README is the canonical entry point for both humans and LLMs.** Every directory in the repo has its own `README.md` describing its contents in detail. Follow the links below — do not assume anything that is not stated here or in the linked docs.

---

## Repository map

| Path | Purpose | Read me first |
|---|---|---|
| [`TUTORIAL.md`](TUTORIAL.md) | Beginner walkthrough — starts in the Jupyter notebook (gentle path), with an advanced CLI track. | Start here if new |
| [`pyproject.toml`](pyproject.toml) | Package metadata and `svc-pipeline` console script. | This file ↓ |
| [`pipeline/`](pipeline/) | Core Python package: CLI, `SigFileProcessor`, `resample_spectra`, `SVCDataProcessor`. | [`pipeline/README.md`](pipeline/README.md) |
| [`tests/`](tests/) | Pytest suite, including the R-vs-Python parity test. | [`tests/README.md`](tests/README.md) |
| [`config/`](config/) | Run configs + instrument calibration JSONs. | [`config/README.md`](config/README.md) |
| [`docs/`](docs/) | Manuscript-grade methods, parity reports, LLM re-test prompt. | [`docs/README.md`](docs/README.md) |
| [`archived_r_scripts/`](archived_r_scripts/) | Frozen R/`spectrolab` reference (Pipeline A) — kept only for parity verification. | [`archived_r_scripts/README.md`](archived_r_scripts/README.md) |
| [`notebooks/`](notebooks/) | Demo and visualization notebooks (not on the production path). | [`notebooks/README.md`](notebooks/README.md) |
| [`naming_ids/`](naming_ids/) | Private CSV lookup tables for grouping scans into samples; only the schema README is tracked. | [`naming_ids/README.md`](naming_ids/README.md) |
| [`FOLDER_STRUCTURE.md`](FOLDER_STRUCTURE.md) | Authoritative tree + reading order. | — |
| [`ACKNOWLEDGMENTS.md`](ACKNOWLEDGMENTS.md) | Attribution, third-party licenses, and citation. | — |
| [`LICENSE`](LICENSE) | GNU General Public License v3.0. | — |

Generated outputs (gitignored): `pipeline_outputs/sig_processed/<run>/`, `pipeline_outputs/sig_resampled/<run>/`.

---

## Quick start

> New to the pipeline? [`TUTORIAL.md`](TUTORIAL.md) starts with the interactive
> Jupyter notebook (the gentle path) and has an advanced track for the CLI below.

**Installed from PyPI, in a new project folder** (no clone of this repo needed):

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install svc-processing

svc-pipeline --init-config   # writes a starter config/config.json here
# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline
```

**Working from a clone of this repo** (config/config.json already exists):

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev,demo]"

# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline config.json
```

Outputs land under `pipeline_outputs/` by default. See [`config/README.md`](config/README.md) for every supported key.

---

## Pipeline architecture

```
   raw *.sig files (sig_input_dir)
            │
            ▼
   ┌────────────────────────────────────┐
   │ Stage 1: SigFileProcessor          │  pipeline/sig_processor.py
   │   - instrument-consistency check   │
   │   - truncate at calibration        │
   │     end-line wavelength            │
   │   - write summary CSV              │
   └────────────────────────────────────┘
            │
            ▼  processed *.sig + *_processed_sig_summary.csv
   ┌────────────────────────────────────┐
   │ Stage 2: resample_spectra()        │  pipeline/resampler.py
   │   - detect sensor segments         │
   │   - guess_splice_at + trim         │
   │   - match_sensors (iter = 1)       │
   │   - smooth_fwhm (k=3 kmeans)       │
   │   - Gaussian resample fwhm=10      │
   │     onto 400–2500 nm @ 1 nm        │
   └────────────────────────────────────┘
            │
            ▼  <run>_merged_spectra.csv
   ┌────────────────────────────────────┐
   │ Stage 3 (post-hoc, optional):      │  pipeline/processor.py
   │   SVCDataProcessor /               │  + notebooks/
   │   SigSpectraAverager — group &     │
   │   average scans into samples.      │
   └────────────────────────────────────┘
```

The `svc-pipeline` console script glues Stages 1 and 2 together through
`pipeline.cli`; Stage 3 is invoked from notebooks against the Stage 2 output.
The pipeline is **single-pass and idempotent per input directory** — previous
processed `.sig` files in the target directory are deleted at the start of each
run.

For the formal algorithmic spec (every constant, every formula), read [`docs/supplementary_methods.md`](docs/supplementary_methods.md). That document — not this README — is the source of truth for *what* the code does.

---

## Running the pipeline

```bash
svc-pipeline [config] [options]
```

| Argument | Meaning |
|---|---|
| `config` | Run-config JSON (positional, optional; default `config/config.json`). Bare names resolve under `config/`, so `config.json`, `config`, and `config/config.json` are equivalent. See [`config/README.md`](config/README.md) for schema. |
| `--input-dir <path>` | Override `sig_input_dir` and process only this directory. |
| `--step {1,2,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1 to have been run); `all` = both (default). |
| `--verbose` | Print INFO/DEBUG logs before and after each stage. |

### Expected output layout

```
pipeline_outputs/
├── sig_processed/<input_dir_name>/
│   ├── <truncated *.sig files>
│   └── <input_dir_name>_processed_sig_summary.csv
└── sig_resampled/<input_dir_name>/
    └── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
```

`summary_csv_name` and `merged_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).

---

## Verification (R-vs-Python parity)

The Python resampler is verified against the legacy R/`spectrolab` script ([`archived_r_scripts/merge_resample_sig.R`](archived_r_scripts/merge_resample_sig.R)). Acceptance threshold: **1 × 10⁻³ absolute reflectance** (0.1 %, well below the HR-1024i radiometric noise floor). Historical results:

| Dataset | Samples | Max abs diff | Mean abs diff |
|---|---|---|---|
| Silver instrument (Serial 1202103) | 66 | 1.10 × 10⁻⁶ | 4.0 × 10⁻⁸ |
| Bronze instrument (Serial 2212118), `a4any_sb_2025-cn_ch-svc-aviris_bottom` | 15 | 9.4 × 10⁻⁷ | 3.4 × 10⁻⁸ |

Run the parity test (skipped automatically without reference data):

```bash
pytest tests/test_resampler_parity.py \
    --r-reference-csv=/path/to/r_output/merged_spectra.csv \
    --r-input-dir=/path/to/processed_sig_files/
```

To re-run parity on a new dataset, hand the prompt at [`docs/parity_retest_prompt.md`](docs/parity_retest_prompt.md) to any capable coding LLM. Full details in [`tests/README.md`](tests/README.md) and [`docs/README.md`](docs/README.md).

---

## Public Python API

The most-used entry points (all importable from their concrete modules — `pipeline/__init__.py` is intentionally empty):

```python
from pipeline.sig_processor import SigFileProcessor   # truncation + instrument inspection
from pipeline.resampler      import process_sig_file, resample_spectra
from pipeline.processor      import (
    SVCDataProcessor,      # chainable load/group/average
    SigSpectraAverager,    # facade — pass a DataFrame, get aggregated DataFrame back
    GroupSpec,             # GroupSpec.from_csv("naming_ids/<file>.csv")
    find_spectra_by_name,  # cross-DataFrame name search
)
```

Detailed signatures and behavioural notes in [`pipeline/README.md`](pipeline/README.md).

---

## For LLMs working in this repo

1. **Read [`FOLDER_STRUCTURE.md`](FOLDER_STRUCTURE.md) first** — it lists every directory and the reading order.
2. **Before modifying [`pipeline/resampler.py`](pipeline/resampler.py), re-read [`docs/supplementary_methods.md`](docs/supplementary_methods.md).** The constants `_FWHM_NM`, `_SIGMA_NM`, `_INTERP_WVL`, `_FIXED_SENSOR`, `_BAND_MIN`, `_BAND_MAX`, and the algorithm steps are load-bearing for the parity claim. Changing any of them requires re-running the parity test and writing a new `docs/parity_<dataset>_<date>.md`.
3. **The R script at [`archived_r_scripts/merge_resample_sig.R`](archived_r_scripts/merge_resample_sig.R) is frozen.** Treat it as a behavioural reference, not as live code to edit.
4. **Never commit machine paths or private data.** Use the placeholders already present in [`config/config.json`](config/config.json) and the notebooks.
5. **Run `pytest` after any change to `pipeline/`.** The parity test will skip if reference data is unavailable; the rest of the suite still runs.

---

## Demo Notebook

The demo notebook uses an external 15-file `.sig` artifact because raw headers
contain GPS/location metadata. Prepare local demo data with:

```bash
python3 scripts/prepare_demo_data.py \
  --source-dir data/a4any_sb_2025-cn_ch-svc-aviris_bottom
```

The notebook is config-driven: edit the settings cell (`DATA_FOLDER`,
`OUTPUT_FOLDER`, `INSTRUMENT`) to run it on your own data and instrument. See
[`notebooks/pipeline_demo/README.md`](notebooks/pipeline_demo/README.md).

---

## Requirements

Python 3.11 is the supported runtime. Runtime, demo, and development
dependencies are declared in [`pyproject.toml`](pyproject.toml).

- `numpy`, `scipy`, `pandas` — numerical core.
- `matplotlib` — demo notebook plotting.
- `pytest>=8.3.0` — test runner for local verification.

Install with `python -m pip install -e ".[dev,demo]"` for local development and
use `svc-pipeline = "pipeline.cli:main"` as the command-line entry point.

R is **not** required for the production pipeline. It is needed only to regenerate Pipeline A parity references; see [`archived_r_scripts/README.md`](archived_r_scripts/README.md).

---

## License

Released under the [GNU General Public License v3.0](LICENSE) © 2026 Cole
Regnier ([regs08](https://github.com/regs08), nr466@cornell.edu), Gold Lab
(GrapeSPEC project), Cornell University. The package uses the SPDX expression
`GPL-3.0-only`, chosen for compatibility with the GPL-3 `spectrolab` reference
implementation.

---

## Acknowledgments & Citation

This project is an independent reimplementation of the **spectrolab** algorithm
(no source copied) and builds on prior field-spectroscopy work. See
[`ACKNOWLEDGMENTS.md`](ACKNOWLEDGMENTS.md) for full attribution, third-party
licenses, and how to cite this software and spectrolab.
