Metadata-Version: 2.4
Name: ecosignallab
Version: 0.3.0
Summary: ecoSignalLab (esl): calibration-aware, multi-channel environmental and architectural acoustics toolkit
Author: Colby Leider and ecoSignalLab contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/TheColby/ecoSignalLab
Project-URL: Documentation, https://github.com/TheColby/ecoSignalLab#readme
Project-URL: Repository, https://github.com/TheColby/ecoSignalLab
Project-URL: Issues, https://github.com/TheColby/ecoSignalLab/issues
Keywords: acoustics,audio,dsp,ecoacoustics,architectural-acoustics,machine-learning,signal-processing,multichannel
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: soundfile>=0.12
Requires-Dist: pandas>=2.0
Requires-Dist: markdown>=3.6
Requires-Dist: pymdown-extensions>=10.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: matplotlib>=3.7
Requires-Dist: h5py>=3.10
Requires-Dist: requests>=2.31
Provides-Extra: archive
Requires-Dist: boto3>=1.34; extra == "archive"
Requires-Dist: psycopg[binary]>=3.1; extra == "archive"
Provides-Extra: ml
Requires-Dist: torch>=2.0; extra == "ml"
Requires-Dist: datasets>=2.16; extra == "ml"
Requires-Dist: huggingface-hub>=0.20; extra == "ml"
Requires-Dist: scikit-learn>=1.4; extra == "ml"
Provides-Extra: io
Requires-Dist: pyarrow>=14.0; extra == "io"
Provides-Extra: plot
Requires-Dist: plotly>=5.18; extra == "plot"
Provides-Extra: features
Requires-Dist: librosa>=0.10; extra == "features"
Provides-Extra: docs
Requires-Dist: playwright>=1.48; extra == "docs"
Requires-Dist: pypdf>=4.0; extra == "docs"
Provides-Extra: mat
Requires-Dist: scipy>=1.10; extra == "mat"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: mypy>=1.9; extra == "dev"
Dynamic: license-file

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/logos/esl/minimal/esl_logo_04_circle_wave_dark.png" />
    <img src="assets/logos/esl/minimal/esl_logo_04_circle_wave.png" alt="ecoSignalLab logo" width="260" />
  </picture>
</p>

## ecoSignalLab (`esl`)

Want to understand what is happening inside your environmental audio recordings? `esl` is an open-source, production-oriented acoustic analytics SDK for environmental, architectural, and industrial audio workflows.

If you know nothing yet except “I have an audio file and need actionable acoustic insights,” this README is for you. On the other hand, if you are an expert in audio analysis, bioacoustics, or acoustic ecology, this repo can provide actionable insights into your long-duration audio files.

**Positioning statement:** `esl` is a **true multichannel and Atmos-aware/capable acoustic analysis** toolkit, capable of analyzing multi-year-long audio files.

## Installation

If you just want `esl` installed with the least drama, use `pipx`.

### Option 1: `pipx` install

```bash
python3 -m pip install --user pipx
pipx install "ecosignallab[io,plot,features]"
```

### Option 2: `uv tool` install

```bash
uv tool install "ecosignallab[io,plot,features]"
```

### Option 3: Homebrew install on macOS

If you want a Homebrew-style install from this repo:

```bash
brew install --HEAD ./Formula/ecosignallab.rb
```

If you have not cloned the repo and want to install directly from GitHub:

```bash
brew install --HEAD https://raw.githubusercontent.com/TheColby/ecoSignalLab/main/Formula/ecosignallab.rb
```

What the Homebrew formula does:
- installs `python@3.12`
- installs `ffmpeg`
- installs `libsndfile`
- installs `esl`
- installs the man pages

### Option 4: Local developer install from source

```bash
bash scripts/install.sh
```

After any install method, run:

```bash
esl doctor
```

Plain English: if `esl doctor` looks happy, you are ready to analyze audio.

## Start in 60 Seconds

If you just downloaded this and want one immediate win, run this:

```bash
esl doctor input.wav
esl analyze input.wav --out-dir out --json out/input.json --plot --device auto
```

Expected result:
- an environment/input readiness report
- a metrics file at `out/input.json`
- generated plots in `out/input_plots/`

If you want the shortest human-readable summary instead of full exports, run:

```bash
esl simple input.wav
```

If you want an `esl` onboarding flow (analysis + clips + plots), run:

```bash
esl quickstart
```

## Pick Your Goal

- I want one navigation page with links to everything:
  - [`docs/INDEX.md`](docs/INDEX.md)
- I need install help first:
  - this README section: [`Installation`](#installation)
- I want the roadmap:
  - [`ROADMAP.md`](ROADMAP.md)
- I want simple copy-paste tasks:
  - [`docs/TASK_RECIPES.md`](docs/TASK_RECIPES.md)
  - Includes a hyperlinked **Recipe Index by Device Type**
- I got an error and need a fix now:
  - [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md)
- I am about to announce this and want the likely complaints handled:
  - [`docs/ANNOUNCEMENT_FAQ.md`](docs/ANNOUNCEMENT_FAQ.md)
- I do not know the terms yet:
  - [`docs/GLOSSARY.md`](docs/GLOSSARY.md)
- I want ready-made scripts:
  - [`scripts/easy/README.md`](scripts/easy/README.md)
- I want numerical KPI comparisons on real audio:
  - [`docs/ALGORITHM_COMPARISON.md`](docs/ALGORITHM_COMPARISON.md)
- I want visual graphs of signals/windows/STFT:
  - [`docs/SIGNAL_WINDOWS_VISUAL_GUIDE.md`](docs/SIGNAL_WINDOWS_VISUAL_GUIDE.md)
- I want to find the most similar files in a folder:
  - [`docs/SIMILARITY_SEARCH.md`](docs/SIMILARITY_SEARCH.md)
- I want to search a long shard archive for the shards most similar to a query:
  - [`docs/SHARD_WORKFLOWS.md#archive-level-similarity-search`](docs/SHARD_WORKFLOWS.md#archive-level-similarity-search)
- I want to search a long shard archive for the exact moments most similar to a query:
  - [`docs/SHARD_WORKFLOWS.md#archive-level-event-retrieval`](docs/SHARD_WORKFLOWS.md#archive-level-event-retrieval)
- I want higher-level soundscape insights like calmness, drift, occupancy, retrieval, and storyboards:
  - [`docs/INSIGHTS.md`](docs/INSIGHTS.md)

## First 5 Minutes (Copy/Paste)

```bash
# 0) Check the environment and file first
esl doctor input.wav

# 1) Analyze one file
esl analyze input.wav --out-dir out --json out/input.json --plot --device auto

# 2) Get a short human-readable summary
esl simple input.wav

# 3) Extract the single most novel moment
esl moments extract input.wav --out out/moments --single --rank-metric novelty_curve --event-window 8

# 4) Export ML-ready frame features
esl features extract input.wav --out out/vectors.npz --feature-set all --meta-json out/vectors_meta.json
```

Expected outputs:
- `out/input.json`
- `out/input_plots/`
- `out/moments/moments.csv`
- `out/moments/clips/moment_0001.wav` (for `--single`)
- `out/vectors.npz`

## What this is

- A CLI-first, production-grade SDK for environmental and architectural acoustic analysis.
- A true multichannel and Atmos-aware/capable acoustic analysis workflow surface.
- A multi-channel, calibration-aware metric engine with reproducible outputs and provenance capture.
- An ML-ready acoustic intelligence toolkit with tabular/tensor exports and anomaly/novelty tooling.
- A plugin-extensible open-source platform for research and industrial interoperability.

## What this is not

- Not a DAW or audio editor for recording/mixing/mastering workflows.
- Not a GUI-first desktop app (interactive plotting is supported; full GUI is optional).
- Not a black-box vendor suite with opaque assumptions or locked data formats.
- Not a substitute for formal compliance certification measurements without calibrated hardware/protocols.

If your goal is creative transformation effects, `esl` is not the primary tool for that workflow.

## What is a shard?

A shard is just one piece of a bigger recording archive.

Plain English:
- if one giant ten-year file sounds absurd, that is because it is
- the sane version is many smaller files
- each smaller file is a shard

Example:
- one hour per file
- one day per file
- one deployment segment per file

So instead of:

- `10_year_recording.rf64`

you might have:

- `2017/01/01/00.wav`
- `2017/01/01/01.wav`
- `2017/01/01/02.wav`

In `esl`:
- `esl shard index` builds the archive map
- `esl shard analyze` analyzes the archive shard by shard
- `esl shard moments` finds archive-level interesting events
- `esl shard similar` finds which shards most resemble a query recording
- `esl shard retrieve` finds which time windows inside the shards most resemble a query recording
- `esl shard insights` summarizes the archive without loading decade-sized audio into memory

Friendly mental model:
- shard = one chapter
- manifest = the table of contents
- archive = the full book

## Friendly Notes

- Start with defaults first, then tune thresholds gradually once you see real outputs.
- If calibration is missing, `esl` still works; just treat SPL-like numbers as uncalibrated estimates.
- For design comparisons, keep assumptions consistent between runs so deltas stay meaningful.
- Keep provenance outputs; they save a lot of debugging time later.

## What Can I Do with esl?

- Analyze single files with calibrated, multi-channel metrics and full provenance output.
- Run batch analysis over large datasets and export JSON/CSV/Parquet/HDF5/MAT.
- Treat hourly/daily shard folders as one ordered archive with `esl shard index` and `esl shard analyze`.
- Find archive-level top-ranked events with `esl shard moments`.
- Find the archive shards most similar to a query recording with `esl shard similar`.
- Find the exact archive windows most similar to a query recording with `esl shard retrieve`, including spatial-aware matching.
- Profile baseline versus spatial archive retrieval throughput before a long run with `esl shard profile-retrieve`.
- Summarize long-archive manifests and reports with `esl shard insights`.
- Run higher-level insight workflows with `esl insights`: scene changes, calmness/chaos/diversity, occupancy, drift, retrieval, embeddings, reports, simulation comparison, and storyboards.
- Generate static and interactive plots, including similarity and novelty matrix views.
- Build ML-ready frame tables and tensors for PyTorch/HuggingFace workflows, including appendable CSV, Parquet-dataset, and HDF5 sidecars.
- Build deterministic ML dataset manifests from exported feature metadata with `esl features manifest`.
- Turn a streamed shard-analysis report into deterministic ML splits that point directly at FrameTable sidecars with `esl shard dataset`.
- Verify calibration math with deterministic reference fixtures using `esl calibrate verify`.
- Emit structured spatial metadata, including Ambisonics order, channel-map, normalization profiles, and validated JSON/YAML sidecar overrides, in analysis outputs.
- Compare architectural design variants (`--project` / `--variant`) with delta reports.
- Ingest online datasets (Freesound/HuggingFace), auto-analyze, and summarize.
- Run regression validation across datasets to catch metric drifts.

Typical flow:

```bash
esl analyze file.wav --json out.json --plot --ml-export
esl batch input_dir --out out_dir --csv --parquet --hdf5
esl project compare --project restaurant_design --root out_dir --baseline A
esl validate input_dir --out validation_out --rules rules.json
esl similar query.wav corpus_dir --top-k 5 --json out/similarity.json
esl shard similar archive_manifest.json query.wav --out out/shard_similarity --top-k 5
esl shard retrieve archive_manifest.json query.wav --out out/shard_retrieval --top-k 10 --window-seconds 8
esl shard retrieve archive_manifest.json query.wav --out out/spatial_retrieval --top-k 10 --window-seconds 8 --spatial-mode append --spatial-weight 0.7
```

Core goals:
- Multi-channel native (`1..N`) with ambisonic-compatible handling
- Atmos-aware/capable channel workflow support (bed/object-aware interop surfaces)
- Calibration-aware (`dBFS <-> SPL`, `dBA`, `dBC`, `dBZ`)
- ML-ready feature export (NumPy, PyTorch, Hugging Face)
- Architectural simulation compatible (IR metrics and project variants)
- Industrial measurement interoperability (HEAD/APx/SoundCheck CSV mappings)
- Reproducible by default (seed, config hash, version stamping)
- Plugin-extensible metric engine

See:
- [`docs/GETTING_STARTED.md`](docs/GETTING_STARTED.md)
- [`docs/TASK_RECIPES.md`](docs/TASK_RECIPES.md)
- [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md)
- [`docs/ANNOUNCEMENT_FAQ.md`](docs/ANNOUNCEMENT_FAQ.md)
- [`docs/GLOSSARY.md`](docs/GLOSSARY.md)
- [`docs/ALGORITHM_COMPARISON.md`](docs/ALGORITHM_COMPARISON.md)
- [`docs/SIGNAL_WINDOWS_VISUAL_GUIDE.md`](docs/SIGNAL_WINDOWS_VISUAL_GUIDE.md)
- [`scripts/easy/README.md`](scripts/easy/README.md)
- [`DESIGN.md`](DESIGN.md)
- [`ARCHITECTURE.md`](ARCHITECTURE.md)
- [`CHANGELOG.md`](CHANGELOG.md)
- [`docs/METRICS_REFERENCE.md`](docs/METRICS_REFERENCE.md)
- [`docs/SCHEMA.md`](docs/SCHEMA.md)
- [`docs/ML_FEATURES.md`](docs/ML_FEATURES.md)
- [`docs/NOVELTY_ANOMALY.md`](docs/NOVELTY_ANOMALY.md)
- [`docs/MOMENTS_EXTRACTION.md`](docs/MOMENTS_EXTRACTION.md)
- [`docs/VALIDATION.md`](docs/VALIDATION.md)
- [`docs/RELEASE.md`](docs/RELEASE.md)
- [`docs/PHASE1_ECOSYSTEM_GAP_ANALYSIS.md`](docs/PHASE1_ECOSYSTEM_GAP_ANALYSIS.md)
- [`docs/REFERENCES.md`](docs/REFERENCES.md)
- [`docs/ATTRIBUTION.md`](docs/ATTRIBUTION.md)

## System Visuals

```mermaid
flowchart LR
    A["Inputs\nWAV FLAC AIFF RF64 CAF MP3 AAC OGG Opus WMA ALAC M4A SOFA"] --> B["Decode + Normalize\nSoundFile / FFmpeg"]
    B --> C["Analysis Context\nSignal + Calibration + Config"]
    C --> D["Metric Registry\nBuiltin + Plugin"]
    D --> E["Result Document\nMetadata + Metrics + Provenance"]
    E --> F["Exports\nJSON CSV Parquet HDF5 MAT Industrial CSV"]
    E --> G["Plots\nPNG + Plotly + Similarity + Novelty"]
    E --> H["ML Artifacts\nNumPy Torch HF Dataset"]
```

```mermaid
flowchart TD
    A["Calibration Inputs"] --> B["dBFS Reference"]
    A --> C["SPL Reference"]
    A --> D["Mic Sensitivity"]
    A --> E["Calibration Tone"]
    B --> F["Offset = SPL_ref - dBFS_ref"]
    C --> F
    F --> G["Level Metrics"]
    D --> G
    E --> H["Drift Check"]
    H --> G
    G --> I["Output Labels\ndBFS dBA dBC SPL"]
```

```mermaid
sequenceDiagram
    participant U as User
    participant CLI as esl CLI
    participant IO as Audio I/O
    participant M as Metric Engine
    participant X as Exporters
    U->>CLI: esl analyze input.wav --plot --ml-export
    CLI->>IO: decode + channel layout detection
    CLI->>M: compute selected metrics
    M-->>CLI: summary + series + confidence + spec
    CLI->>X: write json/csv/parquet/hdf5/mat
    CLI->>X: render plots + ML artifacts
    X-->>U: output files + metadata provenance
```

```mermaid
flowchart LR
    K["esl quickstart"] --> K1["Copy-paste first commands"]
    A["esl analyze"] --> A1["Single-file analysis"]
    B["esl batch"] --> B1["Directory analysis"]
    C["esl pipeline run"] --> C1["Staged workflow"]
    D["esl plot"] --> D1["Render from JSON"]
    E["esl ingest"] --> E1["Online dataset ingestion"]
    F["esl schema"] --> F1["Output JSON schema"]
    G["esl docs"] --> G1["Build HTML/PDF documentation"]
    H["esl stream"] --> H1["Chunk analysis + alerts"]
    I["esl moments extract"] --> I1["Timestamp CSV + clip export"]
    J["esl features extract"] --> J1["Frame-level feature vectors"]
    L["esl similar"] --> L1["Top-K similar files in corpus"]
```

```mermaid
flowchart TD
    A["Architectural Simulation Export"] --> B["IR / Rendered WAV FLAC"]
    B --> C["esl analyze --project --variant"]
    C --> D["RT60 EDT C50 C80 D50 T20 T30 Ts"]
    C --> E["SPL Time Trends"]
    D --> F["Project Index"]
    E --> F
    F --> G["Variant Comparison CSV + JSON"]
```

```mermaid
flowchart LR
    A["Spectral Features"] --> B["Self-Similarity Matrix"]
    B --> C["Checkerboard Kernel"]
    C --> D["Novelty Curve"]
    D --> E["Peak Picking"]
    E --> F["Event Candidates"]
```

```mermaid
flowchart LR
    A["Metric Series"] --> B["Frame Table"]
    B --> C["Wide Matrix"]
    C --> D["Isolation Forest / OCSVM / AE Proxy"]
    D --> E["Anomaly Scores"]
    C --> F["Torch Tensor"]
    C --> G["HF Dataset"]
```

```mermaid
flowchart TB
    A["Reproducibility"] --> B["Seed Control"]
    A --> C["Config Hash"]
    A --> D["Version Stamping"]
    A --> E["Assumption Logging"]
    B --> F["Comparable Runs"]
    C --> F
    D --> F
    E --> F
```

## Installation

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

Or use install script (installs package and man pages):

```bash
bash scripts/install.sh
```

Optional extras:

```bash
pip install -e .[dev,ml,plot,io,docs]
pip install -e .[features]   # librosa-rich feature vectors/similarity
```

System dependencies:
- [FFmpeg](https://ffmpeg.org/) / `ffprobe` on `PATH` for compressed audio decode
- `playwright` browser install for PDF docs generation:

```bash
python -m playwright install chromium
```

## CLI

### Analyze one file

```bash
esl analyze file.wav \
  --json output.json \
  --csv output.csv \
  --parquet output.parquet \
  --hdf5 output.h5 \
  --calibration calibration.yaml \
  --verbosity 2 \
  --debug 1 \
  --plot \
  --show \
  --ml-export \
  --project restaurant_design \
  --variant designA
```

### Multi-resolution profile analyze

```bash
esl analyze file.wav \
  --profile configs/multi_resolution.json \
  --out-dir out_profiles \
  --json out_profiles/file_profile.json
```

Profile file example:

```json
{
  "profiles": [
    {"name": "short", "frame_size": 1024, "hop_size": 256, "metrics": ["rms_dbfs", "snr_db"]},
    {"name": "long", "frame_size": 4096, "hop_size": 1024, "metrics": ["rms_dbfs", "snr_db", "rt60_s"]}
  ]
}
```

### Batch mode

```bash
esl batch input_dir --out out_dir --csv --parquet --hdf5
```

### Validation mode (regression harness)

```bash
esl validate input_dir --out validation_out --rules rules.json
```

Outputs:
- `validation_out/validation_report.json`
- `validation_out/validation_summary.csv`

### Staged pipeline mode (CLI-first)

```bash
esl pipeline run input_dir --out out_dir --plot --interactive --show --ml-export
esl pipeline status --manifest out_dir/pipeline_manifest.json
```

Pipeline mode persists:
- `pipeline_manifest.json` (stage status, timing, counts, errors)
- `pipeline_analysis_index.json` (analysis artifact index)
- `pipeline_digest.csv` and `pipeline_digest.json` (dataset-level summary)

### Streaming chunk analysis + alerts

```bash
esl stream input.wav \
  --out stream_out \
  --chunk-size 131072 \
  --metrics spl_a_db,ndsi,novelty_curve \
  --rules rules/stream_alerts.yaml
```

### Human-readable window and chunk sizes

You can now set frame/hop/chunk values in seconds/minutes/hours/days instead of manual sample math.

Precedence rule:
- duration flags override sample-count flags (`--frame-seconds` over `--frame-size`, etc.)

Supported duration flags:
- `--frame-seconds`
- `--hop-seconds`
- `--chunk-seconds`
- `--chunk-minutes`
- `--chunk-hours`
- `--chunk-days`

Example (24-hour style workflow, no manual sample conversion):

```bash
esl stream input_24h.wav \
  --sample-rate 96000 \
  --frame-seconds 1.0 \
  --hop-seconds 0.5 \
  --chunk-minutes 10 \
  --metrics spl_a_db,ndsi,novelty_curve \
  --out stream_out
```

### Spatial analyze + beam map

```bash
esl spatial analyze multichannel.wav \
  --array-config configs/arrays/stereo_pair.json \
  --beam-map \
  --out-dir spatial_out
```

### Calibration drift check

```bash
esl calibrate check \
  --tone calib/1k_94db.wav \
  --calibration examples/calibration.yaml \
  --dbfs-reference -20.0 \
  --mic-sensitivity-mv-pa 12.5 \
  --preamp-gain-db 34.0 \
  --adc-full-scale-vrms 1.0 \
  --max-drift-db 1.0 \
  --history calib/history.csv \
  --out calib/check_report.json
```

Run all deterministic calibration reference fixtures:

```bash
esl calibrate verify --fixture all --out calib/verify_suite.json
```

### Interesting moments extraction (clips + timestamp CSV)

```bash
esl moments extract input_24h.wav \
  --out out/moments \
  --rules rules_24h.yaml \
  --metrics novelty_curve,spectral_change_detection,isolation_forest_score,spl_a_db \
  --chunk-size 2880000 \
  --sample-rate 96000 \
  --top-k 12 \
  --rank-metric novelty_curve \
  --event-window 8 \
  --merge-gap 2
```

Selection controls:
- `--single` for the single most novel (or highest-ranked) moment
- `--top-k N` for a few highest-ranked moments
- `--all` for all detected moments
- `--window-before S` and `--window-after S` for explicit clip windowing around each event

### Plot from existing JSON

```bash
esl plot results.json --out plots --interactive
```

Plot controls:
- `--metrics spl_a_db,snr_db,novelty_curve`
- `--no-spectral` (skip spectrogram/mel/log/waterfall/LTSA suite)
- `--similarity-matrix` (generate self-similarity matrix plot)
- `--novelty-matrix` (generate Foote-style novelty matrix plot)
- `--sim-feature-set auto|core|librosa|all` (feature backend for similarity/novelty)
- `--feature-vectors vectors.npz` (drive similarity/novelty from external vectors)
- `--show --show-limit 12` (spawn generated plots via system viewer/browser)

### Extract feature vectors (for similarity/anomaly workflows)

```bash
esl features extract input.wav \
  --out vectors.npz \
  --feature-set all \
  --frame-size 1024 \
  --hop-size 256 \
  --meta-json vectors_meta.json
```

### Ingest datasets

```bash
esl ingest --source freesound --query "restaurant ambience" --limit 50 --out ingest_data
```

```bash
esl ingest --source huggingface --query "speechcolab/gigaspeech" --limit 10 --out ingest_data
```

### Build documentation (HTML + PDF)

```bash
esl docs --root . --out docs/build --formats html,pdf
```

Equivalent script entry:

```bash
python scripts/build_docs.py --out docs/build --formats html,pdf
```

### Build the dedicated user guide PDF

```bash
python scripts/generate_user_guide.py
```

This renders [`docs/USERGUIDE.md`](docs/USERGUIDE.md) into HTML under
`docs/build/userguide/` and the printable [`USERGUIDE.pdf`](USERGUIDE.pdf).

### Build the textbook

```bash
python scripts/generate_textbook.py
```

This assembles maintained technical chapters into [`docs/TEXTBOOK.md`](docs/TEXTBOOK.md),
renders HTML under `docs/build/textbook/`, and writes the ready-to-print US Letter
[`TEXTBOOK.pdf`](TEXTBOOK.pdf). The print master uses binding-safe margins, a
New Century Schoolbook-compatible book face, running headers, centered folios,
an unnumbered title page, and monochrome-safe text. It is a single-page print master; a printer performs
imposition and binding.

Docs rendering guarantees:
- Mermaid diagrams are rendered in HTML/PDF.
- If a page has no Mermaid block, an auto-generated visual outline is inserted.
- TeX math is written with GitHub-friendly `$...$` and `$$...$$` delimiters and rendered via MathJax in generated HTML/PDF.

### Print JSON schema

```bash
esl schema
```

### Benchmark compute backend (CPU/CUDA/MPS)

```bash
esl benchmark device --device auto --frames 16384 --features 256 --iters 20
```

## Input format support

Native (`soundfile`):
- WAV
- FLAC
- AIFF/AIF
- RF64
- CAF

Compressed (`ffmpeg` fallback):
- MP3
- AAC
- OGG
- Opus
- WMA
- ALAC
- M4A

Spatial:
- Ambisonic-compatible multichannel WAV workflows
- Atmos-aware/capable multichannel workflow support (layout/channel-aware processing surfaces)
- SOFA IR import (HDF5-based, first measurement decode)

Large files:
- Chunked mode via `--chunk-size` or human-readable `--chunk-seconds|--chunk-minutes|--chunk-hours|--chunk-days`.
- Out-of-core summary workflow via `--streamable-only --summary-only --frame-table-csv --checkpoint-dir --resume`.
- RF64 guidance (limits, math, and workflows): [`docs/RF64_AND_LARGE_FILES.md`](docs/RF64_AND_LARGE_FILES.md)

## What It Takes To Handle A 2-Year 96 kHz Float32 WAV

For uncompressed audio, storage grows with duration, sample rate, channel count,
and the number of bytes used for each sample. The audio payload size is:

$$
B = T \cdot f_s \cdot C \cdot b
$$

where:

- $B$ is the audio payload size in bytes.
- $T$ is duration in seconds.
- $f_s$ is sample rate in sample frames per second.
- $C$ is channel count.
- $b$ is bytes per sample per channel.

Plain English: each second contains $f_s$ frames, and each frame contains $C$
samples of $b$ bytes each. Container headers and metadata add a small amount of
storage beyond this payload estimate.

Using two 365-day years, the duration is:

$$
\begin{aligned}
T &= 2 \cdot 365 \cdot 24 \cdot 3600 \\
  &= 63{,}072{,}000\ \mathrm{s}.
\end{aligned}
$$

where each year has 365 days, each day has 24 hours, and each hour has 3600
seconds. Calendar years that include leap days have a different duration.

At 96 kHz and 32-bit float, $f_s = 96{,}000$ and $b = 4$. For mono audio,
$C = 1$, so the payload is:

$$
\begin{aligned}
B &= 63{,}072{,}000 \cdot 96{,}000 \cdot 1 \cdot 4 \\
  &= 24{,}219{,}648{,}000{,}000\ \mathrm{bytes}.
\end{aligned}
$$

That is about:
- `24.22 TB` for mono
- `48.44 TB` for stereo
- `96.88 TB` for 4 channels
- `387.51 TB` for 16 channels

Practical consequences:
- classic RIFF/WAV is not enough; use `RF64`, `CAF`, or sharded files
- a single full-file decode is not acceptable
- even one sequential read of `24.22 TB` takes about `33.6 h` at `200 MB/s`, `13.5 h` at `500 MB/s`, or `6.7 h` at `1 GB/s`

Plain English: this is not a “load the file and see what happens” problem.

`esl` now supports the correct first-pass workflow for this class of recording:
- out-of-core chunked analysis for streaming-capable metrics
- `--summary-only` JSON for bounded output size
- incremental `--frame-table-csv` export for disk-backed frame features
- resumable checkpoints via `--checkpoint-dir` and `--resume`
- `esl stream` and `esl moments extract` as the coarse-scan and event-extraction path
- `esl shard index` and `esl shard analyze` when the archive is already split into hourly/daily files

Recommended first-pass command:

```bash
esl analyze two_year_capture.wav \
  --out-dir out \
  --chunk-hours 1 \
  --streamable-only \
  --summary-only \
  --frame-table-csv out/two_year_frame_table.csv \
  --checkpoint-dir out/checkpoints \
  --resume
```

Recommended “find the interesting parts first” command:

```bash
esl moments extract two_year_capture.wav \
  --out out/moments \
  --single \
  --rank-metric novelty_curve \
  --chunk-hours 1 \
  --event-window 30
```

If you are about to do this with one giant classic `.wav`, stop and read [`docs/RF64_AND_LARGE_FILES.md`](docs/RF64_AND_LARGE_FILES.md) first.

If you already have shards instead of one giant file, read [`docs/SHARD_WORKFLOWS.md`](docs/SHARD_WORKFLOWS.md).

### Ten-year, 8-channel top-33 novelty workflow

Yes, `esl` can do this if you treat it as an out-of-core discovery problem rather than a normal file load.

One terminology fix first:
- "`32-bit float 24-bit`" is not a single WAV subtype
- choose either `32-bit float` or `24-bit PCM`

At 96 kHz, eight channels, ten 365-day years, and float32 samples, the same
payload-size relationship applies:

$$
B = T \cdot f_s \cdot C \cdot b
$$

where:

- $T = 315{,}360{,}000$ seconds is the duration.
- $f_s = 96{,}000$ sample frames per second is the sample rate.
- $C = 8$ is the channel count.
- $b = 4$ bytes per sample per channel is the float32 sample size.

Substituting these values gives:

$$
\begin{aligned}
B &= 315{,}360{,}000 \cdot 96{,}000 \cdot 8 \cdot 4 \\
  &= 968{,}785{,}920{,}000{,}000\ \mathrm{bytes}.
\end{aligned}
$$

That is about `968.79 TB` (decimal units), excluding container overhead.

Recommended workflow:

```bash
# Pass 1: resumable novelty scan
esl stream ten_year_8ch_capture.rf64 \
  --out out/ten_year_stream \
  --metrics novelty_curve,spectral_change_detection,spl_a_db,ndsi \
  --frame-seconds 1 \
  --hop-seconds 1 \
  --chunk-hours 6 \
  --checkpoint-dir out/ten_year_stream/checkpoints \
  --resume

# Pass 2: export the 33 highest-ranked moments
esl moments extract ten_year_8ch_capture.rf64 \
  --out out/ten_year_moments \
  --stream-report out/ten_year_stream/stream_report.json \
  --top-k 33 \
  --rank-metric novelty_curve \
  --window-before 30 \
  --window-after 90 \
  --merge-gap 0
```

Why this is the right shape:
- `stream` is resumable and keeps chunk summaries on disk
- `moments extract` reuses that report instead of repeating the scan
- `--merge-gap 0` preserves distinct ranked events when you want exactly 33 outputs
- novelty is ranked on the stable mono channel-mean downmix, while the exported clips preserve all original channels

Full details:
- [`docs/MOMENTS_EXTRACTION.md`](docs/MOMENTS_EXTRACTION.md)
- [`docs/RF64_AND_LARGE_FILES.md`](docs/RF64_AND_LARGE_FILES.md)

## Built-in metric families

`esl` ships **74 built-in metrics** by default.

```mermaid
flowchart TB
    M["esl Metrics (74 built-ins)"]
    M --> B["Basic + QC"]
    M --> L["Level + Loudness"]
    M --> N["Noise + SNR"]
    M --> S["Spectral + Temporal"]
    M --> E["Ecoacoustics"]
    M --> P["Spatial + Ambisonic"]
    M --> A["Architectural + Intelligibility"]
    M --> X["Anomaly + Novelty"]
    B --> B1["RMS / Peak / Crest"]
    B --> B2["Clipping / DC / Silence / Dropout"]
    L --> L1["SPL A/C/Z, Leq/Lmax/Lmin"]
    L --> L2["LUFS / LRA / True Peak / SEL"]
    N --> N1["SNR Percentile / Stationarity"]
    S --> S1["Centroid / Bandwidth / Flatness / Rolloff"]
    S --> S2["Octave / Third-Octave / ZCR"]
    E --> E1["BAI / ACI / NDSI / ADI / AEI / Entropy"]
    P --> P1["IACC / ILD / IPD / ITD / Coherence / FOA"]
    A --> A1["RT60 / EDT / T20 / T30 / C50 / C80 / D50 / Ts"]
    X --> X1["Novelty Curve / Isolation Forest / OCSVM / AE Proxy"]
```

See full definitions in [`docs/METRICS_REFERENCE.md`](docs/METRICS_REFERENCE.md), including mathematical equations, plain-English interpretation, and citation links.

## Reproducibility

Each run emits:
- `schema_version`
- `config_hash` (SHA-256 over canonicalized config)
- `pipeline_hash` (config + metric list + window/hop + library versions)
- `esl_version`
- UTC timestamp
- local timestamp
- seed value
- runtime fingerprint (python/platform/hostname)
- compute device provenance (`requested`, `resolved`, CUDA/MPS availability)
- channel layout hint
- decoder provenance (`decoder_used`, ffmpeg version, ffprobe stream summary when available)
- resolved config snapshot + metric catalog/version
- per-channel and aggregate channel summaries with validity flags
- calibration assumptions

## Recent Quality Updates

- Internal/project docs now use repository-relative hyperlinks (no local filesystem paths).
- JSON schema hardened with `schema_version`, `pipeline_hash`, metric catalog versioning, and decoder provenance.
- CLI `schema` command now reports schema version and can write the current schema artifact.
- Canonical ML `FrameTable` contract added for tabular (`CSV/Parquet`) and tensor (`NumPy/Torch`) export semantics.
- Long-duration out-of-core analysis added via `--streamable-only`, `--summary-only`, `--frame-table-csv`, `--checkpoint-dir`, and `--resume`.
- Golden metric tests added for basic level metrics, loudness sanity, IR RT/EDT fit quality, NDSI sign behavior, and multichannel aggregation semantics.

## Calibration model

Calibration file supports:
- `dbfs_reference`
- `spl_reference_db`
- `mic_sensitivity_mv_pa`
- `preamp_gain_db`
- `adc_full_scale_vrms`
- `weighting`: `A|C|Z`
- `calibration_tone_file`

Example: [`examples/calibration.yaml`](examples/calibration.yaml)

Precision note:
- Offset mapping (`dbfs_reference` + `spl_reference_db`) provides `dBFS <-> SPL` conversion.
- Physical pressure conversion (`Pa <-> dBFS`) is supported when all three are provided:
  - `mic_sensitivity_mv_pa`
  - `preamp_gain_db`
  - `adc_full_scale_vrms`

## Project mode

Project/variant analysis for design comparison:

```bash
esl analyze A.wav --project restaurant_design --variant A --out-dir out
esl analyze B.wav --project restaurant_design --variant B --out-dir out
esl project compare --project restaurant_design --root out --baseline A
```

Outputs:
- `out/projects/restaurant_design/index.json`
- `out/projects/restaurant_design/comparison.csv`
- `out/projects/restaurant_design/comparison_report.json`
- `out/projects/restaurant_design/comparison_deltas.csv`

## Documentation

- Advanced review, monitoring, spatial, QA, indexing, uncertainty, protocol,
  embedding, privacy, and archive workflows: [workflow guide](docs/ADVANCED_WORKFLOWS.md).
- Documentation index (hyperlinked map + device-type recipe index):
  - [`docs/INDEX.md`](docs/INDEX.md)
- Man pages:
  - [`docs/MANPAGES.md`](docs/MANPAGES.md)
- Getting started: [`docs/GETTING_STARTED.md`](docs/GETTING_STARTED.md)
- Task recipes: [`docs/TASK_RECIPES.md`](docs/TASK_RECIPES.md)
- Troubleshooting: [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md)
- Glossary: [`docs/GLOSSARY.md`](docs/GLOSSARY.md)
- Real-input KPI algorithm comparison: [`docs/ALGORITHM_COMPARISON.md`](docs/ALGORITHM_COMPARISON.md)
- Signal/window visual guide (generated graphs): [`docs/SIGNAL_WINDOWS_VISUAL_GUIDE.md`](docs/SIGNAL_WINDOWS_VISUAL_GUIDE.md)
- Easy scripts catalog (expanded starter workflows): [`scripts/easy/README.md`](scripts/easy/README.md)
- System design: [`DESIGN.md`](DESIGN.md)
- Architecture: [`ARCHITECTURE.md`](ARCHITECTURE.md)
- Metrics reference: [`docs/METRICS_REFERENCE.md`](docs/METRICS_REFERENCE.md)
- Output schema contract: [`docs/SCHEMA.md`](docs/SCHEMA.md)
- RF64 and large-file guide: [`docs/RF64_AND_LARGE_FILES.md`](docs/RF64_AND_LARGE_FILES.md)
- Shard-manifest workflows: [`docs/SHARD_WORKFLOWS.md`](docs/SHARD_WORKFLOWS.md)
- ML FrameTable contract: [`docs/ML_FEATURES.md`](docs/ML_FEATURES.md)
- Novelty/anomaly semantics: [`docs/NOVELTY_ANOMALY.md`](docs/NOVELTY_ANOMALY.md)
  - Includes committed novelty plot example set: [`docs/examples/novelty_plot_set/`](docs/examples/novelty_plot_set)
- Moments extraction workflow: [`docs/MOMENTS_EXTRACTION.md`](docs/MOMENTS_EXTRACTION.md)
- Similarity search workflow: [`docs/SIMILARITY_SEARCH.md`](docs/SIMILARITY_SEARCH.md)
- Validation harness: [`docs/VALIDATION.md`](docs/VALIDATION.md)
- Release process: [`docs/RELEASE.md`](docs/RELEASE.md)
- Algorithm index: [`docs/ALGORITHM_INDEX.md`](docs/ALGORITHM_INDEX.md)
- Ecosystem gap analysis: [`docs/PHASE1_ECOSYSTEM_GAP_ANALYSIS.md`](docs/PHASE1_ECOSYSTEM_GAP_ANALYSIS.md)
- Documentation automation: [`docs/DOCS_AUTOMATION.md`](docs/DOCS_AUTOMATION.md)
- References: [`docs/REFERENCES.md`](docs/REFERENCES.md)
- NSF ecology + ecoacoustics papers: [`docs/REFERENCES.md#nsf-supported-ecology-and-ecoacoustics-papers`](docs/REFERENCES.md#nsf-supported-ecology-and-ecoacoustics-papers)
- Long-term audio analysis (calmness/diversity): [`docs/REFERENCES.md#long-term-audio-analysis-calmness-and-diversity`](docs/REFERENCES.md#long-term-audio-analysis-calmness-and-diversity)
- Attribution: [`docs/ATTRIBUTION.md`](docs/ATTRIBUTION.md)

## License

MIT. See [`LICENSE`](LICENSE).
