lidarpy operations

Environment Boundaries

Environment	What runs here	What does not run here
Local development	Python imports, pytest, docs generation, package builds and focused RAW fixture checks.	Persistent services, Docker containers or unattended production jobs.
CI	Linux pytest, docs build, Pages publish and PyPI release workflow.	Private SCC credentials, private RAW datasets or local NAS paths.
Docker	No Docker runtime is defined by this repository.	Do not expect a compose stack, Celery UI or HTTP API from `lidarpy`.
Production	Downstream applications install `atmolidarpy` and import `lidarpy`.	Production secrets and heavy operational datasets should not live in this repo.

Local Operational Checklist

Use this sequence when validating a checkout after pulling changes. It starts with cheap checks and only then moves to heavier RAW conversion.

Confirm the working tree and branch: git status --short --branch.
Confirm import from the checkout with PYTHONPATH=src.
Run fast offline tests: synthetic, retrieval, SCC and docs.
Clean local pytest/unzip temporaries before RAW tests.
Check free disk space before converting RAW fixtures.
Run focused RAW groups in chunks.
Build docs or distributions only after tests relevant to the change pass.

$env:PYTHONPATH = "src"
$env:MPLBACKEND = "Agg"
.\.venv\Scripts\python -m pytest tests\synthetic tests\retrieval tests\scc tests\docs -q

CI Workflows

Workflow	Trigger	Expected result
`Tests`	Push or pull request against `main` or `develop`; manual dispatch.	Installs the package editable, sets `PYTHONPATH=src`, runs `pytest tests -q`.
`docs`	Push to `main`; manual dispatch.	Builds static docs, regenerates figures, creates pdoc API HTML and deploys GitHub Pages.
`Publish Package`	Tag matching `v*`; manual dispatch.	Builds sdist/wheel, checks with Twine and publishes to PyPI through Trusted Publishing.

Release and Deployment

Deployment for this repository means publishing a Python distribution to PyPI and publishing documentation to GitHub Pages. There is no service to restart after a release. Downstream systems adopt a release when they install or pin the new atmolidarpy version.

Update version metadata consistently.
Run the relevant test groups locally.
Build distributions locally if the change touches packaging.
Merge to the release branch.
Create and push a v* tag.
Verify the GitHub Actions publish workflow completes.
Verify PyPI shows the expected version and files.

python -m build
python -m twine check dist/*
git tag v0.1.2
git push origin v0.1.2

Expected result: PyPI lists the new atmolidarpy version and downstream users can install it with python -m pip install atmolidarpy==VERSION.

Rollback Strategy

Python package rollback is usually a version pin, not a destructive deletion. If a release is bad, the practical response is to pin downstream environments to the last known good version and publish a fixed patch release. Deleting files from PyPI should be reserved for severe security, legal or credential exposure incidents because it can break reproducible installs.

Situation	Preferred rollback	Why
Bug in latest release	Pin downstream to previous version; publish a patch fix.	Keeps package history intact and avoids broken installs.
Docs publish has wrong content	Revert/fix docs and rerun the Pages workflow.	Pages is rebuilt from repository content.
Secret accidentally committed	Rotate the secret immediately, then clean history if required.	History cleanup alone does not make an exposed secret safe.

SCC Operations

SCC support is part of the package, but the default test suite does not contact a real SCC server. Offline SCC tests cover local imports, package resources and the access-client contract with fake HTTP sessions. Real submission/download workflows need SCC credentials and server state, so they should be run manually or in a controlled integration environment.

Keep SCC credentials outside Git. Use local ignored configuration files or CI secrets in a repository that is explicitly responsible for that integration. This repository should not contain private credentials, private server URLs or real operational measurements.

Logs and Diagnostics

Most package routines run in-process. Diagnostics therefore come from pytest output, Python exceptions, xarray dataset inspection, generated NetCDF files and quicklook figures. Retrieval and preprocessing paths use logging in active code paths; tests should assert data contracts rather than relying on console output.

Need	First check	Expected signal
Confirm package import	`python -c "import lidarpy; print(lidarpy.__version__)"`	Version prints and command exits successfully.
Inspect a NetCDF product	`xarray.open_dataset(path)`	Dataset has expected coordinates and signal variables.
Check plotting contract	Run quicklook tests or docs figure generation.	PNG exists and is not blank.
Check SCC offline behavior	`pytest tests\scc -q`	SCC smoke and access-contract tests pass without network.

Troubleshooting

Symptom	Likely cause	First check
`ModuleNotFoundError: lidarpy`	The package is not installed or `PYTHONPATH` is not set for source checkout use.	Run `$env:PYTHONPATH="src"` in the same PowerShell session.
Tests import installed code instead of local edits	Editable install or `PYTHONPATH` is missing.	Print `lidarpy.__file__` and confirm it points under `src`.
RAW tests fail on Windows temp permissions	System temp paths or stale pytest folders are interfering.	Clean `.pytest_tmp`, `.pytest_cache` and `tmp_unzipped_*`.
Disk fills during tests	RAW fixtures were converted repeatedly and NetCDF temporaries accumulated.	Check `Get-PSDrive -Name C` before large test groups.
Quicklook test fails in headless CI	Matplotlib backend is interactive or unset.	Set `MPLBACKEND=Agg`.
Distribution contains coordination files	Packaging include/exclude rules changed.	Run packaging tests and inspect sdist/wheel names.
SCC real submission fails	External server, credentials or measurement state issue.	First run `pytest tests\scc -q` to separate local client problems from external SCC problems.