Metadata-Version: 2.4
Name: nenuprepipe
Version: 0.4
Summary: Pre-processing pipeline runner for NenuFAR Cosmic Dawn data
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Author: Florent Mertens
Author-email: florent.mertens@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: dev
Requires-Dist: aoquality (>=0.3)
Requires-Dist: click (>=8.0)
Requires-Dist: nenucal (>=0.29)
Requires-Dist: pytest (>=8.0) ; extra == "dev"
Requires-Dist: pytest-cov (>=5.0) ; extra == "dev"
Requires-Dist: ruff (==0.16.1) ; extra == "dev"
Requires-Dist: tabulate (>=0.9)
Requires-Dist: toml (>=0.10)
Project-URL: Homepage, https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe
Project-URL: Issues, https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe/-/issues
Project-URL: Repository, https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe
Description-Content-Type: text/markdown

# NenuPrePipe

NenuPrePipe prepares raw NenuFAR Cosmic Dawn observations for calibration with
[NenuFlow](https://gitlab.com/nenufar-cosmic-dawn/nenuflow). Starting from L1
data, it registers observations, associates targets with calibrators, creates
quality-checked L2 MeasurementSets, identifies bad stations, applies the
resulting flags, and optionally transfers the data to the NenuFlow server.

The intended flow is:

```text
raw L1 archive
    → register observations and nights
    → calibrate the bandpass calibrator and find its bad stations
    → create bandpass-corrected target L2 data
    → apply station flags
    → inspect quality statistics
    → transfer to the NenuFlow server
```

NenuPrePipe orchestrates tools supplied by
[NenuCal](https://nenucal-cd.readthedocs.io) and
[AOQuality](https://pyaoquality.readthedocs.io). Use their documentation when
you need lower-level control over data registration, calibration, transfers,
quality statistics, or manual antenna decisions.

## Installation

NenuPrePipe requires Python 3, `nenucal >= 0.29`, `aoquality >= 0.3`, DP3,
and AOFlagger.

```bash
pip install nenuprepipe
# or the development version
pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe.git
```

On shared machines, make sure DP3 and AOFlagger are available in the shell in
which `prepipe` runs.

> `losoto` installs the legacy `progressbar` package, while NenuCal requires
> `progressbar2`. If the two conflict, restore the latter with:
>
> ```bash
> pip uninstall progressbar
> pip install --force-reinstall progressbar2
> ```

## Quick start

### 1. Create a project directory

Keep the pre-processing and NenuFlow configuration for a field together:

```bash
mkdir -p ~/nenufar-ops/my-field
cd ~/nenufar-ops/my-field
prepipe init --project-code RP3A
```

This creates:

```text
project.toml       prepipe workflow and step selection
data_handler.toml  observation locations, levels, nodes, and calibration map
cal/               calibrator configuration files (calpipe) and sky model
```

`--project-code` fills in the project code in the L1 archive path
(`/databf/nenufar-nri/CODE/...`) and an L2 root under `/data/$USER/CODE`. Then
review the generated files:

`data_handler.toml`

- `[data_levels]`: the `path` of `L1` must match the archive layout of your
  observations, those of `L2_BP` and `L2_12C40S` point to where you want your
  processed data. Their other keys say how they are made from `L1`: bandpass,
  flagging, averaging, and the time chunks of the N2 nights (`split`, 180 time
  slots). See the [NenuCal `[data_levels]` reference](https://nenucal-cd.readthedocs.io/en/latest/nenudata/config.html).
- `[spectral_windows]`: subband ranges of your observation setup (default
  `SW01`–`SW04`, subbands 192–433).
- `cal_obs_id_patterns`: calibrator obs_id patterns, if not only CasA and CygA.
- `beam_corr = true` in `[data_levels.L2_BP]` applies the beam correction
  (see [Target](#target)).
- `[remote_hosts.dawn]`: only needed with `push_l2`.

`project.toml`

- `[update_data_handler]`: `node` assigned to the newly registered
  observations (where their DP3 jobs run, and `%NODE%` in their paths), and
  the `start`/`end` dates (YYYYMMDD) of the observations to register.
- `[update_n2_obs_ids]`: `node`(s) storing the L2 data of the N2 nights.
- `[target.steps]`: `make_beam_corr` (off by default, together with
  `beam_corr` above) and `push_l2` (off by default, see step 6).
- `[target.make_beam_corr]`: `beamsquint_freq`, the `optFrq` of your
  observation parsets.
- `[target.push_l2]`: `remote`, matching a `[remote_hosts.*]` entry.

Relative calibration paths are resolved from the directory containing
`data_handler.toml`.

Projects created before nenuprepipe 0.4 keep working: their `[data_level_path]`
and `[l1_to_l2_config]` are still read by NenuCal, and a `n_slots_per_chunk` in
`[target.l1_to_l2]` is still used, with a warning to move it into `split`.

For a full description of the data-handler fields, see the
[NenuCal data-management documentation](https://nenucal-cd.readthedocs.io/en/latest/nenudata/index.html).

### 2. Register new observations

```bash
prepipe update project.toml --dry-run
prepipe update project.toml
```

`update` scans the configured L1 archive over the date range in
`project.toml`, registers new raw observations, and groups them into N2 night
identifiers such as `20231208_NT04`. Most later commands operate on these N2
identifiers rather than the original short observations.

Run `update` whenever new data arrive or the configured date range changes.

### 3. Preview the run

```bash
prepipe run project.toml "20231208_NT04" SW03 --dry-run
```

Check the selected targets, calibrators, paths, and commands before starting the
real run.

Several targets can be processed in one run, with a glob pattern or a
comma-separated list. Each target brings its own calibrator, so the run then
processes all their calibrators too: every calibrator step runs once over all
of them (`calpipe` once per calibrator source), then the target steps run over
all targets. `SW` is a single spectral window name, such as `SW03`; run each
spectral window separately.

```bash
prepipe run project.toml "202312*_NT04" SW03 --dry-run
prepipe run project.toml "20231208_NT04,20231210_NT04" SW03 --dry-run
```

### 4. Run the pre-processing

```bash
prepipe run project.toml "20231208_NT04" SW03
```

The run first prepares the associated calibrators, derives their bandpass
solutions, and identifies their bad stations. It then creates the target L2
data, applies those station flags, collects and combines quality statistics,
produces diagnostic plots, and pushes the selected L2 data if transfer is
enabled.

Completed per-MS tasks are recorded by NenuCal and skipped safely on a later
run. Use `--force-calibrator`, `--force-target`, or `--force-all` only when the
corresponding products genuinely need to be rebuilt.

Each run ends with a summary of every step (done, skipped, failed, or not run)
and the status table of its targets and calibrators. If a task fails, for
example a DP3 conversion killed for lack of memory, its step lists the failed
tasks with their log files and the run stops there with a non-zero exit status.
Once the cause is fixed, rerun the same command: only the failed and remaining
work is done.

To see how far each observation has got, without running anything:

```bash
prepipe status project.toml "20231208_*"            # summed over SWs
prepipe status project.toml "20231208_*" --per-sw   # one row per SW
```

### 5. Review quality and antenna flags

Inspect the calibrator and target plots written below `quality_l2_cal/` and
`quality_l2_bp/`, including the multi-observation grids. Confirm that the
automatically selected stations are plausible:

```bash
nenudata bad-stations show "20231208_NT04" -c data_handler.toml
nenudata bad-stations show-history -c data_handler.toml
```

Use `nenudata bad-stations add`, `remove`, or `clear` for reviewed manual
changes. NenuCal keeps an audit history. If the registry changes, the next
`prepipe run` reflags the affected targets, but keeps their quality statistics;
rerun with `--force-target` to recompute them on the new flags.

For interpreting plots or running additional diagnostics, see the
[AOQuality guide](https://pyaoquality.readthedocs.io/en/latest/aostats.html).
For registry behavior and propagation from calibrator to target, see
[NenuCal bad stations](https://nenucal-cd.readthedocs.io/en/latest/nenudata/bad_stations.html).

### 6. Continue with NenuFlow

NenuFlow can run on the same server and use the prepared L2 data directly. In
that case, keep `push_l2` disabled (the default), create the MS list from the
current `data_handler.toml`, and start NenuFlow locally.

If NenuFlow runs on another server, enable `push_l2` in `project.toml` and
configure the matching `[remote_hosts.*]` entry in `data_handler.toml`. It
transfers the data and prints the exact import command to run on the
destination server:

```bash
nenudata import-n2 /path/to/n2_obs_ids_push.json -c data_handler.toml
```

On either host, the registered L2 observations can then be turned into an MS
list and processed by NenuFlow. The NenuFlow README contains the complete
one-night example for both arrangements.

Once a night is calibrated and checked, `nenudata make L3` writes its residual
into a compressed L3 level, and `nenudata remove L2_BP` frees the L2 while
keeping the NenuFlow solutions and sky models next to it (see
[NenuCal pipeline](https://nenucal-cd.readthedocs.io/en/latest/nenudata/pipeline.html)).

## What `prepipe run` does

The workflow is configured through the `[calibrator.steps]` and
`[target.steps]` tables in `project.toml`.

### Calibrator

1. Convert raw L1 data to an averaged calibrator L2 level (`nenudata make`, as
   defined by the level in `data_handler.toml`).
2. Run `calpipe` and combine the bandpass solutions.
3. Plot solution spectra and delays.
4. Collect and combine AOQuality statistics.
5. Plot those statistics and identify bad stations.
6. Import the reviewed station list into the data-handler registry.

### Target

1. Convert L1 to bandpass-corrected L2 data (`nenudata make`, as defined by the
   level in `data_handler.toml`).
2. Apply the registered bad-station flags.
3. Collect and combine AOQuality statistics on the flagged data.
4. Produce per-observation and cross-observation diagnostic plots.
5. Transfer the prepared data to the NenuFlow server when requested.

The bandpass carries the mini-array gain towards the calibrator, so the target
keeps its own gain relative to it, which varies over the night with elevation.
To correct it during the L1 to L2 conversion, enable `make_beam_corr` in
`[target.steps]` and set `beam_corr = true` in `[data_levels.L2_BP]` of
`data_handler.toml`; `prepipe run` warns when only one of the two is set. The
correction assumes a perfect analog pointing and leaves errors of a few percent
for the target calibration to absorb. Each correction is
plotted to `beam_corr_plots/<obs_id>_<sw>_beam_corr.pdf`.

Set an individual step to `false` when it is intentionally handled elsewhere:

```toml
[target.steps]
quality_collect = false
quality_combine = false
```

## Command summary

```text
prepipe init [--project-code CODE]         create a deployment in the current
                                           (empty) directory
prepipe update PROJECT.toml                register observations and N2 nights
prepipe run PROJECT.toml OBS_IDS SW        prepare calibrator and target data
prepipe status PROJECT.toml [OBS_IDS]      show per-obs_id processing progress
```

Use `--dry-run` with `update` and `run` to inspect the planned actions. Logs are
written under `logs/` and also printed to the terminal. Run `prepipe COMMAND
--help` for all options.

## Troubleshooting shared machines

On nancep or similar hosts, load DP3 and AOFlagger using the local environment
commands (for example, `use DP3` and `use AOFlagger`). Large parallel runs may
also require higher per-process limits:

```bash
ulimit -s 40960
ulimit -u 40960
ulimit -n 40960
```

