Metadata-Version: 2.3
Name: nenuprepipe
Version: 0.1.0
Summary: Pre-processing pipeline runner for NenuFAR Cosmic Dawn data
License: GPL-3.0-or-later
Author: Florent Mertens
Author-email: florent.mertens@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Provides-Extra: dev
Requires-Dist: click (>=8.0)
Requires-Dist: nenucal (>=0.15)
Requires-Dist: pytest (>=8.0) ; extra == "dev"
Requires-Dist: toml (>=0.10)
Description-Content-Type: text/markdown

# nenuprepipe — NenuFAR Cosmic Dawn pre-processing pipeline

`prepipe` converts raw NenuFAR L1 data to pre-calibrated L2 data ready for
the [nenuflow](https://gitlab.com/cd-eor/nenuflow) calibration pipeline.

It wraps `nenudata`, `calpipe`, and `aostats` from the
[nenucal-cd](https://gitlab.com/cd-eor/nenucal-cd) package and runs one
spectral window at a time.

## Prerequisites

- `nenucal-cd >= 0.15`
- `aoquality >= 0.3`
- DP3
- AOFlagger

> **Note:** `losoto` installs the legacy `progressbar` package while nenucal
> requires `progressbar2`. They share the same import name, so depending on the
> installation order you may need to force `progressbar2` back:
>
> ```bash
> pip uninstall progressbar && pip install --force-reinstall progressbar2
> ```

## Setup

```bash
pip install git+https://gitlab.com/nenufar-cosmic-dawn/nenuprepipe.git

mkdir -p ~/nenufar-ops/my-field && cd ~/nenufar-ops/my-field
prepipe init --project-code RP3A
```

See `prepipe init` below for what `--project-code` fills in. Edit
`project.toml`/`data_handler.toml`/`cal/*` to adapt paths, node names, and
obs_id patterns to your site — every field ships with an inline comment.

Then verify with a dry run before actually running anything (drop
`--dry-run` once the printed plan looks right):

```bash
prepipe update project.toml --dry-run
prepipe run project.toml "202*EOR*" SW03 --dry-run
```

Logs are written to `logs/prepipe_<command>_<SW>_<timestamp>.log` and
printed to the terminal.

## Commands

### `prepipe init DIR`

Like `git init` — sets up the *current* directory (create it and `cd` there
first, it must be empty). Creates `project.toml`, `data_handler.toml`, and a
`cal/` directory with default parsets.

`--project-code` fills in first-guess L1/L2 data paths in `data_handler.toml`.
L1 (raw archive) and L2 (your own processed output) are separate roots,
since they're typically on different filesystems:

- L1 defaults to `/databf/nenufar-nri/{code}` (override with `--l1-root-template`)
- L2 defaults to `/data/{user}/{code}` (override with `--l2-root-template`)

`bp_cal_path` (bandpass solutions) is a plain relative path, not under either
root -- like `dppp_config` in `[l1_to_l2_config.*]`, it's resolved against
`data_handler.toml`'s own directory, not wherever you happen to run
`prepipe` from.

### `prepipe update PROJECT.toml`

Run this periodically (not tied to a particular SW or target) to keep the
data handler in sync with the archive:

- `update_data_handler` registers newly-recorded raw observations (N1 obs_ids)
  between `[update_data_handler].start`/`end` (`YYYYMMDD`) in `project.toml`.
- `update_n2_obs_ids` groups N1 obs_ids -- named `YYYYMMDD_HHMMSS_YYYYMMDD_HHMMSS_FIELD`
  -- by UT12-to-UT12 night and `FIELD` into one N2 obs_id per group, named
  `YYYYMMDD_FIELD` (`YYYYMMDD-2_FIELD`, `-3`, ... if more than one N1 obs_id
  falls on the same night for that field). `l1_to_l2` later splits each N2
  obs_id into fixed-length time chunks, round-robin distributed across the
  node pool assigned here. `prepipe run` and most `nenudata` commands
  process N2 obs_ids, not N1.

### `prepipe run PROJECT.toml TARGET_OBS_IDS SW`

Resolves each obs_id matching `TARGET_OBS_IDS` (for `SW`) to its calibrator
via `data_handler.toml`'s calibration map, calibrates it, then processes the
target.

`l1_to_l2`, `calpipe`, `quality_collect`, and `flag_bad_stations` skip
individual MSs/tasks already marked done by a previous run -- this is tracked
by nenudata/calpipe themselves (crash-safe, per-MS), not by prepipe. Use
`--force-calibrator`, `--force-target`, or `--force-all` on `prepipe run` to
reprocess them anyway.

**Calibrator steps** (run once per calibrator, source auto-detected from the
obs_id, e.g. `..._CASA_...` → `cal/casa_cal_bp.toml`):

| Step | Command | Description |
|---|---|---|
| `l1_to_l2` | `nenudata l1_to_l2` | Convert L1 → L2 via DP3 (flag + average) |
| `calpipe` | `calpipe cal/{source}_cal_bp.toml` | Bandpass calibration with DDECal |
| `make_bp_sol` | `nenudata make_bp_sol` | Average h5parms across SWs → combined bandpass solution |
| `plot_bp_sol` | `soltool plot-spectra` / `plot-delay` | Spectra + delay PDFs of the bandpass solution per obs → `bp_sol_plots/` |
| `quality_collect` | `nenudata quality-collect` | Run `aoquality collect` on every MS in parallel |
| `quality_combine` | `nenudata quality-combine` | Merge per-MS stats → one `quality_l2_cal/<SW>/<obs_id>.qs` per obs |
| `plot_quality` | `aostats plot` | Quality-stat PDF per obs from each combined `.qs` → `quality_l2_cal/<SW>/plots/` |
| `find_bad_stations` | `aostats find-bad-stations` | Flag outlier stations → `bad_stations_casa.json` |
| `import_bad_stations` | `nenudata bad-stations import` | Merge `bad_stations_casa.json` into the data handler registry |

**Target steps** (always processed for `TARGET_OBS_IDS`/`SW`; `only_n2` in
`[target]` restricts resolution to the N2 obs_ids created by `prepipe update`):

| Step | Command | Description |
|---|---|---|
| `l1_to_l2` | `nenudata l1_to_l2` | Convert L1 → L2_BP (apply bandpass, flag, average) |
| `quality_collect` | `nenudata quality-collect` | Run `aoquality collect` on every MS in parallel |
| `quality_combine` | `nenudata quality-combine` | Merge per-MS stats → one `quality_l2_bp/<SW>/<obs_id>.qs` per obs |
| `plot_quality` | `aostats plot` | Quality-stat PDF per obs from each combined `.qs` → `quality_l2_bp/<SW>/plots/` |
| `plot_grid` | `aostats plot-grid` | Diagnostic heatmaps (freq × obs, antenna × obs, LST × obs) |
| `flag_bad_stations` | `nenudata bad-stations flag` | Apply the data handler's bad-stations registry to the target MSs via DP3 |
| `push_l2` | `nenudata push_l2` | Rsync L2 data to the nenuflow server |

After `push_l2`, run `import-n2` on the nenuflow server (the exact command
is printed by `push_l2`):

```bash
ssh dawn
nenudata import-n2 /path/to/n2_obs_ids_push.json -c data_handler.toml
```

## Skipping steps

Set any step to `false` in the relevant `[calibrator.steps]`/`[target.steps]`
table of `project.toml`:

```toml
[target.steps]
quality_collect = false
quality_combine = false
```

## Troubleshooting

### On nancep (or similar shared machines)

Load DP3 and AOFlagger with `use DP3` and `use AOFlagger`.

They can hit default per-process limits when processing many MSs
concurrently. Raise these in your `.bashrc`:

```bash
ulimit -s 40960   # stack size
ulimit -u 40960   # max user processes
ulimit -n 40960   # max open files
```

## Directory layout

```
nenuprepipe/                          # this repo -- the tool, pip-installed
├── pyproject.toml
└── nenuprepipe/
    ├── cli.py                        # init / update / run entry point
    ├── steps.py                      # pipeline step implementations
    ├── tests/                        # pytest suite for steps.py
    └── templates/                    # copied by `prepipe init` into a fresh deployment
        ├── project.toml
        ├── data_handler.toml
        └── cal/
            ├── casa_cal_bp.toml      # calpipe config — CASA / CasA bandpass
            ├── cyga_cal_bp.toml      # calpipe config — CygA bandpass
            ├── casa_cal_rll.toml     # calpipe config — CasA with smoothed solutions
            ├── Ateam_LBA.skymodel    # A-team intrinsic sky model
            ├── dppp_l1_to_l2_bp.parset      # DP3 parset — L1 → L2_BP (apply + flag + average)
            ├── dppp_l1_to_l2_bp_interpolate.parset  # same with frequency interpolation
            ├── dppp_l1_to_l2_cyga.parset    # DP3 parset — L1 → L2_12C40S (calibrator, 40s)
            ├── dppp_l1_to_l2_12c20s.parset  # DP3 parset — L1 → L2_12C20S (alternative averaging, 20s)
            ├── nenufar_largebw_sens1.5_I.lua  # AOFlagger strategy — target data
            └── nenufar_largebw_sens2.5.lua    # AOFlagger strategy — calibrator data

~/nenufar-ops/my-field/                # a deployment, created by `prepipe init` -- separate
├── project.toml                       # your live config (gitignored if you version this dir)
├── data_handler.toml                  # your live data handler config (gitignored)
└── cal/                               # your own copy of the default parsets, freely editable
```

