Metadata-Version: 2.4
Name: sgnl-pastro
Version: 0.2.0
Summary: Tools for calculating astrophysical probability of GW events
Author: Victoria Niu, Leo Tsukada, Anarya Ray
Author-email: Chad Hanna <crh184@psu.edu>
License: GPL-2.0-or-later
Project-URL: Homepage, https://git.ligo.org/greg/sgnl-pastro
Project-URL: Issues, https://git.ligo.org/greg/sgnl-pastro/-/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: h5py
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: platformdirs
Requires-Dist: pyyaml
Requires-Dist: scipy
Requires-Dist: tqdm
Requires-Dist: gw-strike
Requires-Dist: igwn-ligolw
Requires-Dist: lalsuite
Requires-Dist: ligo-gracedb
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Provides-Extra: offline
Requires-Dist: gw-distributions; extra == "offline"
Provides-Extra: manifold
Requires-Dist: sgn-manifold; extra == "manifold"
Provides-Extra: lint
Requires-Dist: black<26,>=25; extra == "lint"
Requires-Dist: flake8; extra == "lint"
Requires-Dist: flake8-bandit; extra == "lint"
Requires-Dist: flake8-black; extra == "lint"
Requires-Dist: flake8-bugbear; extra == "lint"
Requires-Dist: flake8-future-annotations; extra == "lint"
Requires-Dist: flake8-isort; extra == "lint"
Requires-Dist: flake8-logging; extra == "lint"
Requires-Dist: flake8-pyproject; extra == "lint"
Requires-Dist: isort; extra == "lint"
Requires-Dist: mypy; extra == "lint"
Requires-Dist: mypy-extensions; extra == "lint"
Requires-Dist: types-PyYAML; extra == "lint"
Provides-Extra: dev
Requires-Dist: sgnl-pastro[lint]; extra == "dev"
Requires-Dist: sgnl-pastro[test]; extra == "dev"
Dynamic: license-file

# Pastro

Pastro (distributed as `sgnl-pastro`; the Python import package is still `pastro`) is a package to estimate the probabilities of astrophysical origins of the gravitational wave candidates detected by the GstLAL/SGNL pipeline, for the purpose of assisting in real-time multi-messenger follow-up observations. This repository develops and operates our p_astro model for the evaluation of p(category|data) with BNS, BBH, NSBH, and Terrestrial (noise) events.

For more detailed information on the operation manual of pastro, please find the information on **gwsci.org**: https://gwsci.org/ops/pastro 


## Install & Setup

All dependencies are pip-installable, so a plain pip install of the checkout is sufficient on any machine (laptop, ICDS, CIT, ...):

```
$ git clone https://git.ligo.org/greg/sgnl-pastro.git
$ cd sgnl-pastro
$ pip3 install .
```

If you plan to `git pull` updates, install in editable mode instead with `pip3 install -e .` — then the installed package tracks your checkout and you do not need to reinstall after every `git pull`. For development, see the [Development](#development) section below. The package can also be installed the same way inside a GstLAL/SGNL singularity container if you want it co-located with a running analysis, but no container is required.

Installing with pip also installs all of the command-line tools (e.g. `sgnl-pastro-initialize-fgmc-model`, `sgnl-pastro-calculate-fgmc`, ...) as console scripts on your `PATH`. All commands are namespaced under the `sgnl-pastro-` prefix; run `sgnl-pastro-<TAB><TAB>` to list them.

**Notice**: the package no longer imports `gstlal` or `manifold` — those dependencies were ported to `strike`/`igwn_ligolw` (and `cosmology_utils` was vendored into `pastro.cosmology_utils`). Some tools do, however, *read* ranking-statistic PDF files that may be produced by either GstLAL or the sgnl/strike stack: `sgnl-pastro-offline` and the `pastro.pastro` model methods load them through `pastro.io.load_rankstatpdf`, which **auto-detects** the producer by default. You can force the format with the `rankstat_format` config key for `sgnl-pastro-offline` (or the `fmt` argument of `pastro.io.load_rankstatpdf`). GstLAL is not required to be installed to read GstLAL-format files; only the input file itself is needed.

## FGMC method (O4 low-latency)

**Ray, Anarya. et. al. (2023)**, dcc-link: https://dcc.ligo.org/P2300141

This section walks through the **full preparation of the online (low-latency) FGMC p_astro**, from raw inputs to a deployable model file, with working commands. Three input artifacts feed the model build:

| Input | Produced by | Documented in |
|---|---|---|
| rates & sensitive volumes (`rates_V.json`) | `sgnl-pastro-calculate-rates` | [Rates_for_O4.md](Rates_for_O4.md) |
| template-weight files (`ceff*.h5`, one per category) | `manifold` (external) | [template-weights-fgmc-using-manifold.md](template-weights-fgmc-using-manifold.md) |
| ranking-statistic PDF (`*DIST_STAT_PDFS*.xml.gz`) | the running analysis | — |

If you just want to see the whole chain run right now, with no cluster access or real analysis products, skip to **Step 6: runnable quickstart** — `examples/fgmc/online/` generates synthetic stand-ins for all of the inputs and runs Steps 4–5 verbatim.

### Step 1: estimate rates and sensitive volumes

The expected astrophysical rates and the sensitive volumes scaled to the new run are computed by `sgnl-pastro-calculate-rates` from the previous runs' FGMC rate estimates, BNS ranges, run times, and recovered hypervolumes. The full procedure and the provenance of every number is documented in [Rates_for_O4.md](Rates_for_O4.md); the O4 invocation is:

```
sgnl-pastro-calculate-rates \
    --rates-astro '{"BBH": 40.0, "NSBH": 68.0, "BNS": 163.0}' \
    --VT-old '{"BBH": 1.63, "NSBH": 0.048, "BNS": 0.014}' \
    --dbns-old '{"O1": 0.08, "O2": 0.1, "O3": 0.12}' \
    --runtime-old '{"O1": 0.333333333333, "O2": 0.75, "O3": 1.0}' \
    --dbns-new 0.160 \
    --threshold-old 8760.12 \
    --terrestrial-old 11541 \
    --T-inj-start 1262304018 --T-inj-stop 1265760018 \
    --N-inj '{"BBH": 118036.67, "BNS": 118036.67, "NSBH": 118036.67}' \
    --z-max '{"BBH": 1.9, "BNS": 0.15, "NSBH": 0.25}' \
    --output rates_V_O4.json
```

The output JSON holds three dictionaries: `rates_astrophysical` (→ `--rates` in Step 4), `rates_injections` (→ `--rates-inj`; deprecated for O4, see the note in [Rates_for_O4.md](Rates_for_O4.md)) and `V` (→ `--V-new`). The `--T-inj-*`/`--N-inj`/`--z-max` options only affect the deprecated injection rates.

### Step 2: build the template weights

The per-category template weights encode p(template, SNR | category) and are produced from the mass model (with NS-boundary mass cuts) by the external [manifold](https://git.ligo.org/chad-hanna/manifold) tool — condor DAG steps documented in [template-weights-fgmc-using-manifold.md](template-weights-fgmc-using-manifold.md).

**Naming requirement:** the model loader globs `ceff*.h5` in `--weights-dir` and identifies each file's category by substring match on the filename (e.g. `ceff_BBH.h5` → `BBH`). Manifold's outputs are named like `H1L1V1-O4_MANIFOLD_BANK_SALPETER_BBH_-0-2000000000.h5`, so **rename (or symlink) them to `ceff_<CATEGORY>.h5`** when assembling the weights directory. Each file must contain the `coefficients`, `SNR` and `template_id` datasets.

### Step 3: obtain a ranking-statistic PDF

The likelihood-ratio normalization constant `A` is computed from the analysis' ranking-statistic PDF file (e.g. `H1L1V1-GSTLAL_DIST_STAT_PDFS-0-0.xml.gz` from the online run's `dist_stat_pdfs` directory). Both GstLAL-format and sgnl/strike-format documents are read; the producer is auto-detected (see the format notice in the install section above). This input is technically optional (`A` defaults to 1.0 without it) but required for a correctly normalized production model.

### Step 4: build the model file

With the three inputs in hand, build the model `.h5` (values below are the O3-era production numbers from `examples/fgmc/test_model/Makefile`; the FAR threshold is in expected counts per Julian year — 730.5 is 2 per day):

```
sgnl-pastro-initialize-fgmc-model \
    --rankstat-filename /path/to/H1L1V1-GSTLAL_DIST_STAT_PDFS-0-0.xml.gz \
    --weights-dir /path/to/template-weights \
    --rates '{"BBH": 18.0730098234, "NSBH": 52.9087878433, "BNS": 662.448825556}' \
    --rates-inj '{"BBH": 4045.100324417181, "BNS": 1086074.923130263, "NSBH": 269801.54621791927}' \
    --V-new '{"BBH": 3.720777380922114, "NSBH": 0.13036984478340222, "BNS": 0.01447408360378373}' \
    --far-threshold 730.5 \
    --output fgmc_model.h5
```

The `--rates`/`--rates-inj`/`--V-new` dictionaries must all cover the same category set. Optionally pass `--template-database <bank.sqlite>` to take the template-id mapping from a template-bank database instead of the weight files.

### Step 5: validate the model and evaluate events

Evaluate a **single trigger** from hand-entered numbers (template id, network SNR, ln likelihood ratio):

```
sgnl-pastro-calculate-fgmc --fgmc-model fgmc_model.h5 --templateid 7012 --snr 14.487 --lnlr 52.619
{"BBH": 0., "BNS": 0.9999929320428521, "NSBH": 4.1997623780371124e-16, "Terrestrial": 0.}
```

Evaluate **LIGO_LW coinc documents** (e.g. the coinc.xml files the online pipeline uploads to GraceDB) with `sgnl-pastro-calculate-fgmc-coinc`:

```
sgnl-pastro-calculate-fgmc-coinc --fgmc-model fgmc_model.h5 coinc1.xml coinc2.xml.gz ...
```

For every coincident event in every input file this prints one JSON object per line containing the trigger data (end time, combined FAR, SNR, ln LR, template id) and the evaluated `p_astro`. The trigger inputs are read from the standard tables: `likelihood` from `coinc_event`, network `snr`/`end_time`/`combined_far` from `coinc_inspiral`, and the template id from the `Gamma0` column of the first linked `sngl_inspiral` row (override with `--template-id-column` if your documents store it elsewhere). Pass `--far-threshold <Hz>` to skip events whose `combined_far` is above the given rate, mirroring the online FAR cut.

The same tool evaluates **any p_astro model file** with `--model` in place of `--fgmc-model`: the numbered models of this package, or an **sgn-manifold** combined model (`sgn-manifold-cbc-p-astro-combine`, stamped with its runtime normalization by `sgn-manifold-cbc-p-astro-configure`; needs `pip install sgnl-pastro[manifold]`). That model classifies each event from its template id and `combined_far` alone (`likelihood`/`snr` are ignored), returning the same `{BNS, NSBH, BBH, Terrestrial}` JSON, so the two models can be compared event by event on the same documents:

```
sgnl-pastro-calculate-fgmc-coinc --model pastro-model.h5 coinc1.xml coinc2.xml.gz ...
```

`pastro.load(filename)` performs the same dispatch for programmatic use (SGNL's online uploader loads models this way).

To compare against a **GraceDB MDC injection campaign** (requires a valid X.509 proxy and network access to GraceDB):

```
sgnl-pastro-calculate-fgmc-test-inj --output-file pastro_inj.txt --fgmc-model fgmc_model.h5 --injection-file injections.xml.gz
```

Production-style Makefiles with the real cluster paths for Steps 4–5 live in `examples/fgmc/Makefile` and `examples/fgmc/test_model/Makefile`.

### Step 6: runnable quickstart (no cluster access needed)

`examples/fgmc/online/` contains a self-contained version of Steps 4–5: `make_demo_inputs.py` generates synthetic template weights (`ceff_*.h5`) and a two-event `coinc.xml.gz`, and the `Makefile` builds a model from them and evaluates both a hand-entered trigger and the coinc document. The test suite runs this example end-to-end (`tests/test_example_fgmc_online.py`), so it is guaranteed to work:

```
$ cd examples/fgmc/online
$ make
...
cat single_trigger.json
{"BBH": 0.016503467977679633, "BNS": 0.9493390523474371, "NSBH": 0.03400160750475215, "Terrestrial": 0.00015587217013113275}
...
cat coinc_results.jsonl
{"template_id": 100.0, "likelihood": 12.0, "snr": 9.0, "end_time": 1370000000.25, "combined_far": 1e-10, "file": "coinc.xml.gz", "p_astro": {"BBH": 0.016503467977679633, "BNS": 0.9493390523474371, "NSBH": 0.03400160750475215, "Terrestrial": 0.00015587217013113275}}
{"template_id": 300.0, "likelihood": 1.5, "snr": 6.5, "end_time": 1370000100.0, "combined_far": 1e-05, "file": "coinc.xml.gz", "p_astro": {"BBH": 0.5512015358585471, "BNS": 9.699276100104307e-06, "NSBH": 0.00038095949183503054, "Terrestrial": 0.44840780537351776}}
```

The loud, low-FAR event in the BNS-weighted template comes out confidently astrophysical (and identical to the hand-entered evaluation of the same numbers — the two tools share the model math); the marginal, high-FAR event splits between BBH and Terrestrial. The demo model omits `--rankstat-filename` (no analysis behind it, so `A = 1`); everything else is exactly the production path.

### Step 7: deploy for online running

Handing the model file to the running low-latency analysis is an operations procedure, documented at https://gwsci.org/ops/pastro.


## Development

For development, install the package in editable mode together with the test and lint dependencies:

```
$ pip3 install -e ".[dev]"
```

The top-level `Makefile` provides the standard development targets:

```
$ make format      # run isort and black
$ make lint        # run flake8
$ make type-check  # run mypy
$ make test        # run pytest
$ make all         # run all of the above
```

