Metadata-Version: 2.4
Name: asimov-ptadata
Version: 0.1.0
Summary: Pulsar timing array data acquisition and reduction, with an Asimov pipeline plugin
Author-email: Daniel Williams <daniel.williams@ligo.org>
License: MIT License
        
        Copyright (c) 2026 Daniel Williams
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Source code, https://github.com/transientlunatic/asimov-ptadata
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Natural Language :: English
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pint-pulsar
Requires-Dist: numpy
Requires-Dist: astropy
Requires-Dist: pyyaml
Requires-Dist: click
Provides-Extra: asimov
Requires-Dist: asimov<0.8,>=0.7; extra == "asimov"
Provides-Extra: noise
Requires-Dist: asimov<0.8,>=0.7; extra == "noise"
Requires-Dist: enterprise-pulsar; extra == "noise"
Requires-Dist: six; extra == "noise"
Requires-Dist: ptmcmcsampler; extra == "noise"
Requires-Dist: fastshermanmorrison-pulsar; extra == "noise"
Provides-Extra: docs
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: kentigern>=0.3.0; extra == "docs"
Requires-Dist: numpydoc; extra == "docs"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Dynamic: license-file

# asimov-ptadata

Pulsar timing array data acquisition and reduction, usable standalone via the
`ptadata` CLI or as an [Asimov](https://git.ligo.org/asimov/asimov) pipeline
plugin.

`ptadata` locates a pulsar's `.par`/`.tim` files within a pinned checkout of a
PTA data release - either a known release cloned automatically by name, or a
local checkout you already have - then uses
[PINT](https://github.com/nanograv/PINT) to validate and reduce them:
flagging outlier TOAs, checking for clock/ephemeris/binary-model warnings,
optionally running a basic timing-model refit, and writing cleaned
`.par`/`.tim` files plus a machine-readable QC report.

## Standalone use

```console
$ ptadata list-releases
$ ptadata fetch --pulsar J1909-3744 --release-name ipta-dr2 --outdir ./staged
$ ptadata reduce --par ./staged/J1909-3744/J1909-3744.IPTADR2.TDB.par \
    --tim ./staged/J1909-3744/J1909-3744.IPTADR2.tim --outdir ./reduced
```

or against a data release you already have a local checkout of:

```console
$ ptadata fetch --pulsar J1713+0747 --release /path/to/release/checkout --outdir ./staged
```

`ptadata reduce` exits non-zero if the QC report's status isn't `pass`, so it
composes into shell pipelines / CI.

`fetch` copies a pulsar's whole directory when the release lays data out
per-pulsar (checked against a live IPTA DR2 clone: a top-level `.tim` file
commonly `INCLUDE`s per-backend files from a `tims/` subdirectory by relative
path, so that subdirectory has to move with it, not be flattened). When more
than one `.par` candidate is found - real releases often ship a plain file
alongside a unit-conversion variant - the manifest's `"par candidates"` lists
all of them and `"par"` prefers a TDB-convention file, since PINT refuses to
load `UNITS TCB` par files outright (the plainer-looking filename may be the
one that fails).

### Normalisation of staged `.tim` files

tempo2 and PINT read three `.tim`-file conventions differently, and IPTA DR2
relies on both, so `fetch` rewrites them in the **staged copies** (every
`.tim` file, including `INCLUDE`d per-backend files). The release checkout
itself is never modified.

- **TOAs commented out with a `C` prefix.** tempo2 treats any line starting
  with an upper-case `C` as a comment, and DR2 comments out TOAs by
  prefixing them directly (`C???? c015621.align...`, `Cc054887.align...`,
  `C200404522.bb ...`). PINT only recognises `C `, `c `, `CC ` and `#`, so
  it either silently re-includes the TOA (J1713+0747 had 7 of these) or
  fails to parse the file (J1744-1134). These lines are prefixed with
  `C ` so PINT reads them as comments too. A lower-case `c` is left alone:
  to tempo2 it's an ordinary TOA whose archive name starts with "c".
- **Flags with no value** (`... -projid -beconfig -snr 70.99`). tempo2
  accepts these; PINT pairs flag tokens two at a time, so it silently takes
  the next flag's name as the value, and with an odd number of them it
  shifts every later flag or fails to parse (J1824-2452A, J2033+1734). A
  valueless flag carries no information, so it is dropped.
- **`-addsat` arrival-time corrections** (`... -addsat -1`). tempo2 adds
  the flag's value, in seconds, to the site arrival time. DR2 uses it on 39
  active Effelsberg TOAs in 17 pulsars (+1, -1 or -2 s) to correct
  timestamp errors. PINT
  ignores it, so those TOAs are a second out, which shows up as residuals of
  up to half a pulse period (1.8 ms for J1744-1134, raising its post-fit
  RMS from 2.7 to 30.6 µs). It is replaced with PINT's `-to` time offset
  (seconds, applied with the clock corrections), summed with any `-to`
  already on the line.

All three rules were checked against tempo2 itself: after normalisation PINT
loads exactly as many TOAs as tempo2 does from the original files for
J1713+0747 (17487), J1744-1134 (9834), J1824-2452A (276) and J2033+1734
(194), and a converted `-addsat -1` moves a TOA by exactly the 1.0000 s
tempo2 moves it. Across DR2 VersionB, 203 commented TOAs in 18 pulsars and 2587
lines with valueless flags in 17 pulsars are affected. `fetch`'s manifest
records what was changed under `tim normalisation`, and `ptadata run`
copies it into the QC report's `notes` (it doesn't change the QC status).

### Outlier clipping

By default `reduce` moves TOAs whose residual lies more than `sigma
threshold` (5.0) robust standard deviations (MAD-based) from the median into
a `.quarantine.tim` file. These are **pre-fit** residuals of the par file's
timing model alone. Any red or DM noise that model doesn't carry is still in
them, and its excursions get clipped as if they were bad TOAs. That includes
IPTA DR2's TempoNest noise parameters (`TNSUBTRACTDM`, `TNRedAmp`, ...),
which PINT ignores. Those excursions are concentrated at low observing
frequency and towards the ends of the data span.

For IPTA DR2 this removed 8.9% of J1713+0747's TOAs: 725 of 4020 at 800 MHz
(GUPPI), and 14.6% of the earliest fifth of the span against 3.3% of the
middle fifth. That would bias any analysis sensitive to low-frequency or
long-timescale structure: noise fits, and common-signal searches such as a
GWB or a secular trend. For a release that has already been cleaned by its
collaboration, turn clipping off and leave the noise to the noise model:

```yaml
reduction:
  clip outliers: false   # or `ptadata reduce --no-clip`
```

The QC report then notes `outlier clipping disabled` and flags no TOAs.
When clipping is on and removes every TOA a JUMP (or other mask parameter)
selects, the refit now freezes that parameter instead of failing.

## As an Asimov pipeline

Installing with the `asimov` extra (`pip install asimov-ptadata[asimov]`)
registers a `ptadata` entry in the `asimov.pipelines` group. A production
using it is driven by a settings file rendered from `production.meta`:

```yaml
kind: analysis
name: reduce
pipeline: ptadata
status: ready
data:
  release name: ipta-dr2   # or: release root: /path/to/release/checkout
reduction:
  sigma threshold: 5.0
  refit: true
  clip outliers: true   # see "Outlier clipping" above
```

The plugin submits a single `ptadata run --settings <file>` job through
Asimov's scheduler-agnostic `JobDescription` interface (works with both the
HTCondor and Slurm schedulers), and on completion exposes the reduced
`.par`/`.tim` files and QC report as production assets - `qc_report.yml`'s
`status` field (`pass` / `needs-review`) is written to the production's
`review status` metadata as a review gate ahead of any downstream noise/GWB
analysis.

### Noise-fit sampling and convergence

The single-pulsar noise fit (`ptadata-noise` / `asimov_ptadata.noise_fit`)
samples per-backend white noise plus power-law red/DM noise with
`PTMCMCSampler`. A single-stage run over every parameter at once, started
from a random prior draw, is not reliable: on IPTA DR2, J1713+0747's red
and DM noise (a 121-parameter joint fit) stayed about 4 dex below their DR2
(`TempoNest`) values after 100000 iterations - log10 A of -17.95 against
-14.21 - with effective sample sizes of only 3-30. That unmodelled noise
then leaked into the downstream fixed-noise jerk search as an 11 sigma
line-of-sight artefact.

**By default the white noise is fixed at its MAP** (`white noise: map`).
Each backend's EFAC/EQUAD/ECORR and the red/DM hyperparameters are
maximised in turn (bounded Powell) until a round improves the log
posterior by less than 0.1, and then only the red/DM noise is sampled
(stage 2 below). Sampling the white noise as well did not mix for large
models: J1713+0747's ~117 white-noise parameters had an ESS of ~10 after
20000 iterations, while the red/DM noise came out the same either way
(log10 A within ~0.05). The MAP ignores white-noise uncertainty, as PTA
noise analyses commonly do; `white noise: sample` samples it instead:

With `white noise: sample` the fit is two-stage. Stage 1 samples every parameter,
started from an **optimised** point rather than a prior draw (white noise
at nominal values; red/DM hyperparameters at a bounded-Powell likelihood
maximum), and proposes in **parameter groups** (every parameter; each
backend's white noise; the red pair; the DM pair; red+DM; all white noise)
so the few red/DM hyperparameters can move on their own. Stage 2 then
re-samples just the red/DM hyperparameters with white noise fixed at its
stage-1 posterior mean - four parameters mix far better alone than inside
the full model, and they're what the fixed-noise GWB/jerk searches depend
on most.

Each report now carries a `converged` verdict, from whichever stage
decided the red/DM values: every one of those parameters must reach an
effective sample size of at least `min ess` and a split-half (first- vs
second-half mean) shift of at most `max split shift` posterior standard
deviations. `ptadata-noise`'s Asimov pipeline only auto-approves a review
when *every* subject's report has `converged: true` - an unconverged fit,
or one from before this field existed, is left for a human instead of
silently carrying bad noise values downstream.

```yaml
sampler:
  niter: 20000              # stage 1
  burn: 5000                # 2000 was too short: stage 1 was still climbing
  white noise: map          # or sample (stage 1 below)
  two stage: true
  stage 2 niter: 20000      # defaults to niter
  optimise start: true
  min ess: 200
  max split shift: 0.3
  red noise components: 120 # see below
  dm noise components: 120
```

The number of Fourier components matters as much as the sampler. With
30, J1713+0747's sharp DM events could not be modelled, and the fit settled
on a flat (gamma ~ 0.6) achromatic process at log10 A ~ -12.4 to absorb them.
With 120 (the `TNRedC`/`TNDMC` IPTA DR2 itself used for this pulsar), the
likelihood peak moves to red noise log10 A = -14.0, gamma = 3.0, close to
DR2's own -14.2, 3.5. The same counts must then be used by the fixed-noise
GWB and jerk searches.

Building the PTA also picks up `fastshermanmorrison-pulsar` (part of the
`noise` extra) automatically, which made J1713+0747's likelihood evaluation
about 1.6x faster (84ms vs 132ms) - worth having given how many likelihood
calls a sampler like this makes.
### GWB search: fixed spectral index and constrained amplitude

`ptadata-gwb` fits one Hellings-Downs-correlated common process across the
array, with every pulsar's noise held fixed at its `ptadata-noise` values.
Its spectral index is fixed at 13/3 (a background from circular,
GW-driven supermassive black-hole binaries) unless the analysis sets
`sampler: gwb gamma:` to another value, or to `null` to sample it (the
`gwb-fit` CLI takes `--gwb-gamma` / `--sample-gwb-gamma`). With few
pulsars, a sampled index just returns its prior.

`gwb_report.yml` records the fixed values (`fixed`), 5/50/95% summaries of
the sampled parameters (`summary`), and `amplitude_constrained`: whether
the posterior SD of `gwb_log10_A` is below half its prior SD. When it isn't,
the posterior mean mostly reflects the U(-20, -11) prior. The 5-pulsar IPTA
DR2 pilot gave -17.4 +/- 1.6, and a downstream search shouldn't treat that
as a measured background.

## Known releases

`ptadata list-releases` prints the current registry (`asimov_ptadata/sources.py`).
IPTA DR2, EPTA DR2, and InPTA DR2 are git repositories and fetched with a
shallow clone. Every URL was checked reachable with `git ls-remote` and the
IPTA DR2 entry checked against a real clone while building this - but
collaborations do restructure/move these between releases, so treat the
registry as a convenience starting point and check the collaboration's own
data page if a lookup starts failing.

**Not yet supported**: NANOGrav's data sets are Zenodo archives, not git
repositories (e.g. the 15yr set, DOI `10.5281/zenodo.16051178`, is a single
~640MB tarball served via the Zenodo REST API), so they need a
download-and-extract path rather than `clone_release`. Not implemented yet.

## Status

Real IPTA DR2 data (`J1909-3744`, 11,483 TOAs across 11 backends) fetches
and reduces successfully end-to-end, including a real T2-to-ELL1 binary
model conversion correctly landing the QC status on `needs-review`. Notably
not yet solved: NANOGrav/Zenodo fetching (above), a `--version`/combination
selector for releases that ship parallel data combinations (IPTA DR2 has
`VersionA`/`VersionB` - point `--release` at the specific one you want), and
pre-staging clock correction files for HPC worker nodes without outbound
internet access (PINT will otherwise try to fetch missing clock files from
the IPTA clock-corrections repository on first use).

For such nodes, each pipeline passes a `scheduler: environment:` mapping in
its metadata to the job (HTCondor's `environment`, or Slurm's `--export`),
so jobs can be pointed at clock files and ephemerides staged elsewhere, e.g.:

```yaml
scheduler:
  environment:
    PINT_CLOCK_OVERRIDE: /shared/cache/clock
    XDG_CACHE_HOME: /shared/cache/astropy_cache_home
```
