Metadata-Version: 2.4
Name: rimsky
Version: 0.2.0rc1
Summary: Orchestrator for automated online gravitational-wave parameter estimation with bilby
Author-email: Colm Talbot <colm.talbot@ligo.org>
Maintainer-email: Colm Talbot <colm.talbot@ligo.org>
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: asimov>=0.8.1
Requires-Dist: bilby>=2.8.2
Requires-Dist: bilby_pipe==1.10.1
Requires-Dist: graphviz>=0.21
Requires-Dist: igwn-alert>=0.7.0
Requires-Dist: ipython
Requires-Dist: ligo-gracedb
Requires-Dist: ligo.skymap
Requires-Dist: pesummary
Requires-Dist: pip!=26.1.2
Requires-Dist: plotly
Requires-Dist: pydantic-settings>=2.7
Requires-Dist: pytest-cov
Requires-Dist: pyyaml>=6.0
Requires-Dist: sgn-ligo>=0.7.0
Requires-Dist: sgn-ts>=0.11.0
Requires-Dist: watchdog<7
Provides-Extra: dev
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: test
Requires-Dist: ruff>0.1.0; extra == "test"
Dynamic: license-file

<img src="rimsky.png" alt="Rimsky" height="100">

# Rimsky - an orchestrator for automated online parameter estimation

![pipeline](https://git.ligo.org/colm.talbot/rimsky/badges/main/pipeline.svg)
![coverage](https://git.ligo.org/colm.talbot/rimsky/badges/main/coverage.svg)

Rimsky streams detector strain, estimates PSDs, keeps rolling caches of
frames and PSDs, listens for GraceDB alerts, collects the data around each
significant trigger and prepares (and optionally submits) `bilby_pipe`
analyses, then watches for their results, reduces the posterior samples for
upload to GraceDB, and publishes summary pages. See the
[documentation](docs/index.md) for an overview of the pipeline stages.

## Usage

Run the pipeline with a YAML configuration file:

```bash
cd src/rimsky/examples
uv run python -m rimsky example.yaml
```

The example configurations are in [src/rimsky/examples](src/rimsky/examples)
(also linked as `examples/`) and are installed with the package:

- [example.yaml](src/rimsky/examples/example.yaml) streams simulated noise
  and writes the frame and PSD caches, the strain part of the pipeline.
- [pipeline_test.yaml](src/rimsky/examples/pipeline_test.yaml) runs the
  whole pipeline offline in under a minute: simulated noise, mock alerts
  (`alert_stream`), strain collection and `bilby_pipe` configuration
  (`event_sink`), mock results picked up by the result watcher
  (`result_source`, `sample_sink`) and the rescue watcher
  (`rescue_source`). Use it as the reference for the event-handling options.
- [devshm_backfill.yaml](src/rimsky/examples/devshm_backfill.yaml) reads
  live strain from shared memory, see below.
- [arrakis_backfill.yaml](src/rimsky/examples/arrakis_backfill.yaml) reads
  replayed strain from an Arrakis server, see below.

The options of every section are documented in the API reference of
`rimsky.settings`.

### Testing

Run the test suite with

```bash
uv run python -m pytest src/rimsky/tests
```

This includes an end-to-end run of
[pipeline_test.yaml](src/rimsky/examples/pipeline_test.yaml) through the
`rimsky` entry point (`src/rimsky/tests/test_pipeline_run.py`, marked
`pipeline`), which takes about 30 s as the simulated noise is streamed in
real time. Run only it with `-m pipeline`, or skip it with
`-m "not pipeline"`. It needs no network access: alerts and results are
mocked, nothing is uploaded to GraceDB and `bilby_pipe` only writes its
configuration. To run the example by hand from the repository,

```bash
mkdir -p pipeline-test && cd pipeline-test
uv run python -m rimsky ../src/rimsky/examples/pipeline_test.yaml
```

which stops itself after `run_duration` (40 s) and leaves the frames, PSDs,
event directories, pages and `rimsky.log` in `output/`.

### Configuration validation

Unknown configuration keys, e.g., a misspelt `fft_lenght`, are rejected when
the configuration is loaded, with the dotted path of each key. Set
`unknown_keys: warn` to log a warning and ignore them instead, or
`unknown_keys: ignore` to ignore them silently. Top-level keys starting with
`x-` or `_` are ignored and can hold YAML anchors, see
[pipeline_test.yaml](src/rimsky/examples/pipeline_test.yaml). The keys of
`event_sink.bilby_pipe_defaults` and of the `settings` of
`event_sink.trigger_dependent_settings` must be `bilby_pipe` options, and
`prior_bounds` keys must set a prior of their `prior_template`.

The configuration is also checked for consistency, e.g., the
`event_sink.strain_buffer` must cover the data each analysis reads
(`duration - post_trigger_duration` seconds before the trigger and
`post_trigger_duration`, 2 s by default, after it), and the frame cache
(`frame_sink.duration * frame_sink.max_files`) must hold the strain buffer
plus one frame.

### Live data with startup backfill

The `devshm-backfill` data source reads live frames from shared memory (e.g.
`/kafka/H1_O4LLPIC`) like `devshm`, but first replays frame files already in
those directories, so the frame and PSD caches are full shortly after startup
instead of after several minutes. `backfill_duration` sets the maximum number
of seconds to replay; less is used if the directories hold less. See
[devshm_backfill.yaml](src/rimsky/examples/devshm_backfill.yaml).

The `arrakis-backfill` data source does the same for data streamed from an
[Arrakis](https://git.ligo.org/ngdd/arrakis-python) server (set by the
`ARRAKIS_SERVER` environment variable): it requests the stream from
`backfill_duration` seconds before startup, which the server delivers as fast
as it can before continuing with live data. Set `replay_id` to read one of the
server's replays. Because the request has a start time, a dropped connection
resumes from the last sample received rather than filling the outage with
gaps. See [arrakis_backfill.yaml](src/rimsky/examples/arrakis_backfill.yaml).

## Needed functionality

- [x] Streaming simulated data
- [x] Real time PSD estimation and periodic writes to an on-disk cache
- [x] Periodic writes to an on-disk cache of frame data
- [x] Stream real data
  - This can use the `DevShmSource` from `sgnligo`
  - Handled by generic sgnligo data source.
- [x] Listening to GW alerts
- [ ] Read calibration files from specified directories
  - This should keep the latest available calibration file for each detector and watch a regex pattern.
- [x] Writing bilby_pipe config files
- [x] Submitting bilby_pipe jobs
- [x] Adding events to asimov project

## Desired functionality
- [x] Better CI testing
- [x] Stream live injections
  - This should mirror streaming the real data and output frames containing the raw data + injections.
    Given the artificially higher signal rate, the PSD should not include the injected strain.
  - Handled by generic sgnligo data source.

--------

Avatar created by GPT-5.6 Sol.
