Metadata-Version: 2.4
Name: restage
Version: 0.18.0
Author-email: Gregory Tucker <gregory.tucker@ess.eu>
License: BSD-3-Clause
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: zenlog>=1.1
Requires-Dist: platformdirs>=3.11
Requires-Dist: confuse
Requires-Dist: psutil>=5.9.6
Requires-Dist: mccode-antlr>=0.21.0
Requires-Dist: sqlmodel>=0.0.18
Requires-Dist: tqdm>=4.0
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: chopcal>=0.5.1; extra == "test"
Provides-Extra: collectors
Requires-Dist: h5py>=3.0; extra == "collectors"
Requires-Dist: mcstas-readout-master>=0.6.0; extra == "collectors"

# restage
Semiautomatic two-stage McCode simulations for parameter scans.

## Motivation
Some instrument simulations are very costly. 
This is especially true for instruments with a large number of components. 
In order to reduce the runtime of a simulated parameter scan, it is possible to run the simulations in two stages.
In the first stage, the instrument is simulated up to a predefined point and the particle states are saved in an MCPL file. 
The output particle states are then used as input for the second stage, 
where the instrument is simulated from the predefined point to the end of the instrument.
On its own, such a split simulation *is detrimental* to the simulation runtime, 
since writing and reading the MCPL file takes time.

Where such a technique is useful, however, is when the first stage output can be reused 
for multiple second stage simulations.
This is the case, e.g., when a 'standard' sample-rotation scan is performed, 
where the sample is rotated around a single axis and the rest of the instrument is kept fixed.
In this case, the first stage simulation only needs to be performed once,
and the second stage simulations can be performed for each sample rotation angle.

## Installation
The `restage` module is installable via `pip`:
```bash
pip install restage
```

or directly from the source code:
```bash
pip install git+https://github.com/g5t/restage.git
```

## Usage

### `splitrun`
The `restage` module provides a command line program, `splitrun`, which
can be used as a replacement for the `mcrun` function distributed as part of McStas.

`splitrun` produces two instrument stages, before and after a split point, and runs
the specified parameter set(s) in both stages connecting the two through cached MCPL
state files.
The first stage results are stored in a user cache directory and the second stage output
is placed under the working directory following the McStas `mcrun` convention.

Command line flags and parameters are mostly the same as used by `mcrun` with a number
of additions to support `splitrun`'s enhanced behavior.
One optional argument, `split-at`, is added; it should be the name of
an `Arm` component inside the instrument file and defaults to `mcpl_split`.

`split-at` may also name several candidates, separated by commas. Splitting pays only when
scan points share a primary, so `splitrun` takes the latest candidate where the scan needs
fewer primaries than it has points, and runs the scan unsplit when no candidate does.
Only the instrument's author knows where a split is safe -- an MCPL file carries none of
a ray's `USERVARS` -- so `splitrun` chooses among the candidates it is given, and never
inside a `GROUP` or between Union components and their master.
Both halves of a split receive the instrument-level `DECLARE`, `INITIALIZE`, `SAVE` and
`FINAL` blocks, so a scanned parameter mentioned there needs a primary per value.
`splitrun` refuses such an instrument unless `--allow-block-parameters` is given.
In contrast to `mcrun`, instrument parameters for `splitrun` are specified as 'MATLAB'-range style keyword arguments.
A valid range is inclusive of its end points and of the form `start:step:end` or `start:end` (with implicit `step=1`).

The `-N` argument of `mcrun` is removed since the number of scan steps is now determined by the range
of the scan parameter; this means that multiple scanned parameters must have ranges that agree on the number of steps.

The switch to MATLAB-style range specification is done to allow for flexible mesh scans, via the optional `-m` flag.
When the `-m` flag is specified, the scan parameters are interpreted as mesh scan parameters and the number of steps
along each mesh parameter is not constrained to be the same.

#### Example

As an example, for the instrument file `my_instrument.instr`
```
DEFINE INSTRUMENT my_instrument(sample_angle=0, sample_radius)
COMPONENT source = Source(...) AT (0, 0, 0) ABSOLUTE
COMPONENT guide = GuideGravity(l=10) AT (0, 0, 0.1) RELATIVE source
COMPONENT end_of_guide = Arm() AT (0, 0, 10) RELATIVE guide
COMPONENT slits = Slit(...) AT (0, 0, 0.2) RELATIVE end_of_guide
COMPONENT split_at = Arm() AT (0, 0, 0) RELATIVE slits
COMPONENT sample_pos = Arm() AT (0, 0, 0.2) RELATIVE split_at
COMPONENT sample = Sample(radius=sample_radius) AT (0, 0, 0) RELATIVE sample_pos 
                   ROTATED (0, sample_angle, 0) RELATIVE split_at
COMPONENT detector_arm = Arm() AT (0, 0, 0) RELATIVE sample_pos ROTATED (0, 45, 0) RELATIVE sample_pos
COMPONENT detector = Monitor(...) AT (0, 0, 2) RELATIVE detector_arm
END
```

the McStas `mcrun` command
```bash
mcrun my_instrument.instr -N 90 -n 1000000 -d /data/output sample_angle=1,90 sample_radius=10.0
```
can be replaced by the `splitrun` command
```bash
splitrun my_instrument.instr -n 1000000 --split-at split_at -d /data/output sample_angle=1:90 sample_radius=10.0
```


## MCPL components
There are a small collection of `MCPL` input and output components that are provided
along with the McStas and McXtrace distributions, and you may also decide to write your
own specialized variant of those provided.
By default `splitrun` will insert an `MCPL_output` component at the end of the first
stage, and an `MCPL_input` component at the start of the second stage.
Should you prefer to a different component, or need to provide parameter values for
the components, these can be specified in the command line call to `splitrun`.

Components are specified as their file name (minus the `.comp` extension) and
parameters as comma-separated `key:value` pairs, e.g.:

```bash
splitrun -n 1M a3=0:179 --mcpl-input-component MCPL_input_once --mcpl-output-parameters weight_mode:1,double_prec:1 instr.h5 
```

## Collector files
Instruments using the `Collector*` components of
[mcstas-readout-master](https://github.com/mcdotstar/mcstas-readout-master) store
weighted-ray records in HDF5 files for later replay to Event Formation Units. With the
optional extra installed,

```bash
pip install restage[collectors]
```

`splitrun` combines them across both stages:

- the repeated passes of a first-stage simulation are appended into one file in its
  cache directory, as its `.dat` files are combined;
- after the scan, each collector file name becomes one multi-point file in the output
  directory, holding every point's second-stage groups and the groups of the first-stage
  simulation that point used.

A first-stage simulation usually serves many scan points, so its records are stored once
per distinct simulation and each point views them through an HDF5 virtual dataset: a
180-point scan with one first stage holds one copy of that stage's records, not 180.
The file is otherwise an ordinary collector file for `readout-replay` and `readout-combine`.
Without the extra, `splitrun` runs as before and warns that it left collector files
uncombined.


## Cached data
### Default writable cache
A `sqlite3` database is used to keep track of instrument stages, their compiled
binaries, and output file(s) produced by, e.g., `splitrun` simulations.
The default database location is determined by `platformdirs` under a folder
set by `user_cache_path('restage', 'ess')` and the default locations for 
`restage`-compiled instrument binaries and simulation output is determined from
`user_data_path('restage', 'ess')`.

### Override the database and output locations
These default locations can be overridden by setting the `RESTAGE_CACHE` environment
variable to a writeable folder, e.g., `export RESTAGE_CACHE="/tmp/ephemeral"`.

### Read-only cache database(s)
Any number of fixed databases can be provided to allow for, e.g., system-wide reuse
of common staged simulations.
The location(s) of these database file(s) can be specified as a single
environment variable containing space-separated file locations, e.g.,
`export RESTAGE_FIXED="/usr/local/restage /afs/ess.eu/restage"`.
If the locations provided include a `database.db` file, they will be used to search
for instrument binaries and simulation output directories.

### Use a configuration file to set parameters
Cache configuration information can be provided via a configuration file at, 
e.g., `~/.config/restage/config.yaml`, like
```yaml
cache: /tmp/ephemeral
fixed: /usr/local/restage /afs/ess.eu/restage
```
The exact location searched to find the configuration file is platform dependent,
please consult the [`confuse` documentation](https://confuse.readthedocs.io/en/latest/usage.html)
for the paths used on your system.
