Metadata-Version: 2.4
Name: slurmise
Version: 1.0.0
Summary: Simple slurm resource estimation
Author-email: Troy Comi <tcomi@princeton.edu>, Ioannis Paraskevakos <iparask@princeton.edu>
License-Expression: MIT
Project-URL: Documentation, https://github.com/PrincetonUniversity/slurmise#readme
Project-URL: Issues, https://github.com/PrincetonUniversity/slurmise/issues
Project-URL: Source, https://github.com/PrincetonUniversity/slurmise
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: numpy
Requires-Dist: h5py
Requires-Dist: pandas
Requires-Dist: scikit-learn
Requires-Dist: joblib
Requires-Dist: regex
Dynamic: license-file

<table>
  <tr>
    <td>
      <img src="https://raw.githubusercontent.com/PrincetonUniversity/slurmise/main/docs/assets/logo.svg" alt="slurmise logo" width="140">
    </td>
    <td>

# slurmise

[![PyPI - Version](https://img.shields.io/pypi/v/slurmise.svg)](https://pypi.org/project/slurmise)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/slurmise.svg)](https://pypi.org/project/slurmise)
![CI workflow](https://github.com/princetonuniversity/slurmise/actions/workflows/test.yaml/badge.svg)
[![Coverage Status](https://coveralls.io/repos/github/PrincetonUniversity/slurmise/badge.svg?branch=main)](https://coveralls.io/github/PrincetonUniversity/slurmise?branch=main)

</td>
</tr>
</table>

-----

## Table of Contents

- [Installation](#installation)
- [License](#license)

## Installation

```console
pip install slurmise
```

## Usage

### Docs

Visit the [docs](https://princetonuniversity.github.io/slurmise) for more information.

### Configuration

Slurmise requires a configuration file for every command which tells slurmise
where to store important files and how to process input files.  A general CLI
parser is hard to get correct and still doesn't account for all use cases, for
example is the input file size important or number of lines?  In addition to
some built-in file parsers, you have the option of utilizing `awk` to extract
more complex information from a file.

#### Example toml
```toml
[slurmise]
# base directory to store database and optimized models
base_dir = "slurmise_dir"

# how much the database may grow past the records a model was fit on before
# `predict` warns that the model should be refit, as a fraction of those records.
# Default is 0.2, so a model fit on 20 jobs warns once the database holds 25.
retrain_warning_threshold = 0.2

# set to false to silence the refit warning entirely.  Default is true.
retrain_warning_enable = true

# Global resource bounds applied to all jobs unless overridden per-job.
# Built-in defaults: 60 minutes for runtime, 1000 MB for memory.
[slurmise.runtime]
default = 70          # minutes returned when no model is available
minimum = 10          # never predict below this
maximum = 1440        # never predict above this (e.g. 24-hour wall-time limit)
multiply_prediction_by = 1.0   # scale factor applied after prediction
retry_exponent = 1.0  # on Snakemake retry attempt N: prediction *= N**retry_exponent
on_high_uncertainty_return = "prediction"  # "default", "prediction", "max", or "min"

[slurmise.memory]
default = 2000        # MB returned when no model is available
minimum = 500
maximum = 500000
multiply_prediction_by = 1.0
retry_exponent = 1.0
on_high_uncertainty_return = "prediction"

# Per-job overrides: any field from [slurmise.runtime] / [slurmise.memory] can be
# specified here and will override the global value for that job only.
[slurmise.job.job_name.runtime]
default = 80          # override global default for this job

[slurmise.job.job_name.memory]
default = 3000

# the variables section defines types and how to parse the command
[slurmise.job.job_name.variables]
# the job spec determines how to parse commands to extract their relevant,
# independent variables
threads = {type = "numeric"}
complexity = {type = "category"}
```

The `job_spec` key can be added to define how to parse a command string; its placeholders
reference variable names defined in the `variables` section:
```toml
[slurmise.job.job_name]
job_spec = "subcommand -T {threads} -C {complexity}"

[slurmise.job.job_name.variables]
threads = {type = "numeric"}
complexity = {type = "category"}
```

#### Matching job names
The job name of a command can be set in various ways.  First, if the command
starts with the job name, the job name will be detected and removed from the command.
```toml
# slurmise.toml
[slurmise.job.git]
job_spec = "checkout {branch}"

[slurmise.job.git.variables]
branch = {type = "category"}
```
This job specification will match any of the following with the branch set to `my_branch`
```bash
# infer from command
slurmise record "git checkout my_branch"
# tell explicitly
slurmise record --job-name git "checkout my_branch"
```

Since the job name cannot have spaces and some commands have several subcommands,
you can set unique prefixes for certain jobs.
```toml
# slurmise.toml
[slurmise.job.git_checkout]
job_prefix = "git checkout"
job_spec = "{branch}"

[slurmise.job.git_checkout.variables]
branch = {type = "category"}

[slurmise.job.git_merge]
job_prefix = "git merge"
job_spec = "{branch}"

[slurmise.job.git_merge.variables]
branch = {type = "category"}
```
With a job prefix, slurmise will use the prefix instead of job name to infer
job names.
```bash
slurmise record "git checkout my_branch"
# explicit job name
slurmise record -j git_checkout "my_branch"
```
Note that the job name and prefix should not be included in the job specification.
When a job name is explicitly given to slurmise, the corresponding command should
not have the prefix or job name included.

#### Job specifications
Variables are defined in the `[slurmise.job.job_name.variables]` section.  Each
variable requires a `type`:
```toml
[slurmise.job.job_name.variables]
variable_name = {type = "variable_type"}
```
The name should be unique within a job and contain no spaces.  The type can be
one of:
- `numeric`: A single number, used in regression as an independent variable.
Examples include the number of threads, epochs, or replicates to perform.
Integer, decimal, and scientific notation values (e.g. `1e-4`, `2.5E+6`) are
all supported.
- `category`: A string which is used to select the correct model.  Examples include
what algorithm to choose, switches or flags.  Note that a category can be a number,
but will be stored as a string, e.g. "1.0" is different from "1".  For inference,
the categories will be matched to particular, independent model.
- `file`: An input file in plain text.  Can be processed further as described below.
- `gzip_file`: An input file in gzip format.  During processing, the file will
be decompressed to read it's contents, note this can incur memory and cpu drain.
- `file_list`: An input file that contains a list of files to process in turn.

When a `job_spec` is provided, its placeholders reference the variable names:
```
{variable_name}
```
By default each placeholder matches a single whitespace-delimited token.  The
special token `{ignore}` can appear in `job_spec` to match and discard one
token that is not recorded; it does not require a variable name or entry in the
`variables` section.  Use multiple `{ignore}` tokens to skip multiple arguments:
```toml
job_spec = "cmd {input} {ignore} {ignore} {output}"
```

#### Matching multi-word or non-standard tokens

When a placeholder needs to match something other than a single whitespace-delimited
word — for example, multiple arguments to discard, a file path containing spaces, or
a value delimited by something other than whitespace — add a `pattern` key to the
variable entry with a Python regular expression:
```toml
[slurmise.job.my_job.variables]
# absorb a variable number of extra arguments at the end of the command
extra_args = {type = "ignore", pattern = ".+"}

# file path wrapped in double quotes (e.g. cmd "/path/with spaces/file.txt")
input = {type = "file", file_parsers = "file_basename", pattern = '[^"]+'}
```
For the quoted-file case, surround the placeholder with literal quote characters
in `job_spec` so the quotes are consumed but not captured:
```toml
job_spec = 'cmd "{input}" {output}'
```
The `pattern` key is not supported for `numeric` variables, whose matching
pattern is fixed to ensure the captured value can always be converted to a
number.

To include a literal `{` or `}` in a job spec (for example in an `awk` program
or a shell brace expression), double the brace: `{{` produces a literal `{` and
`}}` produces a literal `}`.
```toml
job_spec = "awk '{{print $1}}' {input}"
```
This matches commands like `awk '{print $1}' data.txt` and captures `data.txt`
as the `input` variable.

#### File Parsers
Each file can have one or more parsers associated with its variable name.
Slurmise comes with several built-in options for parsing files:
- `file_size`: The size of the file on disk, in bytes, numeric
- `file_lines`: The number of lines (newlines)  in the file, numeric
- `file_basename`: The base filename, category
- `file_md5`: The md5 digest of the file contents, category

Additionally, custom file parsers can be made using awk.  While somewhat limited,
awk prevents security issues with running arbitrary code.  File parsers require
a unique name in the `slurmise.file_parsers` collection.  The return type is
category by default.  The awk command can be supplied as a string or file path,
which is used with the `-f` flag of awk.  Here are some examples:
```toml
[slurmise.file_parsers.epochs]
type = "numeric"
awk_script = "/^epochs:/ {print $2}"

[slurmise.file_parsers.network]
type = "category"
awk_script = "/^network type:/ {print $3}"

[slurmise.file_parsers.fasta_length]
type = "numeric"
awk_script = "/path/to/awk/file.awk"
script_is_file = True

# contents of file.awk
# /^>/ {if (seq) print seq; seq=0}
# /^>/ {next}
# {seq = seq + length($0)}
# END {if (seq) print seq}
```
The first extracts the token after `epochs: ` as a number and could be used for
getting metadata from a configuration file.  Similarly, the network parser
extracts the `network type` but this time returns the result as a category.

Finally, the `fasta_length` parser takes an awk script file that prints the
length of each sequence in a fasta file, returning the list of numbers as numerics.

To specify which parser a file uses, add a `file_parsers` key to the variable entry:
```toml
[slurmise.job.sample_files]
job_spec = "--reference {reference} {fasta}"

[slurmise.job.sample_files.variables]
reference = {type = "file", file_parsers = ["file_md5", "file_lines"]}
fasta = {type = "file", file_parsers = ["file_size", "fasta_length"]}
```
Each file variable needs a `file_parsers` entry.  The value can be a single string
or a list of parsers.  Here the reference file is parsed with the md5 and number of
lines.  The md5 will create a new category based on the file contents while the lines
will be an independent variable.  In practice, matching md5 will ensure the same number
of lines which doesn't provide additional information to the model.  The fasta file
returns the file size in bytes and the number of nucleotides in each fasta entry.


## License

`slurmise` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.
