Metadata-Version: 2.4
Name: sophios
Version: 0.7.0
Summary: A workflow compiler: write workflows in Python or YAML, check them before they run, and compile them to CWL
Author-email: Jake Fennick <jake.fennick@axleinfo.com>
Project-URL: Homepage, https://github.com/PolusAI/sophios
Project-URL: Bug Tracker, https://github.com/PolusAI/sophios/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cwltool>=3.1.20241217163858
Requires-Dist: graphviz
Requires-Dist: jsonschema
Requires-Dist: pyyaml
Requires-Dist: ruamel.yaml
Requires-Dist: requests>=2.32.2
Requires-Dist: mergedeep
Requires-Dist: cwl-utils>=0.32
Requires-Dist: pydantic>=2.6
Requires-Dist: pydantic[email]
Requires-Dist: docker
Requires-Dist: podman
Requires-Dist: toil[cwl]
Requires-Dist: fastapi
Requires-Dist: uvicorn
Provides-Extra: test
Requires-Dist: networkx; extra == "test"
Requires-Dist: pre-commit; extra == "test"
Requires-Dist: py; extra == "test"
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pytest-parallel; extra == "test"
Requires-Dist: coverage; extra == "test"
Requires-Dist: mypy; extra == "test"
Requires-Dist: numpy; extra == "test"
Requires-Dist: scipy; extra == "test"
Requires-Dist: pylint; extra == "test"
Requires-Dist: autopep8; extra == "test"
Requires-Dist: pre-commit; extra == "test"
Requires-Dist: hypothesis; extra == "test"
Requires-Dist: import-linter; extra == "test"
Provides-Extra: mypy-types
Requires-Dist: lxml-stubs; extra == "mypy-types"
Requires-Dist: types-Pillow; extra == "mypy-types"
Requires-Dist: types-PyYAML; extra == "mypy-types"
Requires-Dist: types-Pygments; extra == "mypy-types"
Requires-Dist: types-colorama; extra == "mypy-types"
Requires-Dist: types-decorator; extra == "mypy-types"
Requires-Dist: types-docutils; extra == "mypy-types"
Requires-Dist: types-html5lib; extra == "mypy-types"
Requires-Dist: types-jsonschema; extra == "mypy-types"
Requires-Dist: types-psutil; extra == "mypy-types"
Requires-Dist: types-python-jose; extra == "mypy-types"
Requires-Dist: types-pytz; extra == "mypy-types"
Requires-Dist: types-redis; extra == "mypy-types"
Requires-Dist: types-requests; extra == "mypy-types"
Requires-Dist: types-setuptools; extra == "mypy-types"
Requires-Dist: types-six; extra == "mypy-types"
Requires-Dist: types-urllib3; extra == "mypy-types"
Provides-Extra: runners
Requires-Dist: toil[cwl]; extra == "runners"
Requires-Dist: cwl-utils>=0.32; extra == "runners"
Provides-Extra: doc
Requires-Dist: sphinx; extra == "doc"
Requires-Dist: myst-parser; extra == "doc"
Requires-Dist: sphinx-autodoc-typehints; extra == "doc"
Provides-Extra: plots
Requires-Dist: matplotlib; extra == "plots"
Provides-Extra: cyto
Requires-Dist: ipycytoscape; extra == "cyto"
Provides-Extra: all-except-runner-src
Requires-Dist: sophios[cyto,doc,mypy-types,plots,test]; extra == "all-except-runner-src"
Provides-Extra: all
Requires-Dist: sophios[cyto,doc,mypy-types,plots,test]; extra == "all"
Dynamic: license-file

# Sophios

[![doc-build-status](https://readthedocs.org/projects/sophios/badge/?version=latest)](https://sophios.readthedocs.io/en/latest/)

Sophios is a workflow compiler. You write a workflow in Python, or in `.wic`, a short YAML format. Sophios holds it in a workflow model of its own, checks all of it before anything runs, and compiles it for a backend. Today that backend is the [Common Workflow Language](https://www.commonwl.org) (CWL), so what it writes runs under any CWL runner, with or without Sophios installed: cwltool on your machine, or Toil, which sends the steps to a cloud or to Slurm, LSF and other cluster schedulers.

An analysis is rarely one program: you clean the raw data, fit a model and compute statistics on it side by side, and put both into a figure. Steps, and the data that flows between them, make a graph. Run by hand or from a shell script, that graph works on your machine today; Sophios makes it a workflow that runs the same way on next month's data, on a colleague's machine and on a cluster. You describe each step: which program it runs, what you give it and which earlier results it reads. Sophios fills in the connections you leave out and tells you what it chose, stops on a value of the wrong type, a missing input file or a container engine that is not running, and runs every program in the container it asks for.

Sophios's Python API has two halves. `sophios.api.python.workflow` builds workflows from `Step` and `Workflow` objects. `sophios.api.python.tool_builder` builds the tools themselves, so a program of your own becomes a step without hand-written CWL.

## Documentation

The documentation is on [Read the Docs](https://sophios.readthedocs.io/en/latest/). Start with [Install](docs/install.md), then [Your first workflow](docs/first-workflow.md).

## Install

1. Install [Docker](https://docs.docker.com/get-docker/) and start it. Sophios runs each tool in the container image it names.
2. Create a conda environment with the programs pip cannot install, Node.js (the newest 26.x) and Graphviz:

   ```bash
   conda create --name sophios --channel conda-forge python=3.11 nodejs=26 graphviz
   conda activate sophios
   ```

3. Install Sophios, and let it write its configuration and copy the tools that ship with it to `~/wic/`:

   ```bash
   python -m pip install sophios
   sophios --generate_config
   ```

4. Check that it works:

   ```bash
   mkdir sophios-check && cd sophios-check
   sophios --yaml ~/wic/examples/multistep1.wic --run_local --quiet
   ```

   The last line starts with `Success! Runner outputs are under`.

The [install guide](docs/install.md) explains each step and what to do when one fails. To work on Sophios itself, start with the [developer guide](docs/dev/devguide.md).

## A first workflow

This workflow makes an empty file, writes a greeting into it, and reads the file back. In a project directory, save either form:

```yaml
# greeting.wic
steps:
- id: touch
  in:
    filename: !ii greeting.txt
  out:
  - file: !& empty_file
- id: append
  in:
    file: !* empty_file
    str: !ii Hello, Sophios
  out:
  - file: !& greeting_file
- id: cat
  in:
    file: !* greeting_file
```

```python
# greeting.py
"""Create a file, write a greeting into it, and read it back."""
from pathlib import Path

from sophios.api.python.workflow import Step, Workflow


#: The tools that ship with Sophios, where `sophios --generate_config` put them.
ADAPTERS = Path.home() / 'wic' / 'cwl_adapters'


def workflow() -> Workflow:
    """The same three steps as greeting.wic."""
    touch = Step(clt_path=ADAPTERS / 'touch.cwl')
    touch.inputs.filename = 'greeting.txt'
    append = Step(clt_path=ADAPTERS / 'append.cwl')
    append.inputs.file = touch.outputs.file
    append.inputs.str = 'Hello, Sophios'
    cat = Step(clt_path=ADAPTERS / 'cat.cwl')
    cat.inputs.file = append.outputs.file
    return Workflow([touch, append, cat], 'greeting')


if __name__ == '__main__':
    workflow().run()
```

`!ii` marks a value written into the workflow; `!&` names a step's output and `!*` feeds it into a later step's input. In Python, an assignment does each, and `ADAPTERS` is where `sophios --generate_config` put the tools that ship with Sophios. Run either form:

```bash
sophios --yaml greeting.wic --run_local
python greeting.py
```

Both compile to the same CWL, check the machine, fetch the container images and run. Sophios writes the compiled CWL, the job file and the results under `autogenerated/` in the directory you run from. `--basepath` (or `basepath=` in Python) lets you choose another directory for them, and `--outdir` one for only the files the run produces. Every problem Sophios finds carries a code, and `sophios --explain <code>` says what to change.

## Writing a tool

When a program has no CWL description yet, the tool builder writes one, and the tool goes straight into a workflow. Here `wc -w` counts the words of the greeting:

```python
# count_words.py
"""Build a tool in Python, then run it in a workflow."""
from pathlib import Path

from sophios.api.python.tool_builder import (
    CommandLineTool, Input, Inputs, Output, Outputs, cwl,
)
from sophios.api.python.workflow import Step, Workflow

#: The tools that ship with Sophios, where `sophios --generate_config` put them.
ADAPTERS = Path.home() / 'wic' / 'cwl_adapters'

word_count = (
    CommandLineTool('word_count',
                    Inputs(text=Input(cwl.file, position=1)),
                    Outputs(counts=Output(cwl.file, glob='counts.txt')))
    .base_command('wc', '-w')
    .stdout('counts.txt')
    .docker('debian:bookworm-slim')
)


def workflow() -> Workflow:
    """Create a file, write a line into it, and count its words."""
    touch = Step(clt_path=ADAPTERS / 'touch.cwl')
    touch.inputs.filename = 'greeting.txt'
    append = Step(clt_path=ADAPTERS / 'append.cwl')
    append.inputs.file = touch.outputs.file
    append.inputs.str = 'Hello, Sophios'
    count = Step(word_count)
    count.inputs.text = append.outputs.file
    return Workflow([touch, append, count], 'count_words')


if __name__ == '__main__':
    word_count.write_cwl('word_count.cwl', validate=True)
    workflow().run()
```

The run writes `word_count.cwl`, a tool file any `.wic` workflow can use, and leaves `counts.txt` in the output directory. [Writing tools in Python](docs/tool-builder.md) takes the builder one part at a time.

## License

MIT; see [LICENSE](LICENSE).
