Metadata-Version: 2.4
Name: panoptes-ui
Version: 1.5.1
Summary: panoptes: monitor computational workflows in real time
Home-page: https://github.com/panoptes-organization/panoptes
Author: panoptes-organization
Author-email: georgekostoulas@gmail.com, agardelakos@gmail.com, fgypas@gmail.com, gntalaperas@gmail.com, dimitris.afe@gmail.com, rekoumisd@gmail.com, vsochat@stanford.edu
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: flask>=1.1.1
Requires-Dist: humanfriendly>=4.18
Requires-Dist: marshmallow>=3.0.1
Requires-Dist: SQLAlchemy>=1.3.7
Provides-Extra: dev
Requires-Dist: pytest>=5.3.0; extra == "dev"
Requires-Dist: requests>=2.22.0; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# ![alt text](panoptes/static/src/img/brand/panoptes.png "panoptes")

[![CI](https://github.com/panoptes-organization/panoptes/actions/workflows/python-package-conda-pip.yml/badge.svg)](https://github.com/panoptes-organization/panoptes/actions/workflows/python-package-conda-pip.yml)
[![PyPI](https://img.shields.io/pypi/v/panoptes-ui.svg)](https://pypi.org/project/panoptes-ui/)
[![Bioconda](https://img.shields.io/conda/vn/bioconda/panoptes-ui.svg)](https://anaconda.org/bioconda/panoptes-ui)
[![Python versions](https://img.shields.io/pypi/pyversions/panoptes-ui.svg)](https://pypi.org/project/panoptes-ui/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE.md)


Bioinformaticians and data scientists, rely on computational frameworks (e.g. [snakemake](https://snakemake.readthedocs.io/en/stable/), [nextflow](https://www.nextflow.io/), [CWL](https://www.commonwl.org/), [WDL](https://software.broadinstitute.org/wdl/)) to process, analyze and integrate data of various types. Such frameworks allow scientists to combine software and custom tools of different origin in a unified way, which lets them reproduce the results of others, or reuse the same pipeline on different datasets. One of the fundamental issues is that the majority of the users execute multiple pipelines at the same time, or execute a multistep pipeline for a big number of datasets, or both, making it hard to track the execution of the individual steps or monitor which of the processed datasets are complete. panoptes is a tool that monitors the execution of such workflows.

panoptes is a service that can be used by:
- Data scientists, bioinformaticians, etc. that want to have a general overview of the progress of their pipelines and the status of their jobs
- Administrations that want to monitor their servers
- Web developers that want to integrate the service in bigger web applications

**Note:** panoptes currently supports workflows written in [snakemake](https://snakemake.readthedocs.io/en/stable/).

> **Snakemake 9 users:** the legacy `--wms-monitor` flag was removed upstream.
> Monitoring is now delivered via a logger plugin — see
> [Snakemake 9 support](#snakemake-9-support) below.

# Installation

Requirements:
- Python>=3.11
- virtualenv
- [sqlite3](https://www.sqlite.org/download.html)

## Local
### pypi

Create virtual environment
```bash
python3 -m venv venv
```

Activate virtual environment
```bash
source venv/bin/activate
```

Install via pypi
```bash
pip install panoptes-ui
```

### conda

Create conda environment
```bash
conda create --name panoptes -c conda-forge -c bioconda panoptes-ui
```

Activate conda environment
```bash
conda activate panoptes
```

### Source code

Clone repo
```bash
git clone https://github.com/panoptes-organization/panoptes.git
```

Enter repo
```bash
cd panoptes
```

Create virtual environment
```bash
python3 -m venv venv
```

Activate virtual environment
```bash
source venv/bin/activate
```

Install all requirements
```bash
pip install .
```

### Run the server
By default the server binds to `0.0.0.0:5000` — i.e. it listens on **all** network interfaces (open http://127.0.0.1:5000 in your browser) — and generates the sqlite database `.panoptes.db`.

Because `0.0.0.0` means every interface, the server is reachable from other machines on your network (and, on Linux, on any `127.x.x.x` loopback address — the whole `127.0.0.0/8` block is loopback by standard). If you only need local access, restrict it with:
```bash
panoptes --ip 127.0.0.1
```

The running version is shown in the web UI under **About** (and is available programmatically as `panoptes.__version__`).

Environment variables:

| Variable | Default | Description |
| --- | --- | --- |
| `PANOPTES_DB_URL` | `sqlite:///.panoptes.db` | Database URL. |
| `PANOPTES_STALE_HOURS` | `48` | Hours without any event before a `Running` workflow is marked `Stale` (e.g. its snakemake process was killed and never reported back). Set to `0` to disable. A `Stale` workflow flips back to `Running` if events resume, and can be deleted directly. |

#### Using the development server
```bash
panoptes
```

#### Using a WSGI server
Install all necessary packages (see above), plus a WSGI server (e.g. [gunicorn](https://gunicorn.org/) or [waitress](https://docs.pylonsproject.org/projects/waitress/en/latest/)), and run the server:
```bash
gunicorn --access-logfile logs/access.log --error-logfile logs/error.log --timeout 120 --bind :5000 panoptes.app:app
```

## Containers
### Docker

Requirements:
- docker

Pull the image that is automatically built from bioconda. Replace `<tag>` with a release from the list of available tags: https://quay.io/repository/biocontainers/panoptes-ui?tab=tags
```
docker pull quay.io/biocontainers/panoptes-ui:<tag>
```

Then run the container with:

```bash
docker run -p 5000:5000 -it <image-id> panoptes
```

> Note: In this case the database is stored within the docker image, so every time you restart the server the database will be empty. You would need to mount the volumes to make the database persistent.

### Docker compose

Requirements:
- docker
- docker-compose

Build
```bash
docker-compose build
```

Run
```bash
docker-compose up -d
```

Server should run on: http://127.0.0.1:8000

Stop
```bash
docker-compose down
```

### Singularity

You can also deploy the server with singularity. To do so pull the image with singularity. Replace `<tag>` with a release from the list of available tags: https://quay.io/repository/biocontainers/panoptes-ui?tab=tags

```bash
singularity pull docker://quay.io/biocontainers/panoptes-ui:<tag>
```

And then we can start the server by running:
```bash
singularity exec panoptes-ui:<tag>
```

# Run an example workflow

A small reference pipeline (samtools sort/index → htseq-count → merge across
four samples) that already wires up `--logger panoptes` lives at
[snakemake_example_workflow](https://github.com/panoptes-organization/snakemake_example_workflow).
Follow the instructions there to exercise this server end-to-end.

# Snakemake 9 support

Starting with Snakemake 9, the `--wms-monitor` flag that older panoptes versions
relied on has been removed. Monitoring is instead delivered through
[logger plugins](https://snakemake.readthedocs.io/en/stable/executing/monitoring.html).

To stream events from a Snakemake 9 workflow to panoptes, install the companion
logger plugin with either pip or conda:

```bash
pip install snakemake-logger-plugin-panoptes
# or
conda install -c conda-forge -c bioconda snakemake-logger-plugin-panoptes
```

Then pass `--logger panoptes` to Snakemake:

```bash
snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000
```

### Re-runs under a stable id

By default every invocation registers a **new** workflow in panoptes. If you
want re-runs of the same pipeline to show up as the *same* workflow — e.g. a
run fails halfway, you fix the problem and resume — give it a stable id:

```bash
snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000 \
    --logger-panoptes-workflow-id my-pipeline
```

When a run starts with an id panoptes already knows, the existing workflow
entry is reset and reused instead of a second entry being created, so the
dashboard tracks the latest state of that pipeline under one entry.

**Restart protection:** the entry is only reset when the previous run is in a
finished state (`Done`, `Error`, `Cancelled`, `Stale`, …). A run interrupted
with Ctrl+C or failed mid-way ends up as `Error`, so the normal
fail-fix-restart cycle just works. But if the previous run still shows
`Running` — either it is genuinely alive, or its process was killed hard
(`kill -9`) and never got to report — panoptes will not wipe its history:
the new run is tracked under a fresh entry with a random suffix (e.g.
`my-pipeline-1a2b3c4d`) instead. To restart under the original id in that
case, first cancel the stuck entry (the ban button in the UI, or
`POST /api/workflow/<id>/cancel`) and then re-run — mirroring the
cancel-before-delete rule.

The plugin lives in its own repository:
[panoptes-organization/snakemake-logger-plugin-panoptes](https://github.com/panoptes-organization/snakemake-logger-plugin-panoptes).
It registers a workflow with panoptes via `GET /create_workflow` on the first
event and then translates Snakemake's `LogEvent` records (`JOB_INFO`,
`JOB_STARTED`, `JOB_FINISHED`, `JOB_ERROR`, `SHELLCMD`, `PROGRESS`, `ERROR`,
`RUN_INFO`) into the JSON payloads that panoptes' `/update_workflow_status`
endpoint already understands. When a run ends without error, the plugin also
emits a `workflow_success` event so panoptes can mark the workflow `Done` even
when Snakemake never reported `done == total` — e.g. with `--until`, where the
reported total is the whole DAG but only a subset actually runs.

Workflows orchestrated by Snakemake &lt; 9 continue to work unchanged via the
legacy `--wms-monitor http://<host>:<port>` flag.

## Using panoptes via the Snakemake Python API

If you drive Snakemake programmatically instead of via the CLI, enable the
panoptes logger by passing the plugin's settings in
`OutputSettings.log_handler_settings` — the API equivalent of
`--logger panoptes --logger-panoptes-address ...`:

```python
from pathlib import Path

from snakemake.api import SnakemakeApi
from snakemake.settings.types import OutputSettings, ResourceSettings
from snakemake_logger_plugin_panoptes import LogHandlerSettings

panoptes_settings = LogHandlerSettings(
    address="http://127.0.0.1:5000",  # where the panoptes server runs
    workflow_id="my-workflow",        # optional: stable id, so re-runs reuse
                                      # the same workflow entry in panoptes
)

with SnakemakeApi(
    OutputSettings(
        log_handler_settings={"panoptes": panoptes_settings},
    )
) as snakemake_api:
    workflow_api = snakemake_api.workflow(
        resource_settings=ResourceSettings(cores=2),
        snakefile=Path("Snakefile"),
    )
    dag_api = workflow_api.dag()
    dag_api.execute_workflow()
```

Requires `snakemake>=9` and `snakemake-logger-plugin-panoptes` (see above) in
the same environment. Everything else (workflow registration, per-job events,
end-of-run success reporting) behaves exactly as with the CLI flags.

## panoptes in action

[![Watch the video](https://img.youtube.com/vi/de-YSJmq_5s/hqdefault.jpg)](https://www.youtube.com/watch?v=de-YSJmq_5s)

## Workflow statuses

| Status | Meaning |
| --- | --- |
| `Running` | The workflow is registered and events are arriving. |
| `Done` | All jobs finished (`done == total`), or the plugin reported end-of-run success (e.g. `--until` runs). |
| `Error` | A job or the workflow reported an error. |
| `Cancelled` | Explicitly cancelled via `POST /api/workflow/<id>/cancel`. |
| `Stale` | No events for more than `PANOPTES_STALE_HOURS` (default 48h) — the snakemake process was probably killed. Reverts to `Running` if events resume. |
| `No Execution` | Snakemake reported there was nothing to be done. |

The web pages poll the JSON API every few seconds and refresh automatically when the data changes, so a dashboard left open tracks running workflows without manual reloads. A workflow page also shows a **per-rule progress breakdown** below the overall progress bar, so a run with hundreds of jobs but few rules stays legible at a glance.

## panoptes API

Panoptes provides the following API endpoints:

Endpoint | Method | Description 
-- | -- | --
`/api/service-info` | `GET` | Server status
`/api/workflows` | `GET` | Get all workflows
`/api/workflow/<workflow-id>` | `GET` | Get workflow status
`/api/workflow/<workflow-id>/jobs` | `GET` | Get all jobs of a workflow
`/api/workflow/<workflow-id>/job/<job-id>` | `GET` | Get job status
`/api/workflow/<workflow-id>` | `PUT` | Rename a workflow  <br>  Expects a dictionary with new name <br> (e.g. `{'name': 'my new workflow name'}`)
`/api/workflow/<workflow-id>/cancel` | `POST` | Cancel a workflow (sets its status to `Cancelled`). Use this to move a workflow that is stuck as `Running` (e.g. a dry run, or a process killed with `kill -9`) into a deletable state.
`/api/workflow/<workflow-id>` | `DELETE` | Delete a workflow. A workflow that is still `Running` is protected and returns `403`; cancel it first via the endpoint above.
`/api/workflows/all` | `DELETE` | Clean up database (deletes all workflows, including running ones)

To communicate with panoptes the following endpoints are used by snakemake:

Endpoint | Method | Description 
-- | -- | --
`/api/service-info` | `GET` | Server status (same as above)
`/create_workflow` | `GET` | Get a unique id/name str(uuid.uuid4()) for each workflow
`/update_workflow_status` | `POST` | Panoptes receives a dictionary from snakemake that contains: <br> - A log message dictionary (JSON-encoded) <br> - The current timestamp <br> - The unique id/name of the workflow. <br> (e.g. `{'msg': json.dumps(message), 'timestamp': time.asctime(), 'id': id}`)

# Contribute

Please see the [Contributing instructions](CONTRIBUTING.md).

## CI server

Changes on master (and pull requests against it) trigger a [GitHub Actions](https://github.com/panoptes-organization/panoptes/actions) build that runs the test suite and a live end-to-end run of the example workflow.

# Contact

In case the [issues section](https://github.com/panoptes-organization/panoptes/issues) is not enough for you, you can also contact us via [discord](https://discord.gg/vMcZCVZ)
