Metadata-Version: 2.4
Name: roejobs
Version: 2026.9.1
Summary: Lightweight job runner and supervisor for long-running tasks
Author-email: Adrian Röfer <aroefer@uni-bremen.de>
License-Expression: GPL-3.0-only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jinja2
Requires-Dist: textual
Requires-Dist: requests
Requires-Dist: psutil
Requires-Dist: pyyaml
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# Roejobs

Roejobs is a local job scheduling and management tool designed for running, monitoring, and controlling long-running tasks on a single machine. It provides:

- A background job server managing multiple tasks simultaneously.
- A CLI for submitting jobs from files or the command line.
- A web dashboard to monitor job status, view logs, cancel or restart jobs.
- Automatic management of working directories, output logs, and job queuing.

## Features

- Queue jobs from files or directly from the CLI.
- Optionally wait for all queued jobs to finish and report their outcome (`--wait`).
- Support for `CWD=...` and `ENV=KEY=VALUE` prefixes per job, or a global CWD override.
- View running, queued, succeeded, failed, or cancelled jobs in a web browser.
- Stream job stdout/stderr to log files and view them via the dashboard.
- Cancel, restart, or permanently remove jobs from the web UI, TUI, or API.
- Live-adjust the number of concurrent jobs without restarting the server.
- Per-job CPU% and RAM usage for running jobs, aggregated across any child processes it spawns.
- Multi-stage pipelines defined in YAML (e.g. preprocess -> train -> evaluate), with arbitrary nesting of parallel/sequential steps, via `roejobs-pipeline`.
- Shutdown the server gracefully from the web UI.
- CLI automatically launches the server in the background if it is not running.
- The dashboard, TUI, and CLI all talk to the same JSON API — anything one can do, the others can too.

## Installation

```bash
pip install roejobs
```

This will install the CLI commands and the package itself.
Or, if you cloned this package locally:

```bash
pip install -e .
```

### Starting the server

Start the job server on the default port (5678):
```bash
roejobs-server
```

Optional arguments:

```bash
roejobs-server --port 1234           # Specify a custom port
roejobs-server --n-processes 20      # Set maximum concurrent jobs
```

- The server runs locally (127.0.0.1) and serves the web dashboard.
- Default maximum concurrent jobs: 10.
- The pool size isn't fixed at startup — it can be changed live at any time, without restarting the server or losing running/queued jobs (see "Live pool resizing" below).

### Access the dashboard

Open your browser at: (http://127.0.0.1:5678/)

From here you can:
- Monitor all jobs (auto-refreshes in place — no page reload, so your scroll position is preserved)
- View logs
- Cancel, restart, or remove jobs
- Live-adjust the concurrent job limit
- Shut down the server

## Submitting jobs via CLI
### 1. From a task file

Each line of a task file is a job:

```bash
# Comment lines start with #
CWD=./exp python train.py --lr 0.001 --batch-size 64
python eval.py --checkpoint ckpt.pt
```

Submit jobs:
```bash
roejobs-cli tasks.txt
```

Optional flags:
- `--override-cwd — ignore CWD=...` in job specs and use the CLI's current working directory.
- `--server` — point to a server on a custom port.
- `--wait` — after queuing, block until all submitted jobs conclude, then print their final states and exit.

### 2. Inline submission

You can also submit jobs directly from the CLI:
```bash
roejobs-cli --jobs "python quick_test.py" "CWD=./exp ./run.sh --config cfg.yaml"
```
- Syntax matches task files.
- Works with or without --override-cwd.

### 3. Lazy server start

If the server is not running, the CLI automatically launches it in the background on the correct port.

### 4. Waiting for jobs to finish

By default the CLI returns as soon as the jobs are queued. Pass `--wait` to block until every job it submitted reaches a terminal state (Succeeded, Failed, or Cancelled):

```bash
roejobs-cli --wait --jobs "python train.py --lr 0.01" "python eval.py"
```

When all jobs conclude, the CLI prints each job's final state and exits. The exit code is `0` only if **every** submitted job succeeded; if any job failed or was cancelled it exits `1`, which makes `--wait` convenient in scripts and pipelines.

There is no timeout — job durations are hard to predict — but you can stop waiting at any time with `Ctrl+C`. Interrupting only stops the CLI from waiting; the jobs keep running on the server and can still be monitored from the dashboard.

## Job spec syntax

- Lines starting with `#` are ignored.
- Optional working directory per job:
```bash
CWD=some/path command args...
```
- Optional per-job environment variables — repeat `ENV=` for each variable:
```bash
ENV=CUDA_VISIBLE_DEVICES=0 ENV=WANDB_MODE=offline python train.py
```
- `CWD=` and `ENV=` can appear in any order, any number of times (repeat `ENV=` per variable), before the command:
```bash
ENV=CUDA_VISIBLE_DEVICES=1 CWD=./exp python train.py --lr 0.001
```
- Env values can't contain spaces (no quoting support yet) — `ENV=MSG=hello world` will parse `world` as part of the command, not the value.
- If no `CWD` is specified, the CLI/server uses the current directory.
- Two jobs are only treated as duplicates (same UID) if they match on command, working directory, *and* env — so the same script with a different `CUDA_VISIBLE_DEVICES` is tracked as a distinct job.
- Commands can be very long; they are truncated in the dashboard for readability, with full output available in the log view.

## Live pool resizing

The number of concurrent jobs can be changed at any time without restarting the server:

```bash
roejobs-cli --set-pool-size 4 --server http://127.0.0.1:5678
```

This only affects new job intake — it never kills already-running jobs. Setting it to `0` pauses the queue (nothing new starts) without touching what's currently running. The same control is available as a field on the web dashboard, and as `+`/`-` keybindings in the TUI.

Equivalent API call (this is what the CLI/dashboard/TUI all use under the hood):
```bash
curl -X POST http://127.0.0.1:5678/pool-size -H 'Content-Type: application/json' -d '{"n_processes": 4}'
```

## Resource monitoring

Running jobs report live CPU% and RAM (RSS, in MB) in the dashboard table, job detail page, TUI, and `/jobs`/`/jobs/<uid>` API responses. Usage is aggregated across the job's process **and any children it spawns** — e.g. if your command is a launcher script (`torchrun`, a shell wrapper, etc.), the actual worker processes' usage is included, not just the launcher's.

Sampled once per scheduler tick (every 0.5s by default), not per request, so checking the dashboard doesn't add polling overhead. Values are `null`/empty once a job isn't running (queued, or finished) — there's nothing to measure. CPU% can briefly read `0` for a child process on the tick it first appears; it's accurate from the next tick onward.

## Web dashboard features

- Job table showing status: Queued, Running, Succeeded, Failed, Cancelled.
- Full command, working directory, and env overrides available per job.
- Cancel, restart, or remove individual jobs.
- Live pool size control.
- Shut down server with a single button (gracefully kills running jobs).
- Commands truncated visually for long parameters, with full command in a tooltip.

### Cancel vs. Remove

- **Cancel** stops a queued or running job but keeps it in the job list/history as `CANCELLED` — useful for a record of what happened.
- **Remove** permanently deletes a job from the list and deletes its logs. Only allowed on queued or already-finished jobs; a running job must be cancelled first.

## API

The dashboard, CLI, and TUI are all just clients of the same local JSON API (default `http://127.0.0.1:5678`) — anything they can do, a script or agent can do too by hitting the same endpoints:

| Method | Path                    | Description |
|--------|-------------------------|--------------|
| GET    | `/jobs`                 | List all jobs (JSON), including live `cpu_percent`/`memory_mb` for running ones. |
| GET    | `/jobs/<uid>`            | Single job status. Returns JSON if `Accept: application/json` is set, otherwise an HTML detail page. |
| GET    | `/jobs/<uid>/logs`       | `{"stdout": "...", "stderr": "..."}` for a job. |
| POST   | `/jobs`                  | Submit a job: `{"cmd": [...], "cwd": "...", "env": {...}}` (`cwd`/`env` optional). |
| POST   | `/jobs/<uid>/cancel`     | Cancel a queued or running job. |
| POST   | `/jobs/<uid>/restart`    | Requeue a finished/cancelled job. |
| POST   | `/jobs/<uid>/remove`     | Permanently delete a queued/finished job and its logs. `409` if still running. |
| GET/POST | `/pool-size`           | Get/set the live concurrent-job limit: `{"n_processes": N}`. |
| POST   | `/shutdown`              | Gracefully shut down the server. |

Set `Accept: application/json` on GET requests if you want JSON back from routes (like `/jobs/<uid>`) that can serve either HTML or JSON.

## Example workflow
```bash
# Start server (background auto-start optional)
roejobs-server --port 5678

# Submit jobs from a file
roejobs-cli tasks.txt

# Submit inline jobs
roejobs-cli --jobs "python train.py --lr 0.01" "CWD=./exp ./run.sh"

# Open browser to monitor jobs
firefox http://127.0.0.1:5678/
```

## Pipelines

Pipelines let you compose jobs into ordered/parallel structures, and those structures nest arbitrarily. Every step defaults to being a single job. A step can also be:
- `parallel: [<step>, ...]` — run all of these together, wait for all to finish
- `steps: [<step>, ...]` (optionally with its own `name`/`on_stage_failure`) — a nested sequence, itself a full sub-pipeline

Since steps can contain other `parallel`/`steps` blocks, you can build things like "N independent preprocess -> train -> eval sub-pipelines running in parallel, followed by one aggregation step that only starts once all N are done":

```yaml
# pipeline.yaml
name: my-experiment
on_stage_failure: halt   # halt (default) | continue

steps:
  - name: all-samples
    parallel:
      - name: sample-0
        steps:
          - CWD=./data/0 python preprocess.py
          - CWD=./exp/0  python train.py
          - CWD=./exp/0  python eval.py
      - name: sample-1
        steps:
          - CWD=./data/1 python preprocess.py
          - CWD=./exp/1  python train.py
          - CWD=./exp/1  python eval.py
      # ... one block per sample
  - CWD=./exp python aggregate.py
```

A flat preprocess -> train -> eval pipeline (no nesting) is just this with `parallel` used once per stage — see the simple 3-line-per-stage examples further down for that shape. Each job line uses the same `CWD=`/`ENV=` syntax as task files. A job can also be written in structured form (`{cmd: [...], cwd: ..., env: {...}}`) if you need something the string syntax can't express, like env values containing spaces.

**`on_stage_failure`** (settable per-sequence, defaults to `halt`):
- `halt` — a failed/cancelled step stops that sequence immediately; later steps in it are never queued.
- `continue` — later steps in that sequence still run. The sequence's own final status is still `FAILED` if anything in it ever failed — `continue` only affects whether execution proceeds, not whether the outcome is reported honestly. This composes: if an inner sequence used `continue` and still ends up `FAILED` overall, an outer sequence with `halt` will correctly stop rather than proceeding past it.

**`parallel` failure handling:** a `parallel` block waits for *every* child to reach a terminal state before deciding pass/fail — it does not cancel still-running siblings the moment one fails, so a failure in one of 20 parallel branches doesn't touch the other 19. Whether the *pipeline* proceeds past that `parallel` block afterward is then decided by the enclosing sequence's `on_stage_failure`, same as any other step.

**A sharp edge worth knowing:** a job's identity elsewhere in roejobs is a hash of `cmd`+`cwd`+`env` (this predates pipelines - see below), and submitting a second job with an identical hash while the first is still queued/running is treated as a no-op, not a second execution. If two parallel branches happen to submit byte-identical commands, they'll share one execution rather than run twice. This won't come up in the common case (branches differing by data path/index), but it's worth having in mind if a branch's command doesn't otherwise vary.

### Running pipelines

```bash
roejobs-pipeline run pipeline.yaml              # submit and return immediately
roejobs-pipeline run pipeline.yaml --wait        # block, print live job-count progress, exit non-zero on failure
roejobs-pipeline run pipeline.yaml --override-cwd  # ignore CWD=... in the file, use the CLI's cwd for every job

roejobs-pipeline list                            # all pipelines, with status and jobs-done count
roejobs-pipeline status <id>                     # full step tree with status, indented to match nesting
roejobs-pipeline cancel <id>                     # cancel every currently in-flight step; steps not yet started never run
```

Every job spawned by a pipeline is a completely ordinary job as far as the rest of roejobs is concerned — it shows up in the dashboard/TUI/`/jobs` with its normal uid, and cancel/restart/remove/CPU+RAM monitoring all work on it unchanged. It's additionally tagged with `pipeline_id`/`pipeline_stage` (visible in `/jobs` and `/jobs/<uid>`) so you can tell which pipeline run it came from.

### Pipeline API

| Method | Path                          | Description |
|--------|-------------------------------|--------------|
| POST   | `/pipelines`                   | Submit a pipeline: `{"name": ..., "root": <recursive step, see below>}`. |
| GET    | `/pipelines`                   | List all pipelines. |
| GET    | `/pipelines/<id>`               | Full step tree with each step's status (and job uid, for job steps). |
| POST   | `/pipelines/<id>/cancel`        | Cancel every currently in-flight step; steps not yet started never run. |

A step in the JSON body is one of:
```jsonc
{"type": "job", "cmd": [...], "cwd": "...", "env": {...}, "name": "..."}
{"type": "parallel", "children": [<step>, ...], "name": "..."}
{"type": "sequence", "children": [<step>, ...], "on_stage_failure": "halt", "name": "..."}
```
`roejobs-pipeline` compiles the YAML above into exactly this shape client-side before posting — the same way `roejobs-cli` already resolves task-file lines before POSTing to `/jobs`. The server never parses YAML.

**A caveat worth knowing:** pipelines have no persistence, same as everything else in roejobs right now (see Notes below). A multi-hour pipeline is exactly the case where that costs the most — a server restart mid-pipeline would leave any already-running job running headless, with nothing left to queue what comes after it.

## Notes

- Roejobs is local-only and designed for single-machine usage.
- The web interface is unprotected, so only bind to 127.0.0.1.
- Logs for each job are stored in temporary directories and accessible from the web UI.
