Metadata-Version: 2.4
Name: runwatch-notebook
Version: 0.2.0
Summary: Durable notebook execution with typed resource monitoring and a mobile dashboard
Author: Agustín Sellanes
License-Expression: MIT
Project-URL: Homepage, https://github.com/EssenceSentry/runwatch
Project-URL: Repository, https://github.com/EssenceSentry/runwatch
Project-URL: Documentation, https://essencesentry.github.io/runwatch/
Project-URL: Bug Tracker, https://github.com/EssenceSentry/runwatch/issues
Keywords: jupyter,notebook,nbclient,monitoring,aws
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.6
Provides-Extra: supervisor
Requires-Dist: async-timeout>=5; extra == "supervisor"
Requires-Dist: boto3>=1.34; extra == "supervisor"
Requires-Dist: beautifulsoup4>=4.12; extra == "supervisor"
Requires-Dist: fastapi>=0.110; extra == "supervisor"
Requires-Dist: httpx>=0.27; extra == "supervisor"
Requires-Dist: jinja2>=3.1; extra == "supervisor"
Requires-Dist: jupyter-client>=8.9.1; extra == "supervisor"
Requires-Dist: nbclient<0.12,>=0.10.4; extra == "supervisor"
Requires-Dist: nbconvert<8,>=7.17.1; extra == "supervisor"
Requires-Dist: nbformat<6,>=5.10; extra == "supervisor"
Requires-Dist: numpy<3,>=1.26; extra == "supervisor"
Requires-Dist: psutil>=5.9; extra == "supervisor"
Requires-Dist: pyyaml>=6; extra == "supervisor"
Requires-Dist: qrcode>=7.4; extra == "supervisor"
Requires-Dist: typer>=0.12; extra == "supervisor"
Requires-Dist: uvicorn[standard]>=0.27; extra == "supervisor"
Requires-Dist: websockets>=12; extra == "supervisor"
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py>=12; extra == "gpu"
Provides-Extra: test
Requires-Dist: ipykernel>=7.3; extra == "test"
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: pytest-asyncio>=0.21; extra == "test"
Requires-Dist: pytest-cov>=5; extra == "test"
Requires-Dist: tqdm<5,>=4.66; extra == "test"
Provides-Extra: dev
Requires-Dist: black>=22; extra == "dev"
Requires-Dist: build<1.5.1,>=1.5; extra == "dev"
Requires-Dist: flake8>=7.1; extra == "dev"
Requires-Dist: flake8-cognitive-complexity>=0.1; extra == "dev"
Requires-Dist: pre-commit>=2.17; extra == "dev"
Requires-Dist: pyright<2,>=1.1.409; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: tomli<3,>=2; python_version < "3.11" and extra == "dev"
Requires-Dist: twine<7,>=6; extra == "dev"
Requires-Dist: pyyaml>=6; extra == "dev"
Requires-Dist: types-PyYAML<7,>=6.0.12; extra == "dev"
Provides-Extra: docs
Requires-Dist: myst-parser>=4; extra == "docs"
Requires-Dist: pydata-sphinx-theme>=0.16; extra == "docs"
Requires-Dist: sphinx>=8; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=3; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
Dynamic: license-file

<p align="center">
  <img src="web_artifacts/runwatch/mascot/ready.png" width="160" alt="Runwatch mascot">
</p>

<h1 align="center">Runwatch</h1>

<p align="center"><strong>Turn long-running notebooks into durable, observable jobs—without turning them into pipelines.</strong></p>

<p align="center">
  Run the notebook you already have. Follow it from your phone. Recover where the work failed.
</p>

<p align="center">
  <a href="https://github.com/EssenceSentry/runwatch/actions/workflows/quality-gate.yml"><img src="https://github.com/EssenceSentry/runwatch/actions/workflows/quality-gate.yml/badge.svg?branch=main" alt="Quality gate"></a>
  <a href="pyproject.toml"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
  <a href=".pre-commit-config.yaml"><img src="https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&amp;logoColor=white" alt="pre-commit enabled"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue" alt="MIT license"></a>
</p>

Runwatch is a reliability layer for Jupyter notebooks. It combines cell-by-cell
`nbclient` execution with durable checkpoints, live resource monitoring, notifications,
and a mobile-friendly dashboard. When a cell fails, the run pauses with its state and
monitors intact, ready for you—or your coding agent—to repair and continue.

**Keep notebook ergonomics. Gain the confidence of an operational job.**

[Get started](#first-run) · [Run from VS Code](#run-from-vs-code) ·
[Explore resource monitoring](#resource-emission) ·
[See failure recovery](#failure-and-recovery) · [Read the docs](https://essencesentry.github.io/runwatch/)

## Built for notebook jobs that have outgrown “Run All”

- **Recover instead of rerun.** Runwatch checkpoints every settled cell, preserves a
  live kernel after failure, and supports resume or clean replay after a process or
  kernel restart.
- **Walk away without losing visibility.** Follow cells, outputs, tracebacks, progress,
  metrics, logs, and external resources from a responsive dashboard on desktop or
  mobile.
- **Get an ETA that adapts to the work.** Runwatch mirrors existing `tqdm` bars without
  changing the loop, adds an uncertainty range, and recalibrates when throughput shifts.
- **See the whole workload, not just the notebook.** Built-in adapters watch SageMaker
  Processing, S3, CloudWatch, host and kernel metrics, NVIDIA GPUs, local files, logs,
  and linked localhost dashboards.
- **Know when something changes.** Durable webhook and `ntfy` delivery can notify you
  with periodic status reminders, notebook section transitions, failures, completion,
  and refreshed Cloudflare sharing links.
- **Keep the execution host awake.** An optional native IOKit assertion on macOS or
  logind inhibitor on Linux prevents idle system sleep while the run is live.
- **Share access intentionally.** Keep the dashboard on localhost, expose it to a
  trusted LAN, or use a self-healing `cloudflared` quick tunnel with pairing-token
  authentication.
- **Work equally well with humans and coding agents.** The editable run-owned notebook,
  structured context command, and explicit resume/restart workflow use ordinary files
  and CLI commands—no agent lock-in.
- **Get the result back where you expect it.** Settled sources, execution counts, and
  outputs are written atomically to the notebook you launched, with conflict detection
  to protect outside edits.
- **Avoid a trail of abandoned state.** Successful and cancelled runs clean themselves
  up after the dashboard closes, while incomplete notification delivery and
  `--keep-run` preserve state when it still matters.

The dashboard is deliberately observation-first. Its one remote mutation is narrowly
scoped: stopping an owned, stoppable resource also cancels the run. Runwatch records
actions, source hashes, kernel epochs, adapter cursors, observations, logs,
notifications, and bounded events in durable SQLite state so recovery does not depend
on one long-lived process.

## Installation

Python 3.10 or newer is required. Runwatch supports local POSIX filesystems on Linux
and macOS; those are the platforms exercised in CI. Windows and shared/network
filesystems are not currently supported execution targets.

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[supervisor]'
```

Notebook kernels that only emit generic Runwatch protocol events can use the
Pydantic-only base install. The CLI, notebook runner, dashboard, notifications, and
built-in AWS/local adapters require the `supervisor` extra shown above.

Install optional NVIDIA monitoring support with:

```bash
python -m pip install -e '.[supervisor,gpu]'
```

For development:

```bash
uv sync --extra supervisor --extra test --extra dev --extra docs
uv run pytest tests
uv run ruff check src tests
```

The selected notebook kernel must be able to import `runwatch` when cells emit resources.

Third-party supervisor adapters can extend Runwatch without adding provider-specific
code to this package. See [Third-party resource adapters](docs/resource-events.md#third-party-resource-adapters)
for the entry-point contract and adapter API.

## First run

```bash
runwatch init-config runwatch.yaml
runwatch validate notebook.ipynb --config runwatch.yaml
runwatch execute notebook.ipynb --config runwatch.yaml
```

`init-config` writes an annotated starter file. `validate` checks nbformat structure,
kernel availability, paths, sharing prerequisites, configured adapters, and terminal
conditions without starting a kernel, dashboard, or provider resource. Resources
emitted dynamically by cells cannot be predicted during preflight.

Runwatch prints the run directory, editable notebook, and pairing URL. With Cloudflare
sharing, that pairing URL is the authenticated localhost dashboard; the current public
link and QR live on that page.
By default the dashboard remains available for 90 seconds after the run reaches a
terminal state, then closes automatically. If the run succeeded or was cancelled,
Runwatch removes that run directory and removes the empty `.runwatch/runs` and
`.runwatch` parents. When
notifications are configured, cleanup first waits up to
`notifications.terminal_drain_timeout_seconds` for terminal event routing and delivery
attempts. If the outbox is still nonterminal, the run is retained and the CLI
prints a reason plus `runwatch open RUN_DIR`; opening it restarts notification delivery
without rerunning the notebook. The recovery controller conservatively retains the run
when that dashboard closes because it did not itself observe normal notebook
finalization. After confirming delivery, remove the retained state explicitly if it is
no longer needed. Set `server.linger_seconds: 0` to close immediately, set it to `null`
to keep the dashboard open until Ctrl+C, or use `--keep-run` to retain successful or
cancelled state after the original execution's dashboard closes.

For a no-AWS replay with live progress, local metrics, file monitoring, log tailing,
and final notebook results, use the repository's
[Runwatch fake session](web_artifacts_fake_sessions/runwatch/README.md):

```bash
web_artifacts_fake_sessions/runwatch/run.sh
```

Every run contains:

```text
.runwatch/runs/<timestamp>-<notebook>-<id>/
├── access-token.txt
├── input.ipynb               # immutable original snapshot
├── source.ipynb              # agent/human editable nbformat document
├── executed.partial.ipynb    # runner-owned rolling checkpoint
├── executed.ipynb            # final executed notebook
├── writeback-state.json      # conflict guard for the user-owned notebook
├── run-manifest.json
└── runwatch.sqlite3
```

Each per-run directory is restricted to the current user (`0700`) and new run-state
files are created with mode `0600`. Retained state is nevertheless sensitive: it can
contain notebook source and output, tracebacks, local paths, resource identifiers and
logs, notification destinations and payloads, actions, and the dashboard bearer token.
The mode of the original notebook passed to `execute` is preserved during write-back.

Runwatch versions its persisted and wire contracts independently. New run manifests
and SQLite databases use schema version 3; configuration and kernel resource events use
schema version 2; S3 progress manifests use schema version 1. Legacy schema-version-2
run directories reopen conservatively. Runwatch intentionally does not migrate 0.1 run
directories.

## Run from VS Code

Turn the active notebook into a Runwatch job from VS Code's task picker. The setup
below is project-local: it uses the project's `.venv`, reads notification secrets from
an ignored `.env`, prepares a matching Jupyter kernel, and runs whichever saved
notebook is active in the editor.

First install Runwatch, `ipykernel`, and the `dotenv` command in the project's virtual
environment:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install 'runwatch-notebook[supervisor]' ipykernel 'python-dotenv[cli]'
```

Create `.vscode/runwatch.yaml`:

```yaml
notebook:
  kernel_name: runwatch-workspace

host:
  prevent_system_sleep: true

notifications:
  ntfy_base_url: ${RUNWATCH_NTFY_BASE_URL}
  ntfy_topic: ${RUNWATCH_NTFY_TOPIC}
  ntfy_on_section_start: true
```

Keep the ntfy destination in the project's `.env` and make sure that file is ignored
by Git:

```dotenv
RUNWATCH_NTFY_BASE_URL=https://ntfy.sh
RUNWATCH_NTFY_TOPIC=your-private-topic
```

Then add the following tasks to `.vscode/tasks.json` (merge them into its existing
`tasks` array if the project already has one):

```json
{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "notebook: prepare Runwatch kernel",
      "type": "process",
      "command": "${workspaceFolder}/.venv/bin/python",
      "args": [
        "-m",
        "ipykernel",
        "install",
        "--prefix",
        "${workspaceFolder}/.venv",
        "--name",
        "runwatch-workspace",
        "--display-name",
        "Runwatch workspace (.venv)"
      ],
      "problemMatcher": []
    },
    {
      "label": "notebook: run active notebook",
      "type": "process",
      "command": "${workspaceFolder}/.venv/bin/dotenv",
      "args": [
        "--file",
        "${workspaceFolder}/.env",
        "run",
        "--",
        "${workspaceFolder}/.venv/bin/runwatch",
        "execute",
        "${file}",
        "--config",
        "${workspaceFolder}/.vscode/runwatch.yaml",
        "--share",
        "cloudflared"
      ],
      "options": {
        "cwd": "${workspaceFolder}",
        "env": {
          "PATH": "${workspaceFolder}/.venv/bin:${env:PATH}",
          "JUPYTER_PATH": "${workspaceFolder}/.venv/share/jupyter:${env:JUPYTER_PATH}"
        }
      },
      "dependsOn": "notebook: prepare Runwatch kernel",
      "dependsOrder": "sequence",
      "problemMatcher": [],
      "presentation": {
        "reveal": "always",
        "panel": "dedicated",
        "clear": true
      }
    }
  ]
}
```

Save and focus the `.ipynb` you want to run, open **Tasks: Run Task** from the Command
Palette, and select **notebook: run active notebook**. The terminal shows the Runwatch
dashboard pairing URL while the task keeps running. This example uses a Cloudflare
quick tunnel, so `cloudflared` must be available on `PATH`; change the final share value
to `lan` for a trusted local network or `none` for a local-only dashboard.

The task configuration enables sleep inhibition and ntfy section announcements. If
notifications are not wanted, remove the `notifications` block from
`.vscode/runwatch.yaml` and invoke `runwatch` directly instead of through `dotenv`.

## Keep the execution host awake

For an unattended run on macOS or Linux, opt into native idle-sleep inhibition:

```yaml
host:
  prevent_system_sleep: true
```

macOS uses an IOKit `NoIdleSleepAssertion`; Linux holds a logind inhibitor through
`systemd-inhibit`. The inhibitor starts before the notebook runtime and remains active
through execution, blocking-resource waits, failure pauses, recovery, cancellation, and
finalization. It is released before the post-run dashboard linger begins and during
abnormal supervisor cleanup. Display sleep is unaffected.

Linux hosts need `systemd-inhibit` and a reachable logind service. Runwatch fails the
start if the explicitly requested inhibitor cannot be acquired; `runwatch validate`
checks that the platform backend can be constructed before execution. Windows is not a
supported execution target.

## Failure and recovery

After every settled cell attempt, Runwatch atomically writes the executed notebook
state back to the notebook passed to `execute`. This includes repaired source,
execution counts, outputs, and tracebacks. If that notebook changes outside Runwatch,
write-back stops rather than overwriting the external edit, and the run-owned partial
checkpoint is retained.

Rolling checkpoints serialize an immutable notebook generation on the event-loop
thread before filesystem I/O. Requests that arrive during a write remain pending, and
transient checkpoint failures are journaled and retried with bounded backoff until the
worker recovers. Checkpoint and write-back publication use a temporary file, file
`fsync`, atomic replacement, and parent-directory `fsync` where the local filesystem
supports them.

Original-notebook conflict detection is a best-effort portable compare-and-replace. It
checks a content and metadata fingerprint again immediately before `os.replace`, which
detects external saves during preparation and makes the remaining race very small. No
portable filesystem primitive can make the final comparison and rename indivisible, so
another writer in that last window can still be overwritten. Avoid editing the original
notebook while Runwatch is executing; edit the run-owned `source.ipynb` for recovery.

When a cell fails or reaches its configured timeout, Runwatch persists its outputs and
traceback, pauses the notebook, and keeps the kernel and resource monitors alive. A
timed-out kernel is interrupted and synchronized before live resume is allowed. Edit
`source.ipynb` with normal `nbformat`:

```python
from pathlib import Path

import nbformat

path = Path(".runwatch/runs/.../source.ipynb")
notebook = nbformat.read(path, as_version=4)
notebook.cells[6].source = "result = repaired_input()"
nbformat.write(notebook, path)
```

If only the failed or future cells changed, resume in the live kernel:

```bash
runwatch resume .runwatch/runs/...
```

If imported source or an already-executed cell changed, create a new kernel and replay:

```bash
runwatch restart .runwatch/runs/...
```

Replay starts at cell zero. An explicit override may start later, but the operator owns
reconstruction of any missing kernel state:

```bash
runwatch restart .runwatch/runs/... --from-cell 4
```

`N` is zero-based and must identify a cell in the current `source.ipynb`.

If the Runwatch process is no longer live, `resume` first journals the recovery action,
reopens the persisted run, restores resource monitors and cursors, starts a new kernel
epoch, and replays from cell zero. A crash during recovery leaves an action that the
next invocation can safely recover.

Cancellation is bounded rather than relying on one cooperative interrupt. Runwatch
persists `cancelling`, then advances through kernel interrupt, graceful shutdown,
provisioner terminate, and provisioner kill using the four `notebook.cancel_*_grace_seconds`
settings. Stage failures are journaled, repeated cancellation requests are idempotent,
and an execution client that still does not return is detached so the run can settle as
`cancelled` instead of hanging indefinitely.

For an agent-oriented dossier:

```bash
runwatch context .runwatch/runs/... --format markdown
runwatch context .runwatch/runs/... --json
```

## Resource emission

Resources are emitted as structured Jupyter MIME output immediately after they are
created or selected. Runwatch injects the active run, cell, attempt, and kernel epoch.

### SageMaker Processing

```python
from runwatch import aws

sagemaker.create_processing_job(**request)
aws.emit_owned_sagemaker_processing_job(
    request["ProcessingJobName"],
    region="us-east-1",
    logical_key="feature-build",
    output_prefixes=["s3://bucket/run/output/"],
)
```

SageMaker Processing is blocking but borrowed and observation-only by default. Use
`emit_owned_sagemaker_processing_job` only for a job the current run created and may
stop; that explicit helper claims exclusive ownership and enables provider stop during
run cancellation. Existing calls that explicitly pass `stop_on_cancel=True` continue
to opt into exclusive ownership. Set `ownership="exclusive"` with
`stop_on_cancel=False` when manual stop should be available without joining the
cancellation cascade.

### S3 prefix

```python
aws.emit_s3_prefix(
    "s3://bucket/run/output/",
    expected_count=400,
    completion_marker="_SUCCESS",
    blocking=True,
    full_rescan_seconds=300,
)
```

A blocking prefix must define an expected count or completion marker. Truncated scans
continue from a persisted key on the next poll instead of restarting at page one. After
the initial reconciliation, polls scan only keys after the last observed key and perform
a full reconciliation every `full_rescan_seconds` (five minutes by default). Incremental
counts therefore assume append-only keys between reconciliations; the dashboard exposes
the scan mode, reconciliation time, and lower-bound warning. Set
`full_rescan_seconds=0` when every poll must be a full exact scan.

### S3 manifest

```python
aws.emit_s3_manifest("s3://bucket/run/progress.json", blocking=True)
```

The manifest is a small JSON document:

```json
{
  "schema_version": 1,
  "status": "running",
  "completed": 180,
  "total": 400,
  "message": "Building features",
  "metrics": {"rows": 9000000}
}
```

`status` must be `running`, `completed`, or `failed`; metrics must be finite scalar
values and cannot override reserved manifest fields. Manifests are capped at 1 MiB.

### CloudWatch

```python
aws.emit_cloudwatch_metric(
    namespace="MyPipeline",
    metric_name="ChannelsProcessed",
    dimensions={"RunId": "abc"},
)

aws.emit_cloudwatch_logs(
    log_group="/aws/my-pipeline",
    stream_prefix="run-abc/",
)
```

CloudWatch metrics and logs are always nonblocking. Metric cards retain the full current
lookback window, while observation history stores only new or revised timestamp samples
and bounds its deduplication cursor to 1,440 entries. Log stream discovery rotates across
bounded pages when more streams exist than can be displayed at once, shares each poll's
line budget across busy streams, and prunes tokens after a complete discovery rotation.
Terminal SageMaker jobs drain paginated logs until caught up or report an explicit
truncation.

### Local system, files, and dashboards

```python
from runwatch import local

local.emit_system_metrics(include_host=True, include_kernel=True, gpu="all")

local.emit_file_count(
    "artifacts/parts",
    pattern="*.parquet",
    expected_count=400,
    blocking=True,
)

local.emit_line_count(
    "logs/inference.log",
    expected_lines=10000,
    tail_lines=100,
    blocking=True,
)

local.emit_dashboard(
    "http://127.0.0.1:8501",
    name="Training dashboard",
    health_path="/_stcore/health",
    logical_key="training-dashboard",
)
```

CPU and memory use `psutil`. NVIDIA metrics use optional NVML support and degrade to an
`nvidia_available=false` metric when unavailable. File-count scans stream metadata into
exact count, total-byte, and newest-modification-time aggregates without retaining one
stat object per file. That settlement signature is an aggregate, not content identity;
content edits that preserve all three values may not reset settlement. Line monitoring
is optimized for append-only logs: it detects replacement, truncation, changes near the
committed offset, and non-growing files whose mtime changes, while keeping partial-line
memory bounded. An earlier in-place edit combined with later file growth can remain
undetected if it does not touch the offset fingerprint. Conversely, a metadata-only mtime
change on a non-growing file is conservatively treated as a rewrite and may replay the
bounded log tail. `line_count` includes records terminated by LF or CRLF; a CR is counted
once a following byte proves it is not the start of CRLF. A final unterminated fragment,
including a trailing lone CR, is excluded until completed and is reported through
`partial_line_pending`, `partial_line_buffered_bytes`, and `partial_line_truncated`.

`emit_dashboard` registers an already-running local web application. Runwatch monitors
its health and adds an **Open Training dashboard** action to the resource card. With
`--share lan`, that action opens a separate authenticated LAN reverse proxy; with
`--share cloudflared`, it opens a separate authenticated quick tunnel. HTTP streaming,
redirects, cookies, WebSockets, and SSE pass through at the proxy root, which avoids
breaking applications that do not support a configurable path prefix.

Only explicit `localhost` and loopback IP URLs are accepted. Linked dashboards are
external, nonblocking, and observation-only: Runwatch neither starts nor stops the
application. The registration survives Runwatch recovery, while its proxy port and
tunnel URL are recreated for the current process and are never persisted in dashboard
state. Stop the linked application through its own controls or local process manager.

### Notebook progress

```python
from runwatch import emit_progress

emit_progress(180, total=400, unit="partitions", message="Building features")
```

Python kernels also mirror `tqdm`, `tqdm.auto`, and `tqdm.notebook` bars into the
same dashboard progress area without notebook changes:

```python
from tqdm.auto import tqdm

for item in tqdm(items, desc="Building features", unit="items"):
    process(item)
```

Runwatch preserves tqdm's normal notebook output and emits structured updates at most
twice per second by default. Updates reuse one hidden notebook display per bar, so a
long loop does not append an output for every refresh. The dashboard scopes progress
to the current cell and prefers the outermost bar when bars are nested. It keeps tqdm's
recent-rate ETA and adds a Bayesian median finish time with an 80%
posterior-predictive range
using the current run's elapsed time and completed items. It compares a shared-rate
model with a two-rate model, including the posterior probability that the new regime is
materially slower. When that probability remains high, the estimate recalibrates from
the inferred cache boundary instead of treating cache hits as real-work throughput.
It then keeps testing recent windows against the current regime in both directions,
requiring 99.99% posterior probability before rebasing. Large rate shifts accumulate
that evidence quickly, while smaller shifts require longer observations. Set
`notebook.capture_tqdm: false` to disable automatic capture, or adjust
`notebook.tqdm_min_interval_seconds`. Progress created in a separate process is outside
the notebook kernel and is not captured automatically.

### Notebook section notifications

Runwatch can announce notebook section transitions through ntfy:

```yaml
notifications:
  ntfy_base_url: https://ntfy.sh
  ntfy_topic: your-private-topic
  ntfy_on_section_start: true
```

After a non-empty code cell finishes, Runwatch inspects the Markdown cells before the
next non-empty code cell. If they contain headings, it emits a durable section-start
event and sends the nearest heading to ntfy immediately before that next code cell
begins. The notification is ntfy-only even when generic webhooks are also configured.
The first code cell is not announced because no preceding code cell has completed.

Heading text leaves the machine when this option is enabled. Keep secrets out of
Markdown headings and use a private ntfy topic.

## Dashboard and remote stop

The dashboard shows notebook state, reported progress, outputs, tracebacks, resource
metrics and charts, log tails, and the durable event journal.

The dashboard header provides an **Open notebook** action beside the ntfy action. It
opens a separate, authenticated, read-only rendering of the saved notebook in a new
tab. While the notebook is executing, the view uses `executed.partial.ipynb`; after
notebook cells finish, it uses the final output; before the first checkpoint, it renders
the saved source. The page does not update automatically: the browser's normal refresh
control (including pull-to-refresh on mobile) loads the newest durable save. The page
reports the snapshot timestamp and settled-cell count. On mobile, scrolling down in the
notebook collapses that metadata header; selecting its compact summary expands it again.

This full-notebook view is intentionally different from the bounded dashboard timeline:
it can contain all notebook source and saved outputs. Runwatch removes active and
navigation-capable HTML, omits JavaScript-only outputs, and isolates the rendering in a
sandboxed frame with scripts and network access disabled. The child retains only its
same-origin identity so the trusted wrapper can observe scroll position for the compact
mobile header. Treat access to it as access to the notebook itself.

The browser API is an explicit presentation model rather than a dump of SQLite state.
It allowlists display-safe run, cell, resource, metric, and event fields and omits
notebook source, resolved configuration, notification endpoints, provider cursors and
raw responses, controller tokens, and dedicated internal-path fields. Output,
traceback, log, and chart payloads are bounded, but their user-generated text can still
contain sensitive values. SSE carries only sequence, timestamp, and event type as an
invalidation signal; the browser then refreshes the sanitized snapshot. Dashboard
pages, snapshots, SSE, and authenticated redirects use `Cache-Control: no-store`.

Registered localhost dashboards appear as normal resource cards. Their Open action is
available only while the authenticated share is ready and uses the same pairing session
as the Runwatch dashboard.

Only active, exclusive resources whose adapter supports stop show a Stop button. The
confirmation lists the selected job and every other resource affected by cancellation.
Confirming the action interrupts the notebook, stops all eligible owned resources, and
finishes the run as `cancelled`. The provider resource is revalidated and accepts the
stop request before notebook cancellation begins; a stale or superseded resource has no
cancellation side effect.

The equivalent local command is:

```bash
runwatch resource stop RUN_DIR RESOURCE_ID
```

## Other CLI commands

```text
runwatch status RUN_DIR [--json]
runwatch validate NOTEBOOK [--config PATH] [--json]
runwatch events RUN_DIR [--follow] [--json]
runwatch open RUN_DIR
runwatch notifications rotate RUN_DIR --config PATH
runwatch notifications purge RUN_DIR --yes
runwatch version
```

`open` serves persisted state without starting notebook execution and retries durable
notification delivery. It conservatively retains the run when the dashboard closes:
only the controller that observed normal notebook finalization may authorize automatic
run cleanup.

`status`, `context`, and `events` emit bounded presentation schema version 1 in JSON
mode. They expose recovery-relevant lifecycle fields and safe summaries, not the raw
SQLite snapshot, resolved notification configuration, controller credentials, or raw
event payloads.

Notification credential maintenance is offline and lock-fenced. `notifications
rotate` reads only the notification settings from `--config`, atomically records them
as desired state in the run manifest, then rewrites every persisted delivery for the
same webhook/ntfy topology. It also rearms failed deliveries, clears legacy transport
errors, and sanitizes pre-presentation intents. To change topology, purge the old
outbox first; a subsequent rotate may enable the new topology because no old delivery
rows remain. `notifications purge --yes` disables routing without replay, removes the
outbox, scrubs notification diagnostics, and best-effort compacts SQLite.

Runwatch validates resource lifecycle semantics at every entry point—not just in the
convenience emitters. Metrics, logs, and system monitors are always nonblocking;
conditional S3 and local-file monitors require a concrete terminal condition; and
`stop_on_cancel` requires an exclusive adapter that implements stop.

## Sharing and security

Runwatch requires CurveZMQ encryption for the local Jupyter manager-to-kernel
channels and fails kernel startup rather than falling back to plaintext TCP. The
kernel must advertise Curve support; IPython kernels require `ipykernel>=7.3`.

Localhost is the default. For a trusted LAN:

```bash
runwatch execute notebook.ipynb --share lan
```

For a temporary public tunnel, install `cloudflared` and use `--share cloudflared`.
Runwatch prints only the local loopback pairing URL. Open it to get the current public
Cloudflare link and QR. Runwatch checks that public route throughout the run and, after
repeated failures, replaces only the tunnel: the notebook kernel and local dashboard
keep running while the link and QR update. If ntfy is configured, the replacement
pairing URL is also sent as the notification's clickable target.

The pairing URL is a bearer credential. Configure ntfy only with a private topic you
trust to receive it. Otherwise keep the URL private. See the
[security guide](docs/security.md).

## Agent workflow

Runwatch does not launch or communicate with agents. A repository-local skill under
`agent_skills/runwatch/` teaches Codex or Claude to inspect `runwatch context`, edit
`source.ipynb`, choose resume versus restart, and verify the resulting state.

Install or update that skill for Codex, VS Code Copilot, and Claude Code with:

```bash
uv run python scripts/install_skills.py update --target all
```

The installer copies the skill to the personal skill directories for each selected
agent. Pass `runwatch` instead of `update` for a non-replacing first installation, or
use `--target codex`, `--target copilot`, or `--target claude` for one destination.

## License

MIT. See [LICENSE](LICENSE).
