Metadata-Version: 2.4
Name: daylily-ephemeral-cluster
Version: 19.0.124
Summary: Infrastructure-as-code for ephemeral AWS ParallelCluster environments for bioinformatics
Author-email: Daylily Informatics <daylily@daylilyinformatics.com>
License-Expression: GPL-3.0-only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: ruamel.yaml>=0.18.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: typer<1,>=0.12
Requires-Dist: cli-core-yo==2.2.0
Requires-Dist: rich<15.0.0,>=14.0.0
Requires-Dist: setuptools<81
Requires-Dist: setuptools-scm>=8
Requires-Dist: requests>=2.31.0
Requires-Dist: tabulate>=0.8.10
Requires-Dist: python-dateutil>=2.8.2
Requires-Dist: aws-parallelcluster==3.15.0
Requires-Dist: typeguard~=2.13.3
Requires-Dist: pytest>=7.4.0
Requires-Dist: pytest-cov>=4.1.0
Requires-Dist: moto>=4.2.0
Requires-Dist: black>=23.0.0
Requires-Dist: ruff>=0.1.0
Requires-Dist: mypy>=1.5.0
Requires-Dist: boto3-stubs[ec2,s3]>=1.28.0
Requires-Dist: ipython>=8.0.0
Requires-Dist: prompt-toolkit<3.0.52,>=3.0.41
Requires-Dist: yamllint>=1.35.0
Provides-Extra: reports
Requires-Dist: reportlab<5,>=4.0; extra == "reports"
Dynamic: license-file

# Daylily Ephemeral Cluster

Daylily Ephemeral Cluster, usually called DYEC or DayEC, is the CLI control plane for short-lived AWS ParallelCluster bioinformatics work. Release `19.0.91` creates and configures clusters, mounts sequencing-run data into FSx, launches pinned workflow repositories on the headnode, monitors exact analysis roots, moves files between local and headnode storage, and exports finished results to S3 with receipts.

DYEC is not an identity service and not a workflow engine. It does not require or contact a metadata or identity service. It consumes explicit local configuration, explicit manifests, explicit S3 paths, and explicit command-catalog entries. DayOA owns its workflow rules and `dy-r` execution. DYEC owns cluster/headnode orchestration and the launch/export envelope.

> **Agent and operator starting point:** read [docs/agent_cli_guide.md](docs/agent_cli_guide.md) before operating a cluster or DayOA analysis. It maps the common DYEC CLI paths, the interactive DayOA contract, and the stop conditions. `dyec agent guidance` is the matching compact terminal reminder.

## Current operator model

Most work follows this shape:

```bash
cd /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster
source ./activate

export AWS_PROFILE=lsmc
export REGION=us-west-2
export CLUSTER=ifx-p2-1000-120-0715
export STAGING_S3_URI=s3://<bucket>/<temporary-dyec-payload-prefix>/
export ANALYSIS_ID=<analysis-id>
```

Inspect the installed CLI before mutating anything:

```bash
dyec --json version
dyec --help
dyec agent guidance
dyec --json catalog list --dyec-version 19.0.91
```

### Optional project-local context

When working repeatedly from one DYEC checkout, save the four recurring DYEC
values in its ignored local context file rather than exporting a second set of
environment variables:

```bash
dyec set-vars \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --region-az us-west-2d \
  --cluster-admin-email operator@example.org

dyec -v --json info
```

The project-local context loader reads only `$PWD/.dyec.config.yaml`; it does
not use environment variables as a second local-context store. The file
accepts only the four documented string keys;
malformed YAML, unknown keys, and non-string values fail clearly. Blank and
whitespace-only values are unset. For `--profile`, `--region`, and
`--region-az`, an explicit flag wins over this local file, which wins over the
command's existing behavior. A required option may therefore be omitted only
when its matching local value is present. Use explicit flags to override the
local context for one command, `dyec unset-vars --region` to clear one value,
or `dyec unset-vars` to clear the file. `-v` prints the resolved local-context
diagnostic to stderr before the subcommand, preserving JSON stdout.

Use `--cluster` for DYEC commands. Keep `--cluster-name` for tools such as `pcluster` that require that spelling.

## Safety contracts

- Use `dyec`; do not launch DayOA by invoking raw `snakemake`.
- Upstream services use the installed `dyec` console script for every cluster
  lifecycle operation. They do not run `pcluster`, import DYEC Python
  internals, or use a module entrypoint as an alternate control path.
- Saved cluster templates keep `SpotPrice: CALCULATE_MAX_SPOT_PRICE` on every
  Spot compute resource. Root `dyec create` resolves the normal config and
  performs live spot-price calculation before provider dry-run/create; saved
  templates never persist a calculated numeric bid.
- New DayOA checkouts must be explicit-tag checkouts.
- A DYEC controller never mutates a pinned DayOA release: no runtime rule/script/environment/config patches, source overlays, or generated helpers in the checkout. It verifies the selected ref is clean before dispatch and after the workflow returns. Missing behavior is a hard error that must be fixed and released in DayOA, never repaired on the headnode.
- Headnode work uses a cluster-appropriate remote user selected by platform: Ubuntu/Intel DayOA headnodes use `ubuntu`; DRAGEN/RHEL-style headnodes use `ec2-user`.
- DYEC-created headnode shells must be bash login/interactive contexts and source `~/.bashrc`; workflow controllers still run in persistent `tmux` panes.
- DYEC CLI launch helpers create the supported headnode controller for you; they do not require an interactive SSM session for standard catalog launches.
- Missing files, missing credentials, unsafe identity fields, malformed manifests, unexpected legacy input shapes, and insufficient staging permissions fail hard.
- S3 relay prefixes for upload/download and staged workflow launch must be readable/writable by both the local operator credentials and the headnode instance role.
- `/fsx/analysis_results/**` workflow writes, unlocks, deletes, restarts, and kills require analysis-root write-lock ownership.
- Headnode inspection commands are supported; Slurm/node administration is not implied by inspection.

## Command groups

Run `dyec --help` for the live list. Current major groups are:

| Group | Purpose |
|---|---|
| `version`, `info`, `runtime`, `env`, `resources`, `state`, `set-vars`, `unset-vars` | Versioned resource/runtime introspection and per-checkout local context. |
| `agent` | Compact operational guidance for an automated or human operator. |
| `preflight`, `create`, `drift`, `delete` | Cluster lifecycle; root create retains the original profile/region-AZ/cluster-type entrypoint and automatic resolution. |
| `create-request` | Standalone protected request and admission tooling; it is not required or invoked by root create. |
| `cluster`, `cluster-info` | ParallelCluster inspection, guarded all-resource MaxCount updates, compute-fleet lifecycle, and tag helpers. |
| `headnode` | SSM-backed headnode connection, command execution, file transfer, and observability. |
| `mounts`, `mount` | FSx run-directory Data Repository Associations. |
| `workflow`, `repositories`, `catalog` | Standard workflow clone/launch/status helpers plus repository and command-catalog discovery, exact rendering, and launch. |
| `samples` | Older sample staging/launch helpers for catalog contracts that still use them. |
| `identities` | Provider-neutral local manifest validation and receipt handling; no network service calls. |
| `analysis` | Analysis-root visit logging, status reporting, lock ownership, and guarded commands. |
| `command` | Command-family progress views such as HIOMRS sample stats and DAG download. |
| `export`, `exports`, `runtime-cache` | FSx analysis export and DRA-only runtime-cache preservation through explicit S3 receipts. |
| `pricing`, `cost-centers`, `aws`, `slurm-accounting` | Cost, quota, AWS readiness, read-only accounting-topology inspection, and guarded accounting support. |
| `overview`, `cost-controls`, `prompt`, `cockpit` | Typed read-only regional state, explicit budget/cost-center status, a nonblocking cached shell summary, and a Glances-style evidence cockpit. |
| `theme`, `shell` | DYEC-local palettes, explicit terminal tint apply/restore, and opt-in Zsh/Bash prompt initialization. |
| `tests` | Local pytest and catalog validation helpers. |

Detailed option-level examples live in [docs/cli_reference.md](docs/cli_reference.md). The task-oriented path is [docs/agent_cli_guide.md](docs/agent_cli_guide.md).

## Guarded AWS Budget updates

`dyec aws budget set-limit` changes only an existing fixed monthly USD AWS
Budget. The caller must provide the exact currently observed cap, and can prove
the whole contract without mutating AWS by using `--dry-run`:

```bash
dyec aws budget set-limit "$CLUSTER" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --expected-current-monthly-cap-usd 200 \
  --monthly-cap-usd 300 \
  --dry-run
```

After the required budget-change approvals, repeat the same command without
`--dry-run`. A stale current-cap value, an unsupported budget shape, or a
readback mismatch fails explicitly; requesting the already-current cap is a
successful no-op.

## Quick cluster and queue checks

Use the aggregate read-only view to see which clusters are ready and how many
Slurm jobs each has. A provisioning or teardown cluster is shown as
`CLUSTER_NOT_READY`, not as an idle queue.

```bash
dyec cluster jobs --profile "$AWS_PROFILE" --region "$REGION"
```

For the exact job list on one cluster, use the drill-down command:

```bash
dyec headnode jobs --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER"
```

## Read-only overview, prompt, and cockpit

Collect one explicit region as typed JSON, or inspect one exact cluster's two
cost controls separately:

```bash
dyec --json overview snapshot --profile "$AWS_PROFILE" --region "$REGION"
dyec --json cost-controls status \
  --profile "$AWS_PROFILE" --region "$REGION" \
  --cluster "$CLUSTER" --cluster-budget-name "$CLUSTER" \
  --cost-center "${CLUSTER}-ccenter"
```

The overview's cluster burn/hour includes active EC2, EBS, FSx, and public
IPv4. It excludes S3, transfer, NAT, and CUR-delayed costs. An unavailable
required component makes the total unknown (`null` in JSON and `?` in text),
never a false zero.

Configure an ordered, cached shell summary and opt into the generated hook:

```bash
dyec prompt configure --profile "$AWS_PROFILE" --region "$REGION" --label UW2
dyec prompt refresh
eval "$(dyec shell init zsh --prompt)"  # use bash for Bash
```

Prompt rendering reads only
`$XDG_CACHE_HOME/daylily/prompt-v1.json` (or
`~/.cache/daylily/prompt-v1.json`). At the next prompt, a cache at least 15
minutes old starts one locked refresh asynchronously; shell startup and prompt
rendering never wait on AWS.

For the full-screen read-only view and its DYEC-local themes:

```bash
dyec cockpit --profile "$AWS_PROFILE" --region "$REGION"
dyec theme list
dyec theme preview daylight
dyec theme set daylight
```

The cockpit shows its underlying commands and JSON and has no mutation keys.
Terminal tint is a separate explicit `dyec theme terminal apply <theme>` /
`restore` action. See
[overview_prompt_and_cost_controls.md](docs/overview_prompt_and_cost_controls.md)
and [cockpit_and_themes.md](docs/cockpit_and_themes.md).

## CLI-first catalog launch

Catalog launch is the preferred path for known DayOA commands because it
renders the exact `dyec workflow launch` command before it starts anything.
The cluster AWS Budget is `--cluster-budget-name` and must equal `--cluster`;
the Slurm attribution identity is `--cost-center`. If omitted, DYEC resolves
them to the exact cluster and `<cluster>-ccenter`, respectively. See
[docs/cost_context.md](docs/cost_context.md) for the receipt, hash, and narrowly
supported migration aliases.

Render first:

```bash
dyec --json catalog render hybrid_ilmn_ont_hiomr_kitchensink \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --analysis-id "$ANALYSIS_ID" \
  --executing-entity "$CLUSTER" \
  --manifest-dir ./config \
  --payload-staging-s3-uri "$STAGING_S3_URI" \
  --session-name "${ANALYSIS_ID}-dryrun" \
  --cluster-budget-name "$CLUSTER" \
  --cost-center "${CLUSTER}-ccenter" \
  --dry-run
```

Launch the rendered dry run:

```bash
dyec --json catalog launch hybrid_ilmn_ont_hiomr_kitchensink \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --analysis-id "$ANALYSIS_ID" \
  --executing-entity "$CLUSTER" \
  --manifest-dir ./config \
  --payload-staging-s3-uri "$STAGING_S3_URI" \
  --session-name "${ANALYSIS_ID}-dryrun" \
  --cluster-budget-name "$CLUSTER" \
  --cost-center "${CLUSTER}-ccenter" \
  --dry-run
```

If the dry-run plan is bounded and correct and the production contract requires
a same-root continuation, do not issue a second catalog launch against the
existing root. Start a new `dyec workflow launch` controller with
`--reuse-existing-analysis-dir --input-contract none --no-input-staging`, the
exact ref/commit, and the rendered live `dy-r` argv with only `-n` removed.
The full continuation command and its lock boundary are in
[docs/agent_cli_guide.md](docs/agent_cli_guide.md#5-preferred-path-catalog-render-dry-controller-and-live-controller).

For scalar DayOA runtime config, pass explicit `key=value` overrides. DYEC
appends them to the `dy-r ... --config` section and does not reinterpret their
workflow-specific meaning:

```bash
dyec --json catalog render hybrid_ilmn_ont_hiomr_kitchensink \
  --analysis-id hg003-hg004-ds025-ont0to6 \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --manifest-dir ./config \
  --payload-staging-s3-uri "$STAGING_S3_URI" \
  --dry-run \
  --dy-config use_fq_data_starting_hrs=0 \
  --dy-config use_fq_data_up_to_hrs=6 \
  --dy-config global_sr_subsample_pct=0.25 \
  --dy-config global_ont_subsample_pct=0.25
```

Typed YAML analysis configuration must be materialized inside the cloned DayOA
analysis directory, normally under `config/`, and passed to `dy-r` through an
in-clone `--configfile` path. Never retain a controller-specific YAML/config
under `/home/ubuntu`, `/tmp`, or another path outside the clone. Together with
the in-clone manifests and saved command/receipts, the clone is the rerunnable
analysis capsule; only the explicitly declared source reads, CRAMs, references,
licenses, and runtime assets remain external.

Large local payloads are staged through S3 with `--payload-staging-s3-uri`. DYEC uploads a tarball containing input manifests, a payload manifest, and the controller launch script. The headnode downloads and expands that tarball into the workflow run directory, starts the tmux controller, and then saves the exact executed script under `<analysis-root>/bin/dyec-controller-launch.sh` after `day-clone` creates the analysis root. This avoids SSM document-size limits without pre-creating the analysis root.

### Immutable command shapes

Catalog version 6 requires an exact numeric `--dyec-version` for every public
catalog action. The `19.0.26` snapshot is immutable; there is no mutable
`current` alias or implicit catalog selection:

```bash
dyec --json catalog list --dyec-version 19.0.26 --type prod
dyec --json catalog render <command-id> --dyec-version 19.0.26 ...
```

A build may also declare one-hop, same-build aliases. An alias inherits one
direct command, applies typed metadata overrides, and either extends its DayOA
targets/config or supplies a complete replacement of `targets`, `dy_command`,
and `dryrun_dy_command`. Alias chains, cycles, cross-build references, missing
bases, duplicate IDs, mixed extension/replacement modes, and partial command
replacements fail catalog validation. Existing catalog APIs return aliases as
fully resolved `AnalysisCommand` records.

The `19.0.26` catalog preserves the `19.0.25` command shapes and targets DayOA
`16.0.6`. Public catalog output includes a
derived `validation_pending` field: `true` means the command now targets a
different DayOA tag than its retained `validated_version`. It is a visibility
signal only; it does not relabel older validation receipts or block a launch.

Each command may declare an explicit `validation_evidence_s3_uri_prefix`. The
prefix must contain `command_registry.json` and `summary.json` from a successful
`dyec tests command-catalog` run. Compare it read-only with:

```bash
dyec --json catalog validation-compare <command-id> \
  --dyec-version 19.0.26 --profile "$AWS_PROFILE" --region "$REGION"
```

The comparison fails hard when no prefix is declared, the receipts are missing,
the captured DayOA pin differs, or the recorded command phase did not succeed.

## Workflow launch without the catalog shortcut

Use `dyec workflow launch` when you already know the exact DayOA command string or are launching a non-catalog repository command:

```bash
dyec workflow launch \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --analysis-id "$ANALYSIS_ID" \
  --executing-entity "$CLUSTER" \
  --git-tag 16.0.75 \
  --manifest-dir ./config \
  --payload-staging-s3-uri "$STAGING_S3_URI" \
  --session-name "$ANALYSIS_ID" \
  --cluster-budget-name "$CLUSTER" \
  --cost-center "$COST_CENTER" \
  --dry-run \
  --rerun-triggers mtime \
  --dy-command "dy-r produce_sentdhiomr_snv_vcf produce_sentdhiomr_sv produce_sentdhiomr_cnv -j 100 -p -T 0 --rerun-incomplete"
```

`--dry-run` appends `-n` to the effective `dy-r` command even when
`--dy-command` is supplied; do not duplicate `-n` merely to make a direct
launch dry. Use repeatable `--rerun-triggers` for explicit Snakemake trigger
selection (for example, `mtime`). When that public option is present,
`--dy-command` must not also contain `--rerun-triggers`. For the same-root live
continuation, remove only `--dry-run` and retain the same rerun-trigger option,
analysis ID, DayOA commit, manifests, and in-clone configuration.

## Six-manifest DayOA inputs

Current DayOA sample-analysis launches use exactly six manifest files:

```text
specimens.tsv
samples.tsv
libraries.tsv
sequencing_inputs.tsv
analysis_units.tsv
analysis_unit_inputs.tsv
```

Validate them locally before launch:

```bash
dyec --json identities validate --manifest-dir ./config
dyec --json identities status --manifest-dir ./config
```

The identity commands are local and provider-neutral. They never create or resolve production identities. EUID fields may be blank for ordinary analysis. Test-only EUID-like values must use the reserved `Z-` prefix and are not valid customer-release identities.

## Bjuice prevalence manifest helper

`dyec catalog config-bjuice-preval` builds a six-manifest directory from reviewed Bjuice evidence files. It is deliberately specific to that reviewed evidence contract; it is not a general identity resolver.

```bash
dyec --json catalog config-bjuice-preval \
  --sample HG003 \
  --sample HG004 \
  --output-dir ./config-hg003-hg004 \
  --source-manifest-json /path/to/source_manifest_resolved.json \
  --run-evidence-json /path/to/run_evidence_v2.json \
  --library-run-matrix-tsv /path/to/bjuice_preval_library_run_matrix.tsv \
  --sample-metadata-tsv /path/to/samples.tsv \
  --legacy-units-tsv /path/to/units.tsv \
  --sr-subsample-pct 0.25 \
  --ont-subsample-pct 0.25 \
  --profile "$AWS_PROFILE" \
  --region "$REGION"
```

The helper writes the six manifest TSVs plus `bjuice_preval_config_receipt.json`, validates the manifest set, and records file hashes. It fails if reviewed sample metadata, legacy unit metadata, S3 listings, or expected ILMN/ONT source groups are absent or ambiguous.

## Bjuice v2 HG002 full-prevalence multi-AU helper

`dyec catalog config-bjuice-v2-hg002-multi-au` is a separate, fixed contract for the seven HG002 analysis units `p5xp5`, `1x1`, `3x3`, `5x5`, `10x5`, `15x5`, and `15x10`. It accepts one direct Illumina coverage denominator and requires a matching terminal receipt; it never derives coverage from total or hybrid evidence.

```bash
dyec --json catalog config-bjuice-v2-hg002-multi-au \
  --output-dir ./config-hg002-bjuice-v2 \
  --source-manifest-json /path/to/source_manifest_resolved.json \
  --run-evidence-json /path/to/run_evidence_v2.json \
  --library-run-matrix-tsv /path/to/bjuice_preval_library_run_matrix.tsv \
  --sample-metadata-tsv /path/to/samples.tsv \
  --legacy-units-tsv /path/to/units.tsv \
  --direct-ilmn-coverage-x "$C_ILMN" \
  --direct-ilmn-coverage-evidence /path/to/direct_ilmn_terminal_receipt.json \
  --profile "$AWS_PROFILE" \
  --region "$REGION"
```

The receipt must use `dyec.bjuice_v2_direct_ilmn_coverage_receipt.v1`, identify `HG002`, have terminal status, and provide a matching `ilmn_direct_coverage_x`. The output has blank nullable live EUID fields, writes per-AU `ONT_FQ_START_HOUR`/`ONT_FQ_END_HOUR`, and calculates `SUBSAMPLE_PCT = target_x / C_ILMN` at 12 decimal places with `ROUND_DOWN`. It fails before writing output if the receipt is missing or ambiguous, coverage is non-positive, or a requested target exceeds the verified denominator.

## Headnode commands and transfer

Open a full interactive shell when you need one:

```bash
dyec headnode connect \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER"
```

Run a bounded command non-interactively and get stdout/stderr back:

```bash
dyec --json headnode run \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  'hostname; command -v aws; aws sts get-caller-identity --output json'
```

Copy files through an explicit S3 relay:

```bash
dyec --json headnode upload -r ./config \
  /tmp/dyec-staged-config \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --staging-s3-uri "$STAGING_S3_URI"

dyec --json headnode download -r \
  /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID"/daylily-omics-analysis/results/day/hg38/reports \
  ./downloaded-reports \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --staging-s3-uri "$STAGING_S3_URI"
```

The relay prefix is intentionally explicit and auditable. DYEC does not delete relay objects automatically.

## Monitoring exact analyses

### Cluster monitor: current CLI

```bash
dyec cluster monitor \
  --profile "$AWS_PROFILE" --region "$REGION" \
  --clusters "$CLUSTER" [--clusters "$ANOTHER_CLUSTER"] \
  --monitor-type <cluster|headnode|controller|jobs|au> \
  --activity <active|inactive|all> \
  --detail-level <summary|high> \
  --output-mode <ascii|ansi|table|json|fancy|html-server> \
  [--server-port <1024-65535>] [--tail] [--frequency <duration>]
```

Repeat `--clusters` for clusters in the same region. `--activity` applies to
the `controller`, `jobs`, and `au` monitor types. The default mode is `ascii`;
`ansi` and `fancy` require an interactive terminal. `html-server` additionally
requires an explicit non-root `--server-port`.

### Jobs: complete `sq` table

For a full active-queue view, use the monitor directly. It returns every active
`sq` row in `sq` field order, with `CLUSTER` added as the first column:

```bash
source ./activate

dyec cluster monitor \
  --profile "$AWS_PROFILE" --region "$REGION" \
  --clusters "$CLUSTER" \
  --monitor-type jobs --activity active --detail-level high \
  --output-mode table
```

The result uses lossless SSM transport: DYEC does not apply a monitor row or
receipt-size cap. At high detail, `ascii`, `ansi`, `table`, and `fancy` all
preserve this complete table rather than replacing it with a summary. Use
`--output-mode json` for the same complete typed frame.

![Current complete high-detail jobs table](docs/cluster_monitor_19_0_78_assets/jobs-table.png)

### Fancy terminal overview

Use `fancy` for an interactive, compact visual overview of a monitor type such
as controllers. For high-detail jobs, use the table above so no queue field or
row is hidden.

```bash
dyec cluster monitor \
  --profile "$AWS_PROFILE" --region "$REGION" \
  --clusters "$CLUSTER" \
  --monitor-type controller --activity active --detail-level high \
  --output-mode fancy
```

![Current fancy controller monitor](docs/cluster_monitor_19_0_78_assets/controller-fancy.png)

### Live HTML dashboard

`html-server` starts a local, loopback-only dashboard and refreshes its typed
frame at the selected cadence. It needs no root privileges; open the printed
`http://127.0.0.1:<port>/` URL in a browser and use `Ctrl-C` to stop it.

```bash
dyec cluster monitor \
  --profile "$AWS_PROFILE" --region "$REGION" \
  --clusters "$CLUSTER" \
  --monitor-type jobs --activity active --detail-level high \
  --output-mode html-server --server-port 8891 --frequency 5m
```

The page shows the complete active `sq` table with `CLUSTER` first and updates
it in place. The typed monitor frame remains available in the page for direct
inspection.

![Live updating jobs dashboard](docs/cluster_monitor_19_0_78_assets/jobs-html-server.png)

For a workflow launched by DYEC, inspect the exact run-state receipt, controller
target, active Snakemake log, progress, submitted/finished jobs, and current
Slurm states together:

```bash
dyec --json workflow status \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --session "$ANALYSIS_ID"

dyec --json workflow status \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --session "$ANALYSIS_ID" --rule sentdhiomr2_hybrid_cli172i_core

dyec workflow logs \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --session "$ANALYSIS_ID" \
  --stream snakemake \
  --lines 200

dyec workflow logs \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --session "$ANALYSIS_ID" --stream snakemake \
  --match "Error in rule" --before-lines 40 --after-lines 80 --max-matches 1
```

`workflow status` emits exactly one derived `state`: `RUNNING`, `SUCCEEDED`,
`FAILED`, or `UNKNOWN`. `SUCCEEDED` and a terminal exit code require the
matching DYEC `status.json`; a successful status-inspection command is never
the workflow exit code. `CONFIGURING` and `RUNNING` Slurm jobs are ongoing
work, and an empty queue is never success. Failure detection uses anchored
Snakemake terminal markers and deliberately ignores generic `ERROR` text and
printed shell bodies.

Submitted, finished, and active Slurm record collections are complete in the
transport payload. An exact `--rule` selector adds job and unique-AU counts for
that rule, including finished work and each current Slurm state; returned AU
names are complete as well.

The Snakemake stream attributes and reads the log in one remote probe. Its
requested tail is compressed, integrity-checked, and decoded locally. The
selected `--lines` tail is returned in full through the lossless transport.
Literal `--match` mode searches the complete exact attributed log and returns
the complete user-selected context, allowing old errors to be retrieved after a
verbose `-p` log has rolled beyond the requested tail.

For a controller started manually during recovery, provide its identity
explicitly; DYEC does not discover a checkout or guess the newest log:

```bash
dyec --json workflow status \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --repo-path /fsx/analysis_results/<owner>/<analysis-id>/daylily-omics-analysis \
  --controller-pid <pid> \
  --session <exact-tmux-session>
```

If the controller is no longer live or descriptor correlation is unavailable,
add the exact
`--snakemake-log <repo-path>/.snakemake/log/<timestamp>.snakemake.log`.
Manual inspection can prove `RUNNING` or a high-signal `FAILED` state, but it
cannot prove `SUCCEEDED` or invent a terminal RC without a matching DYEC launch
receipt.

DYEC-launched controllers redirect output directly to the regular
`.dyec/controller.log` file; they never put `dy-r` behind `tee`. Immediately
after `dy-r` returns, the launcher atomically records `workflow_completed_at`
and `workflow_exit_code` in the matching `status.json`, before DAG evidence,
export, or other post-processing. Final controller `completed_at`/`exit_code`
fields take precedence if later post-processing changes the launch outcome.
For an exceptional manual recovery, likewise redirect `dy-r` directly to a
regular file and follow it from a separate `tail -f` process. Do not use
`dy-r ... | tee ...`: orphaned workflow helpers can inherit the pipe and delay
the shell from persisting its RC. A manual printed `RETURN CODE` remains
non-authoritative to `workflow status`; use the standard DYEC launcher when a
terminal success receipt is required.

Record visits before analysis-root reads:

```bash
dyec analysis visit \
  --analysis-root /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID" \
  --mode read \
  --intent "status check"
```

Use the exact-root status reporter:

```bash
dyec analysis status slim \
  --analysis-root /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER"

dyec analysis status full \
  --analysis-root /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --tail-lines 1000
```

For HIOMR command-family progress and optional DAG PNG download:

```bash
dyec --json command sample-stats hiomr-kitchensink \
  --name hg003_hiomrs \
  --analysis-root /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --dag-output ./tmp/hg003_hiomrs_dag.png
```

Queue emptiness is never success. Success requires controller exit code plus expected terminal artifacts for the workflow.

## Run-directory QC commands

Run QC catalog entries are run-context commands. Mount the run directory first, then pass a run-context TSV to `catalog render` or `catalog launch`.

```bash
dyec --json mounts create s3://<sequencing-run-bucket>/<run-prefix>/ \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --platform ILMN \
  --read-only \
  --wait \
  --timeout-seconds 5400
```

`run_context` requires all of these exact TSV headers. For a
`run_dra_required` command, `RUN_DIR` is the verified mounted path and
`MOUNT_ID` is the explicit mount identifier; do not substitute the old
three-column shape:

```tsv
RUNID	PLATFORM	RUN_DIR	SOURCE_S3_URI	MOUNT_ID	SAMPLE_SHEET	BASECALLING_STATE	RUN_STATUS	OUTPUT_ROOT	REGION	PROFILE
```

Render each supported run-QC catalog command before launching it:

```bash
dyec --json catalog render illumina_run_qc \
  --analysis-id ilmn-runqc-20260722 \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --run-context-file ./runs.tsv \
  --dry-run

dyec --json catalog render ont_run_qc \
  --analysis-id ont-runqc-20260722 \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --run-context-file ./runs.tsv \
  --dry-run

dyec --json catalog render ultima_run_qc \
  --analysis-id ultima-runqc-20260722 \
  --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER" \
  --run-context-file ./runs.tsv \
  --dry-run
```

The catalog enforces each command’s required `PLATFORM` value.

## Export

Export one completed analysis directory:

```bash
dyec analysis visit \
  --analysis-root /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID" \
  --mode export \
  --intent "export completed pipeline results to $DESTINATION_S3_URI without FSx cleanup"

dyec export \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --source-path /fsx/analysis_results/"$CLUSTER"/"$ANALYSIS_ID" \
  --destination-s3-uri "$DESTINATION_S3_URI" \
  --destination-policy new \
  --output-dir ./export-receipts/"$ANALYSIS_ID"
```

`dyec catalog list`, `show`, and `render` expose the same `result_export`
contract. DayOA never exports: after the controller succeeds, run the displayed
DYEC visit and DRA export commands from the analysis root. The default
`--destination-policy new` requires an empty prefix. Explicit
`--destination-policy update-existing` adds new objects and replaces objects
for modified FSx files in an existing exact destination. It never deletes S3
objects for files absent from FSx and is not a mirror or two-way sync.
`dyec export` records a local receipt and uses an explicit DRA/export path.
Verify `status=success`, `phase=complete`, `task_lifecycle=SUCCEEDED`,
`detached=true`, `s3_delete_requested=false`, and the expected S3 objects. FSx
data is preserved unless a separately approved FSx cleanup is explicitly run;
such cleanup never deletes S3 data.

## Development and tests

Use the repo environment:

```bash
cd /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster
source ./activate
python -m pytest tests/test_cli_registry_v2.py -q
git diff --check
```

The active catalog has release-agnostic local contract coverage. Historical
catalog snapshots remain in immutable Git tags rather than the ordinary test
suite:

```bash
python -m pytest \
  tests/test_current_catalog_contract.py \
  tests/test_tests_runner.py \
  -q
```

Release tags are numeric, annotated semver tags with no leading `v`. Do not move pushed tags. The checked-in release target is `19.0.26`; tag only the exact clean release commit after required acceptance.
