140:| 1077 | 2026-06-06T05:40:15Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Launch ONT mounted FASTQ kitchen-sink dry-run via jem-dev day-clone | AWS_PROFILE=lsmc dyec workflow launch --profile lsmc --region us-west-2 --cluster hyb-only --samples-file /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_samples.tsv --units-file /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_units.tsv --session-name hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun --analysis-id hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun --executing-entity ubuntu --git-tag jem-dev --genome hg38_broad --dy-command bin/day_run\ produce_alignstats\ produce_sentdont_snv_vcf\ produce_snv_concordances\ produce_relatedness\ produce_vep\ produce_multiqc_all\ --config\ \'multiqc_qc=\{\"enable_tools\":\[\"vep\"\]\}\'\ -p\ -j\ 5\ -k\ -n --dry-run | 1 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01077_ont_dryrun_launch.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01077_ont_dryrun_launch.stderr.txt | dyec_workflow_launch | no | none | dry-run includes -n; source FASTQs are mounted paths |
154:| 1091 | 2026-06-06T05:47:42Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Write ONT generated samples/units TSVs to headnode in SSM-safe chunks | python /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/put_remote_config_chunks.py --instance-id i-05374380b57fad901 --region us-west-2 --profile lsmc --remote-dir /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_samples.tsv /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_units.tsv | 1 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01091_ont_remote_config_chunk_put.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01091_ont_remote_config_chunk_put.stderr.txt | ssm_file_transfer | yes | rm -rf /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config | config TSVs only; no FASTQ data; needed because local --samples-file/--units-file exceeded SSM document size |
155:| 1092 | 2026-06-06T05:48:27Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Retry ONT config TSV chunk transfer with short SSM comments | python /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/put_remote_config_chunks.py --instance-id i-05374380b57fad901 --region us-west-2 --profile lsmc --remote-dir /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_samples.tsv /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_units.tsv | 1 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01092_ont_remote_config_chunk_put_retry.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01092_ont_remote_config_chunk_put_retry.stderr.txt | ssm_file_transfer | yes | rm -rf /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config | retry after row 1091 failed on SSM comment length before writing |
157:| 1094 | 2026-06-06T05:49:47Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Retry ONT config TSV chunk transfer after append syntax fix | python /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/put_remote_config_chunks.py --instance-id i-05374380b57fad901 --region us-west-2 --profile lsmc --remote-dir /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_samples.tsv /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/ont_config/20260606T053415Z_13973636_units.tsv | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01094_ont_remote_config_chunk_put_retry2.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01094_ont_remote_config_chunk_put_retry2.stderr.txt | ssm_file_transfer | yes | rm -rf /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config | retry after row 1092 failed before completing samples.tsv |
158:| 1095 | 2026-06-06T05:52:05Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Retry ONT mounted FASTQ kitchen-sink dry-run using headnode stage-dir | AWS_PROFILE=lsmc dyec workflow launch --profile lsmc --region us-west-2 --cluster hyb-only --stage-dir /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config --session-name hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun --analysis-id hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun --executing-entity ubuntu --git-tag jem-dev --genome hg38_broad --dy-command bin/day_run\ produce_alignstats\ produce_sentdont_snv_vcf\ produce_snv_concordances\ produce_relatedness\ produce_vep\ produce_multiqc_all\ --config\ \'multiqc_qc=\{\"enable_tools\":\[\"vep\"\]\}\'\ -p\ -j\ 5\ -k\ -n --dry-run | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01095_ont_dryrun_launch_remote_stage.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01095_ont_dryrun_launch_remote_stage.stderr.txt | dyec_workflow_launch | no | none | replaces failed row 1077; no FASTQ copy; --git-tag jem-dev |
160:| 1097 | 2026-06-06T05:53:17Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT and ILMN dry-run workflow status | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z_dryrun | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01097_dryrun_status_poll_1.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01097_dryrun_status_poll_1.stderr.txt | monitoring | no | none | monitoring only; no Slurm intervention |
161:| 1098 | 2026-06-06T05:53:35Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Tail ONT and ILMN dry-run tmux logs | AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun --lines 120; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z_dryrun --lines 120 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01098_dryrun_logs_poll_1.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01098_dryrun_logs_poll_1.stderr.txt | monitoring | no | none | monitoring only; no Slurm intervention |
162:| 1099 | 2026-06-06T05:54:27Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT and ILMN dry-run terminal status | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z_dryrun | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01099_dryrun_status_poll_2.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01099_dryrun_status_poll_2.stderr.txt | monitoring | no | none | monitoring only |
163:| 1100 | 2026-06-06T05:55:03Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Tail ONT dry-run tmux log after stage-dir retry | AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun --lines 120 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01100_ont_dryrun_logs_poll_2.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01100_ont_dryrun_logs_poll_2.stderr.txt | monitoring | no | none | monitoring only |
164:| 1101 | 2026-06-06T05:57:41Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT dry-run after DAG expansion | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z_dryrun | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01101_dryrun_status_poll_3.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01101_dryrun_status_poll_3.stderr.txt | monitoring | no | none | status-only monitoring |
165:| 1102 | 2026-06-06T05:58:37Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Launch ONT mounted FASTQ kitchen-sink live workflow via jem-dev day-clone | AWS_PROFILE=lsmc dyec workflow launch --profile lsmc --region us-west-2 --cluster hyb-only --stage-dir /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config --session-name hybonly_ont_kitchensink_mounted_20260606T053415Z --analysis-id hybonly_ont_kitchensink_mounted_20260606T053415Z --executing-entity ubuntu --git-tag jem-dev --genome hg38_broad --dy-command bin/day_run\ produce_alignstats\ produce_sentdont_snv_vcf\ produce_snv_concordances\ produce_relatedness\ produce_vep\ produce_multiqc_all\ --config\ \'multiqc_qc=\{\"enable_tools\":\[\"vep\"\]\}\'\ -p\ -j\ 5\ -k | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01102_ont_live_launch.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01102_ont_live_launch.stderr.txt | dyec_workflow_launch | no | none | dry-run row 1101 rc=0; stage-dir config only; no FASTQ copy |
167:| 1104 | 2026-06-06T05:59:48Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT and ILMN live workflow status immediately after launch | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01104_live_status_poll_1.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01104_live_status_poll_1.stderr.txt | monitoring | no | none | monitoring only; no Slurm intervention |
169:| 1106 | 2026-06-06T06:02:29Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll live workflow status after initial DAG/env prep | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01106_live_status_poll_2.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01106_live_status_poll_2.stderr.txt | monitoring | no | none | status-only monitoring |
171:| 1108 | 2026-06-06T06:03:36Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch filtered log markers for ONT and ILMN live controllers | printf '## ONT\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z --lines 500 \| grep -E 'Executing:\|Building DAG\|Job stats\|rule \|Submitted\|Error\|ERROR\|WorkflowError\|Nothing to be done\|DAG of jobs\|Waiting\|Creating conda\|container\|Singularity\|Apptainer' \| tail -n 120; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 500 \| grep -E 'Executing:\|Building DAG\|Job stats\|rule \|Submitted\|Error\|ERROR\|WorkflowError\|Nothing to be done\|DAG of jobs\|Waiting\|Creating conda\|container\|Singularity\|Apptainer' \| tail -n 120 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01108_live_log_markers_1.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01108_live_log_markers_1.stderr.txt | monitoring | no | none | monitoring only; grep filters avoid huge DAG context output |
172:| 1109 | 2026-06-06T06:07:20Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll live workflow status during pre-submit setup | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01109_live_status_poll_3.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01109_live_status_poll_3.stderr.txt | monitoring | no | none | status-only monitoring |
174:| 1111 | 2026-06-06T06:07:47Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch filtered live log markers during pre-submit setup | printf '## ONT\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z --lines 500 \| grep -E 'Executing:\|Building DAG\|Job stats\|rule \|Submitted\|Error\|ERROR\|WorkflowError\|Nothing to be done\|DAG of jobs\|Waiting\|Creating conda\|environment\|container\|Singularity\|Apptainer' \| tail -n 80; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 500 \| grep -E 'Executing:\|Building DAG\|Job stats\|rule \|Submitted\|Error\|ERROR\|WorkflowError\|Nothing to be done\|DAG of jobs\|Waiting\|Creating conda\|environment\|container\|Singularity\|Apptainer' \| tail -n 120 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01111_live_log_markers_2.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01111_live_log_markers_2.stderr.txt | monitoring | no | none | monitoring only |
175:| 1112 | 2026-06-06T06:11:31Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll live workflow status after conda cache rebuild marker | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01112_live_status_poll_4.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01112_live_status_poll_4.stderr.txt | monitoring | no | none | status-only monitoring |
177:| 1114 | 2026-06-06T06:11:58Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch filtered live log markers after conda cache rebuild marker | printf '## ONT\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z --lines 500 \| grep -E 'Building DAG\|Job stats\|Submitted\|Error\|ERROR\|WorkflowError\|Nothing to be done\|Creating conda\|Environment for\|Removing incomplete\|Downloading\|container\|Singularity\|Apptainer\|rule ' \| tail -n 100; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 800 \| grep -E 'Building DAG\|Job stats\|Submitted\|Error\|ERROR\|WorkflowError\|Nothing to be done\|Creating conda\|Environment for\|Removing incomplete\|Downloading\|container\|Singularity\|Apptainer\|rule ' \| tail -n 140 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01114_live_log_markers_3.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01114_live_log_markers_3.stderr.txt | monitoring | no | none | monitoring only |
178:| 1115 | 2026-06-06T06:12:53Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Tail ONT live tmux log to identify terminal rc=1 cause | AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z --lines 260 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01115_ont_live_failure_tail.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01115_ont_live_failure_tail.stderr.txt | debugging | no | none | ONT terminal rc=1 at 06:08:23Z before Slurm jobs |
180:| 1117 | 2026-06-06T06:13:38Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Extract ONT failure text from tmux log with byte-bound and error filters | AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z --lines 2000 \| tail -c 30000 \| perl -ne 'print if /Error\|ERROR\|Exception\|Traceback\|Missing\|Ambiguous\|Wildcard\|RuleException\|WorkflowError\|InputFunction\|KeyError\|failed\|Failed\|exit code\|Exiting/i' | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01117_ont_live_failure_error_extract.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01117_ont_live_failure_error_extract.stderr.txt | debugging | no | none | ONT terminal rc=1; previous tail dominated by huge wildcard line |
182:| 1119 | 2026-06-06T06:14:42Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Summarize ONT live tmux log with bounded lines and failure-pattern context | python /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/extract_remote_tmux_log.py --instance-id i-05374380b57fad901 --region us-west-2 --profile lsmc --session hybonly_ont_kitchensink_mounted_20260606T053415Z --context 8 --width 1200 | 1 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01119_ont_live_remote_log_summary.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01119_ont_live_remote_log_summary.stderr.txt | debugging | no | none | ONT terminal rc=1; direct dyec log output dominated by huge wildcard line |
184:| 1121 | 2026-06-06T06:15:28Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Retry bounded ONT tmux log summary after helper placeholder fix | python /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/extract_remote_tmux_log.py --instance-id i-05374380b57fad901 --region us-west-2 --profile lsmc --session hybonly_ont_kitchensink_mounted_20260606T053415Z --context 8 --width 1200 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01121_ont_live_remote_log_summary_retry.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01121_ont_live_remote_log_summary_retry.stderr.txt | debugging | no | none | retry after row 1119 local helper bug |
187:| 1124 | 2026-06-06T06:20:21Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Retry ONT live after conda cache failure using jem-dev day-clone | AWS_PROFILE=lsmc dyec workflow launch --profile lsmc --region us-west-2 --cluster hyb-only --stage-dir /home/ubuntu/daylily-staged-configs/hybonly_ont_kitchensink_mounted_20260606T053415Z_config --session-name hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --analysis-id hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --executing-entity ubuntu --git-tag jem-dev --genome hg38_broad --dy-command bin/day_run\ produce_alignstats\ produce_sentdont_snv_vcf\ produce_snv_concordances\ produce_relatedness\ produce_vep\ produce_multiqc_all\ --config\ \'multiqc_qc=\{\"enable_tools\":\[\"vep\"\]\}\'\ -p\ -j\ 5\ -k\ --rerun-triggers\ mtime | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01124_ont_live_retry1_launch.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01124_ont_live_retry1_launch.stderr.txt | dyec_workflow_launch | no | none | previous ONT live failed before Slurm on mosdepth conda env create; ILMN now submitting/running Slurm jobs; retry adds --rerun-triggers mtime |
188:| 1125 | 2026-06-06T06:21:00Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN live status plus Slurm queue | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01125_live_status_jobs_poll_8.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01125_live_status_jobs_poll_8.stderr.txt | monitoring | no | none | monitoring only |
189:| 1126 | 2026-06-06T06:24:42Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN after ONT retry launch | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01126_live_status_jobs_poll_9.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01126_live_status_jobs_poll_9.stderr.txt | monitoring | no | none | monitoring only |
190:| 1127 | 2026-06-06T06:25:09Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch ONT retry1 and ILMN live markers after ONT retry launch | printf '## ONT retry1\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --lines 700 \| grep -E 'Building DAG\|Job stats\|Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|Creating conda\|Environment for\|Removing incomplete\|CreateConda' \| tail -n 120; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|Complete\|failed\|Failed\|Creating conda\|Environment for\|Removing incomplete\|CreateConda' \| tail -n 140 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01127_live_log_markers_6.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01127_live_log_markers_6.stderr.txt | monitoring | no | none | monitoring only |
198:| 1136 | 2026-06-06T06:35:34Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN after cache-prefix probe | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01136_live_status_jobs_poll_10.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01136_live_status_jobs_poll_10.stderr.txt | monitoring | no | none | monitoring only |
199:| 1137 | 2026-06-06T06:42:37Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN 5m after cache plan lock | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01137_live_status_jobs_poll_11.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01137_live_status_jobs_poll_11.stderr.txt | monitoring | no | none | monitoring only |
200:| 1138 | 2026-06-06T06:43:04Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch ONT retry1 and ILMN live markers 5m after cache plan lock | printf '## ONT retry1\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --lines 600 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats' \| tail -n 120; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 600 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats' \| tail -n 120 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01138_live_log_markers_7.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01138_live_log_markers_7.stderr.txt | monitoring | no | none | monitoring only |
201:| 1139 | 2026-06-06T06:49:07Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN status with cache sync gate enforced | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01139_live_status_jobs_poll_12.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01139_live_status_jobs_poll_12.stderr.txt | monitoring | no | none | monitoring only; cache sync blocked until both exit_code=0 |
202:| 1140 | 2026-06-06T06:49:34Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch ONT retry1 and ILMN live markers with cache sync gate enforced | printf '## ONT retry1\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete' \| tail -n 120; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete' \| tail -n 120 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01140_live_log_markers_8.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01140_live_log_markers_8.stderr.txt | monitoring | no | none | monitoring only; cache sync blocked until both exit_code=0 |
203:| 1141 | 2026-06-06T06:55:34Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN status; cache sync remains gated | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01141_live_status_jobs_poll_13.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01141_live_status_jobs_poll_13.stderr.txt | monitoring | no | none | monitoring only; no cache sync until both exit_code=0 |
204:| 1142 | 2026-06-06T06:56:00Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch ONT retry1 and ILMN live markers; cache sync remains gated | printf '## ONT retry1\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete\|Nothing to be done' \| tail -n 100; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete\|Nothing to be done' \| tail -n 100 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01142_live_log_markers_9.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01142_live_log_markers_9.stderr.txt | monitoring | no | none | monitoring only; no cache sync until both exit_code=0 |
205:| 1143 | 2026-06-06T07:01:59Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN status; cache sync gated by rc=0 | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01143_live_status_jobs_poll_14.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01143_live_status_jobs_poll_14.stderr.txt | monitoring | no | none | monitoring only; no cache sync until both exit_code=0 |
206:| 1144 | 2026-06-06T07:02:27Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch ONT retry1 and ILMN live markers; cache sync gated by rc=0 | printf '## ONT retry1\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete\|Nothing to be done' \| tail -n 100; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete\|Nothing to be done' \| tail -n 100 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01144_live_log_markers_10.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01144_live_log_markers_10.stderr.txt | monitoring | no | none | monitoring only; no cache sync until both exit_code=0 |
207:| 1145 | 2026-06-06T07:10:22Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 and ILMN status; no cache sync before dual rc=0 | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01145_live_status_jobs_poll_15.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01145_live_status_jobs_poll_15.stderr.txt | monitoring | no | none | monitoring only; no cache sync until both exit_code=0 |
208:| 1146 | 2026-06-06T07:10:49Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Fetch ONT retry1 and ILMN markers; no cache sync before dual rc=0 | printf '## ONT retry1\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete\|Nothing to be done' \| tail -n 100; printf '## ILMN\n'; AWS_PROFILE=lsmc dyec workflow logs --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z --lines 700 \| grep -E 'Submitted job\|rule \|Error\|ERROR\|WorkflowError\|Finished job\|failed\|Failed\|CreateConda\|DAG of jobs\|Job stats\|Complete\|Nothing to be done' \| tail -n 100 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01146_live_log_markers_11.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01146_live_log_markers_11.stderr.txt | monitoring | no | none | monitoring only; no cache sync until both exit_code=0 |
209:| 1147 | 2026-06-06T07:12:55Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 + ILMN live snakecontrollers and Slurm queue | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01147_live_status_jobs_poll_12.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01147_live_status_jobs_poll_12.stderr.txt | monitoring | no | none | monitoring only; cache sync blocked until both snakecontrollers rc=0 |
210:| 1149 | 2026-06-06T07:19:06Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | User-requested status and df -h /fsx | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode run-shell --profile lsmc --region us-west-2 --cluster hyb-only --command "df -h /fsx" | 127 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01149_live_status_fsx_poll_user_01.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01149_live_status_fsx_poll_user_01.stderr.txt | monitoring | no | none | status only; no cache sync until both snakecontrollers rc=0 |
212:| 1151 | 2026-06-06T07:19:26Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | User-requested status and df -h /fsx via DAY-EC | AWS_PROFILE=lsmc conda run -n DAY-EC dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc conda run -n DAY-EC dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc conda run -n DAY-EC dyec headnode run-shell --profile lsmc --region us-west-2 --cluster hyb-only --command "df -h /fsx" | 2 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01151_live_status_fsx_poll_user_02.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01151_live_status_fsx_poll_user_02.stderr.txt | monitoring | no | none | status only; no cache sync until both snakecontrollers rc=0 |
216:| 1154 | 2026-06-06T07:21:02Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 + ILMN live snakecontrollers and Slurm queue | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01154_live_status_jobs_poll_13.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01154_live_status_jobs_poll_13.stderr.txt | monitoring | no | none | monitoring only; cache sync blocked until both snakecontrollers rc=0 |
217:| 1156 | 2026-06-06T07:27:19Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Inspect latest Snakemake progress and DAG artifacts for ONT/ILMN controllers | cmd_id=$(AWS_PROFILE=lsmc aws ssm send-command --region us-west-2 --instance-ids i-05374380b57fad901 --document-name AWS-RunShellScript --parameters commands="runuser -u ubuntu -- bash -lc 'set -euo pipefail\nfor id in hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1 hybonly_ilmn_kitchensink_mounted_20260606T053415Z; do\n  repo=/fsx/analysis_results/ubuntu/$id/daylily-omics-analysis\n  echo \"## $id\"\n  echo \"repo=$repo\"\n  if [ ! -d \"$repo\" ]; then echo \"missing_repo=1\"; continue; fi\n  latest=$(ls -1t \"$repo\"/.snakemake/log/*.log 2>/dev/null \| head -1 \|\| true)\n  echo \"latest_log=$latest\"\n  if [ -n \"$latest\" ]; then\n    echo \"progress_lines_last5\"\n    grep -E \"[0-9]+ of [0-9]+ steps\|Finished job\|Submitted job\|Error in rule\|Failed jobs\|Exiting because\|Complete log\" \"$latest\" \| tail -n 80 \|\| true\n    echo \"summary_counts\"\n    python3 - \"$latest\" <<'PY'\nimport re, sys\np=sys.argv[1]\ntext=open(p, errors="replace").read().splitlines()\nprogress=[]\nfinished=0\nsubmitted=0\nerrors=[]\nfailed_jobs=[]\nfor line in text:\n    if "Finished job" in line:\n        finished += 1\n    if "Submitted job" in line:\n        submitted += 1\n    if "Error in rule" in line or "Failed jobs" in line or "Exiting because" in line:\n        errors.append(line)\n    m=re.search(r"(\\d+) of (\\d+) steps \\(([^)]+)\\) done", line)\n    if m:\n        progress.append((int(m.group(1)), int(m.group(2)), m.group(3), line))\nif progress:\n    done,total,pct,line=progress[-1]\nelse:\n    done=total=pct=None\nprint(f"finished_lines={finished}")\nprint(f"submitted_lines={submitted}")\nprint(f"latest_done={done}")\nprint(f"latest_total={total}")\nprint(f"latest_pct={pct}")\nif total is not None and done is not None:\n    print(f"remaining_by_latest_progress={total-done}")\nprint(f"error_marker_count={len(errors)}")\nfor e in errors[-10:]: print("error_marker="+e[:500])\nPY\n  fi\n  echo \"dag_files\"\n  find \"$repo\" -maxdepth 5 -type f \( -name \"*.dag\" -o -name \"*dag*.svg\" -o -name \"*dag*.png\" -o -name \"*dag*.pdf\" -o -name \"dag.svg\" -o -name \"dag.png\" -o -name \"dag.pdf\" \) -printf \"%p\\t%s\\t%TY-%Tm-%TdT%TH:%TM:%TS%TZ\\n\" 2>/dev/null \| sort \|\| true\n  echo\n done'" --query Command.CommandId --output text); echo COMMAND_ID=$cmd_id; AWS_PROFILE=lsmc aws ssm wait command-executed --region us-west-2 --command-id "$cmd_id" --instance-id i-05374380b57fad901; AWS_PROFILE=lsmc aws ssm get-command-invocation --region us-west-2 --command-id "$cmd_id" --instance-id i-05374380b57fad901 --query "{Status:Status,Stdout:StandardOutputContent,Stderr:StandardErrorContent}" --output json | 1 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01156_snakemake_progress_dag_probe_01.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01156_snakemake_progress_dag_probe_01.stderr.txt | monitoring | no | none | inspection only; no cache sync until both snakecontrollers rc=0 |
221:| 1159 | 2026-06-06T07:29:38Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Poll ONT retry1 + ILMN live snakecontrollers and Slurm queue | AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1; AWS_PROFILE=lsmc dyec --json workflow status --profile lsmc --region us-west-2 --cluster hyb-only --session hybonly_ilmn_kitchensink_mounted_20260606T053415Z; AWS_PROFILE=lsmc dyec headnode jobs --profile lsmc --region us-west-2 --cluster hyb-only | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01159_live_status_jobs_poll_14.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01159_live_status_jobs_poll_14.stderr.txt | monitoring | no | none | monitoring only; cache sync blocked until both snakecontrollers rc=0 |
227:| 1166 | 2026-06-06T07:42:45Z | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster | local | Find prior ONT workflow launch command for dry-run restart shape | grep -n 'hybonly_ont_kitchensink_mounted_20260606T053415Z_retry1\\|hybonly_ont_kitchensink_mounted_20260606T053415Z_dryrun\\|workflow launch' docs/ONT_ILMN_SOLO_CMD_LOG.md \| tail -n 20 | 0 | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01166_prior_ont_launch_command_probe_01.stdout.txt | /Users/jmajor/projects/lsmc/daylily-ephemeral-cluster/docs/plans/20260605T000000Z_hyb_only_mounted_fastq_kitchensink_logs/01166_prior_ont_launch_command_probe_01.stderr.txt | monitoring | no | none | inspection only; cache sync blocked |
