DAY-EC activated.
DAY-EC activated.
=== now ===
2026-05-30T23:48:42Z
=== status.json ===
{
  "command": "bin/day_run produce_illumina_run_qc_and_bclconvert -p -j 20 -k --config run_context_file=config/runs.tsv bootstrap_bclconvert=true",
  "completed_at": null,
  "exit_code": null,
  "repo_path": "/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis",
  "session_name": "ccv20260530r57_illumina_run_qc_bclconvert",
  "started_at": "2026-05-30T21:52:38Z"
}
=== slurm ===
=== fsx ===
Filesystem              Type    Size  Used Avail Use% Mounted on
10.0.1.96@tcp:/d4tzvb4v lustre  8.8T  8.1T  707G  93% /fsx
tmpfs                   tmpfs   199G     0  199G   0% /dev/shm
/dev/root               ext4    408G   42G  367G  11% /
=== controller processes ===
2348153   81703 Ss+   0.0  0.0    01:56:05 bash -lc source /home/ubuntu/daylily-runs/ccv20260530r57_illumina_run_qc_bclconvert/launch.sh >>/home/ubuntu/daylily-runs/ccv20260530r57_illumina_run_qc_bclconvert/tmux.log 2>&1
2422216 2348153 S+    0.0  0.0       15:01 /home/ubuntu/miniconda3/envs/DAY-EC/bin/python /home/ubuntu/miniconda3/envs/DAY-EC/bin/dyec export --region us-west-2 --cluster dyec-test --source-path /fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert --destination-s3-uri s3://lsmc-ssf-sequencing-data/derived/validation/dyec-test/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/ --output-dir /home/ubuntu/daylily-runs/ccv20260530r57_illumina_run_qc_bclconvert/export
=== run-dir projection link ===
link_absent /fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/config/run_dir_links/20260514_LH01106_0009_B23TVLGLT4
=== latest snakemake log tail ===
latest_log=/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/.snakemake/log/2026-05-30T233340.873699.snakemake.log
WorkflowError in file /fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/workflow/rules/common.smk, line 426:
The units table was not found at /fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/config/units.tsv. Create config/units.tsv or set --config units_table=/path/to/units.tsv.
  File "/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/workflow/Snakefile", line 66, in <module>
  File "/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/workflow/rules/global_common.smk", line 151, in <module>
  File "/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/workflow/rules/common.smk", line 1089, in <module>
  File "/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/workflow/rules/common.smk", line 426, in _load_units_table
=== tmux log tail ===
        : > results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/logs/bclconvert_metrics_multiqc_exports.log
        python workflow/scripts/bclconvert_metrics_to_multiqc.py \
          --demux-tsv results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/demultiplex_stats.tsv \
          --unknown-tsv results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/unknown_barcodes.tsv \
          --hopping-tsv results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/index_hopping.tsv \
          --fastq-manifest-tsv results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/fastq_manifest.tsv \
          --rollup-json results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/rollup.json \
          --demux-out results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/bclconvert_demux_mqc.tsv \
          --unknown-out results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/bclconvert_unknown_barcodes_mqc.tsv \
          --hopping-out results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/bclconvert_index_hopping_mqc.tsv \
          --fastq-manifest-out results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/bclconvert_fastq_manifest_mqc.tsv \
          --lane-summary-out results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/bclconvert_lane_summary_mqc.tsv \
          >> results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs/logs/bclconvert_metrics_multiqc_exports.log 2>&1

Submitted job 19 with external jobid '4661'.
[Sat May 30 23:30:09 2026]
Finished job 19.
18 of 21 steps (86%) done
Select jobs to execute...

[Sat May 30 23:30:09 2026]
rule multiqc_bclconvert:
    input: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/bclconvert.done, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/fastqs/Reports/Demultiplex_Stats.csv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/fastqs/Reports/fastq_list.csv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert_metrics_mqc.done
    output: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_report.html
    log: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/multiqc_bclconvert.log
    jobid: 20
    benchmark: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/benchmarks/multiqc_bclconvert.bench.tsv
    reason: Missing output files: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_report.html; Input files updated by another job: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/bclconvert.done, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/fastqs/Reports/Demultiplex_Stats.csv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/fastqs/Reports/fastq_list.csv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert_metrics_mqc.done
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=


        set -euo pipefail
        mkdir -p results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert
        : > results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/multiqc_bclconvert.log
        multiqc -f \
          -m bclconvert \
          -m custom_content \
          --config config/external_tools/multiqc_config.yaml \
          --template default \
          --filename multiqc_report.html \
          --outdir results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert \
          results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/fastqs/Reports \
          results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_inputs \
          >> results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/multiqc_bclconvert.log 2>&1
        test -s results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_report.html

Submitted job 20 with external jobid '4662'.
[Sat May 30 23:31:29 2026]
Finished job 20.
19 of 21 steps (90%) done
Select jobs to execute...

[Sat May 30 23:31:29 2026]
localrule produce_bclconvert_fastqs_and_metrics:
    input: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/validated.ok, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/bclconvert.done, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/tables/generated.units.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/demultiplex_stats.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/unknown_barcodes.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/index_hopping.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/fastq_manifest.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/rollup.json, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert_metrics_mqc.done, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_report.html
    output: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert.bootstrap.complete
    jobid: 6
    reason: Missing output files: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert.bootstrap.complete; Input files updated by another job: results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/index_hopping.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/demultiplex_stats.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/multiqc_report.html, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/rollup.json, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/unknown_barcodes.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/metrics/fastq_manifest.tsv, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert_metrics_mqc.done, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/bclconvert.done, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/logs/validated.ok, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/tables/generated.units.tsv
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=

touch results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert.bootstrap.complete
Touching output file results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert.bootstrap.complete.
[Sat May 30 23:31:29 2026]
Finished job 6.
20 of 21 steps (95%) done
Select jobs to execute...

[Sat May 30 23:31:29 2026]
localrule produce_illumina_run_qc_and_bclconvert:
    input: results/runs/20260514_LH01106_0009_B23TVLGLT4/run_qc/illumina/summary.html, results/runs/20260514_LH01106_0009_B23TVLGLT4/run_qc/illumina/multiqc_report.html, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert.bootstrap.complete
    jobid: 0
    reason: Input files updated by another job: results/runs/20260514_LH01106_0009_B23TVLGLT4/run_qc/illumina/summary.html, results/runs/20260514_LH01106_0009_B23TVLGLT4/run_qc/illumina/multiqc_report.html, results/runs/20260514_LH01106_0009_B23TVLGLT4/bclconvert/bclconvert.bootstrap.complete
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=

[Sat May 30 23:31:29 2026]
Finished job 0.
21 of 21 steps (100%) done
Complete log: .snakemake/log/2026-05-30T215301.097987.snakemake.log


	WARNING!!!! ALL Directories Are Remaining Unlocked. To have directories lock automatically upon successful completion, set --lock-analysis-dirs when calling day-run.


[1m[48;5;205m[30m
]]]]--------WORKFLOW SUCCESS====================]]]]]--[0m
[5m[48;5;118m[30mRETURN CODE: 0[0m
[INFO] Removing run-directory projection before export: /fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/daylily-omics-analysis/config/run_dir_links/20260514_LH01106_0009_B23TVLGLT4 -> /fsx/run_dir_mounts/20260514_LH01106_0009_B23TVLGLT4
⚠ Active conda environment is 'DAYOA', not DAY-EC. Continuing anyway; the
supported path is `source ./activate`.

── EXPORT ──
  › Preparing direct FSx export DRA
2026-05-30 23:33:41,064 INFO Found credentials from IAM Role: dyec-test-RoleHeadNode-eJMKI1vZVQBE
  · Attached export DRA dra-00a88684e523c8868 at
/fsx/analysis_results/ubuntu/ccv20260530r57_illumina_run_qc_bclconvert/
2026-05-30 23:36:31,116 INFO Task task-0d328c73f7d8ec534 status: PENDING
2026-05-30 23:37:01,474 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:37:31,687 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:38:01,907 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:38:32,282 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:39:02,492 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:39:32,728 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:40:02,971 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:40:33,322 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:41:03,550 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:41:33,892 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:42:04,152 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:42:34,454 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:43:04,640 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:43:34,978 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:44:05,315 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:44:35,506 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:45:05,692 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:45:35,966 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:46:06,169 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:46:36,362 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:47:06,592 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:47:36,768 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:48:06,964 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
2026-05-30 23:48:37,291 INFO Task task-0d328c73f7d8ec534 status: EXECUTING
=== export files ===
