DAY-EC activated.
DAY-EC activated.
[INFO] pycoQC readonly-sort repair already present: /fsx/resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_/lib/python3.13/site-packages/pycoQC/pycoQC_plot.py
Remote call detected. Activating conda hack
Your config files in /fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis/config/day_profiles/slurm are newer than the templates. clear 2 go.
Executing: snakemake --profile=/fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis/config/day_profiles/slurm produce_ont_run_qc -p -j 5 -k --config run_context_file=config/runs.tsv samples_table=.test_data/data/samples.tsv units_table=.test_data/data/units.tsv
Config file config/global.yaml is extended by additional config specified via the command line.
loading global: /fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis/config/global_AWSPC.yaml
Config file config/global_AWSPC.yaml is extended by additional config specified via the command line.
loading profile rule_config: /fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis/config/day_profiles/slurm/rule_config.yaml
Config file config/day_profiles/slurm/rule_config.yaml is extended by additional config specified via the command line.
INFO::: The genome build hg38_broad is supported.  The genome build prefix is 'chr''.
...WARNING: No aligners set in the config.
aligners (final): []
...INFO: No dedupers set in config. Defaulting to na (no dedup).
deduper (final): [na]
...WARNING: No snv_callers set in the config.
SNV Callers (final): []
Somatic SNV Callers:[senttn]
... WARNING: No sv_callers set in the config.
SV Callers (final): []
A    N   A   L  Y S I S    SAMPLE TABLE DETECTED ::: /fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis/.test_data/data/samples.tsv
A    N   A   L  Y S I S    UNIT TABLE DETECTED ::: /fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis/.test_data/data/units.tsv
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_1x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_3x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_5x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_7x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_10x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_15x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_20x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/HG003_30x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_40x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_1x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_3x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_5x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_7x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_10x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_15x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_20x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/HG003_30x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_40x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_1x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_3x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_5x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_7x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_10x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_15x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_20x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/HG003_30x_R1.fastq.gz'
[Errno 2] No such file or directory: '/fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/NovaSeqX_WHGS_TruSeqPF_HG002-007/downsampled/HG003_40x_R1.fastq.gz'
Building DAG of jobs...
Using shell: /home/ubuntu/miniconda3/envs/DAY-EC/bin/bash
Provided cluster nodes: 5
Job stats:
job                             count    min threads    max threads
----------------------------  -------  -------------  -------------
ont_run_qc_collect_summaries        1              1              1
ont_run_qc_multiqc                  1              1              1
ont_run_qc_nanoplot                 1              8              8
ont_run_qc_pycoqc                   1              1              1
ont_run_qc_report                   1              1              1
produce_ont_run_qc                  1              1              1
total                               6              1              8

Select jobs to execute...

[Sat May 30 05:45:40 2026]
localrule ont_run_qc_collect_summaries:
    output: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    log: results/runs/20260513_ONT_HG003/run_qc/ont/logs/collect_summaries.log
    jobid: 3
    benchmark: results/runs/20260513_ONT_HG003/run_qc/ont/benchmarks/collect_summaries.bench.tsv
    reason: Missing output files: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=


        set -euo pipefail
        mkdir -p $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt) $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/logs/collect_summaries.log)
        : > results/runs/20260513_ONT_HG003/run_qc/ont/logs/collect_summaries.log
        if [ -z /fsx/run_dir_mounts/20260513_ONT_HG003/ ]; then
            echo "config/runs.tsv with PLATFORM=ONT and RUN_DIR is required for mounted ONT run QC" >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/collect_summaries.log
            exit 2
        fi
        test -d /fsx/run_dir_mounts/20260513_ONT_HG003/
        find /fsx/run_dir_mounts/20260513_ONT_HG003/ -type f \( -name 'sequencing_summary*.txt' -o -name 'sequencing_summary*.txt.gz' \) \
          | sort > results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
        if [ ! -s results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt ]; then
            echo "No sequencing_summary*.txt files found under /fsx/run_dir_mounts/20260513_ONT_HG003/" >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/collect_summaries.log
            exit 2
        fi
        
Activating conda environment: ../../../../resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_
[Sat May 30 05:45:40 2026]
Finished job 3.
1 of 6 steps (17%) done
Select jobs to execute...

[Sat May 30 05:45:40 2026]
localrule ont_run_qc_pycoqc:
    input: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    output: results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html, results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
    log: results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log
    jobid: 2
    benchmark: results/runs/20260513_ONT_HG003/run_qc/ont/benchmarks/pycoqc.bench.tsv
    reason: Missing output files: results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json, results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html; Input files updated by another job: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=


        set -euo pipefail
        mkdir -p results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log)
        : > results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log
        command -v pycoQC >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        mapfile -t summary_files < results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
        pycoQC \
          -f "${summary_files[@]}" \
          -o results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html \
          -j results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json \
          --report_title 20260513_ONT_HG003 \
          >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
        
Activating conda environment: ../../../../resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_

[Sat May 30 05:45:40 2026]
localrule ont_run_qc_nanoplot:
    input: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    output: results/runs/20260513_ONT_HG003/run_qc/ont/nanoplot/nanoplot.done
    log: results/runs/20260513_ONT_HG003/run_qc/ont/logs/nanoplot.log
    jobid: 5
    benchmark: results/runs/20260513_ONT_HG003/run_qc/ont/benchmarks/nanoplot.bench.tsv
    reason: Missing output files: results/runs/20260513_ONT_HG003/run_qc/ont/nanoplot/nanoplot.done; Input files updated by another job: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    threads: 8
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=


        set -euo pipefail
        mkdir -p results/runs/20260513_ONT_HG003/run_qc/ont/nanoplot $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/logs/nanoplot.log)
        : > results/runs/20260513_ONT_HG003/run_qc/ont/logs/nanoplot.log
        command -v NanoPlot >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/nanoplot.log 2>&1
        mapfile -t summary_files < results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
        NanoPlot \
          --summary "${summary_files[@]}" \
          --loglength \
          --tsv_stats \
          --info_in_report \
          -t 8 \
          -o results/runs/20260513_ONT_HG003/run_qc/ont/nanoplot \
          -p 20260513_ONT_HG003. \
          >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/nanoplot.log 2>&1
        
Activating conda environment: ../../../../resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_
[Sat May 30 05:46:18 2026]
Error in rule ont_run_qc_pycoqc:
    jobid: 2
    input: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    output: results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html, results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
    log: results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log (check log file(s) for error details)
    conda-env: /fsx/resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_
    shell:
        
        set -euo pipefail
        mkdir -p results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log)
        : > results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log
        command -v pycoQC >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        mapfile -t summary_files < results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
        pycoQC \
          -f "${summary_files[@]}" \
          -o results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html \
          -j results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json \
          --report_title 20260513_ONT_HG003 \
          >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 2.
Select jobs to execute...

[Sat May 30 05:46:18 2026]
localrule ont_run_qc_pycoqc:
    input: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    output: results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html, results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
    log: results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log
    jobid: 2
    benchmark: results/runs/20260513_ONT_HG003/run_qc/ont/benchmarks/pycoqc.bench.tsv
    reason: Missing output files: results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json, results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html; Input files updated by another job: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=


        set -euo pipefail
        mkdir -p results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log)
        : > results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log
        command -v pycoQC >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        mapfile -t summary_files < results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
        pycoQC \
          -f "${summary_files[@]}" \
          -o results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html \
          -j results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json \
          --report_title 20260513_ONT_HG003 \
          >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
        
Activating conda environment: ../../../../resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_
Touching output file results/runs/20260513_ONT_HG003/run_qc/ont/nanoplot/nanoplot.done.
[Sat May 30 05:46:25 2026]
Finished job 5.
2 of 6 steps (33%) done
[Sat May 30 05:46:53 2026]
Error in rule ont_run_qc_pycoqc:
    jobid: 2
    input: results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
    output: results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html, results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
    log: results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log (check log file(s) for error details)
    conda-env: /fsx/resources/environments/conda/ubuntu/ip-10-0-0-88/a99ea18d415639f67a6e85afdbe2fe1b_
    shell:
        
        set -euo pipefail
        mkdir -p results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc $(dirname results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log)
        : > results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log
        command -v pycoQC >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        mapfile -t summary_files < results/runs/20260513_ONT_HG003/run_qc/ont/tables/sequencing_summary_files.txt
        pycoQC \
          -f "${summary_files[@]}" \
          -o results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html \
          -j results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json \
          --report_title 20260513_ONT_HG003 \
          >> results/runs/20260513_ONT_HG003/run_qc/ont/logs/pycoqc.log 2>&1
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.html
        test -s results/runs/20260513_ONT_HG003/run_qc/ont/pycoqc/pycoQC.json
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message
 
        Womp Womp.  something went awry---- 
RETURN CODE: 1
[INFO] Workflow exited with status 1
DAY-EC activated.
(DAY-EC) ubuntu@ip-10-0-0-88:/fsx/analysis_results/ubuntu/ccv20260529r21_ont_run_qc/daylily-omics-analysis$
