DAY-EC activated.
DAY-EC activated.
    benchmark: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/benchmarks/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pre_prep_ultima_cram.bench.tsv
    reason: Missing output files: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram, results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.crai
    wildcards: sample=R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA, sample_lane=R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA
    threads: 48
    resources: mem_mb=20000, mem_mib=19074, disk_mb=9578, disk_mib=9135, tmpdir=<TBD>, time=100, partition=i192,i128, threads=48, vcpu=48


        (mkdir -p $(dirname results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log) || echo results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log dir exists) >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;
        export TMPDIR=$(dirname results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log)/tmpdir;
        mkdir -p $TMPDIR || echo results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log dir exists >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;

        if [[ 'na' != 'na' ]]; then
            echo 'downsampling to na' >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;
            samtools view -@ 48 -T /fsx/references/genomic_data/organism_references/H_sapiens/hg38_broad/Homo_sapiens_assembly38.fasta -C -s 33.na /fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/agbt_2026/ug/HG003_3x.cleaned.cram -o results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;

            sleep 5;
            samtools index results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;

        else
            echo 'not downsampling: na' >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;
             ln -s   /fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/agbt_2026/ug/HG003_3x.cleaned.cram results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;
            sleep 5;
             ln -s   /fsx/control_data/genomic_data/organism_reads/H_sapiens/giab/agbt_2026/ug/HG003_3x.cleaned.cram.crai results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.crai >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.log 2>&1;
        fi
        

[Tue Jun  9 10:51:32 2026]
localrule workflow_staging:
    input: logs/supporting_data_staging.done
    output: logs/workflow_staging.done
    log: logs/workflow_staging.log
    jobid: 3
    benchmark: results/day/hg38_broad/benchmarks/workflow_staging.bench.tsv
    reason: Missing output files: logs/workflow_staging.done; Input files updated by another job: logs/supporting_data_staging.done
    resources: mem_mb=<TBD>, disk_mb=<TBD>, tmpdir=/dev/shm, time=100

(mkdir -p results/ && mkdir -p logs/ && mkdir -p tmp/ && mkdir -p results/day/hg38_broad/reports/ && mkdir -p results/day/hg38_broad/other_reports ) > logs/workflow_staging.log 2>&1;mkdir -p results/day/hg38_broad/other_reports;touch logs/workflow_staging.done

[Tue Jun  9 10:51:32 2026]
Job 2: You can override the defalt mesage output to the terminal per rule execution if you wish, including wildcards and config variables.  This is for sample sample.
Reason: Missing output files: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.ready; Input files updated by another job: logs/workflow_staging.done


        ( mkdir -p $(dirname results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.ready ) || echo "ERROR:: mkdir $(dirname results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.ready )  FAILED";
        touch results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.ready; ) > results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/logs/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.log 2>&1;
        

[Tue Jun  9 10:51:32 2026]
rule sentieon_pangenome_ug:
    input: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.ready, results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram, results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.crai
    output: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz, results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz.tbi
    log: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log
    jobid: 1
    benchmark: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/benchmarks/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.bench.tsv
    reason: Missing output files: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz.tbi; Input files updated by another job: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram, results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram.crai, results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.dirsetup.ready
    wildcards: sample=R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA
    priority: 5
    threads: 128
    resources: mem_mb=200000, mem_mib=190735, disk_mb=<TBD>, tmpdir=<TBD>, time=100, attempt_n=1, partition=i384nvme,i192nvme,i192hugenvme,i128, threads=128, vcpu=128, constraint=



        if [ -z "$SENTIEON_LICENSE" ]; then
            echo "SENTIEON_LICENSE not set." >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            exit 3;
        fi

        if [ ! -f "$SENTIEON_LICENSE" ]; then
            echo "SENTIEON_LICENSE file does not exist: '$SENTIEON_LICENSE'" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            exit 4;
        fi

        TOKEN=$(curl -s -X PUT 'http://169.254.169.254/latest/api/token' -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600');
        itype=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-type);
        echo "INSTANCE TYPE: $itype" > results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log;
        start_time=$(date +%s);

        # Prepend patched KMC (Sentieon fork with stdin support) to PATH
        export PATH="$DAY_ROOT/resources/kmc/bin:$PATH";
        if ! command -v kmc &>/dev/null; then
            echo "ERROR: patched kmc not found at $DAY_ROOT/resources/kmc/bin/kmc" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            exit 6;
        fi
        echo "Using patched KMC: $(which kmc)" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;

        ulimit -n 65536 || echo "ulimit mod failed" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;

        timestamp=$(date +%Y%m%d%H%M%S);
        export TMPDIR="/fsx/scratch/pangenome_ug_tmp_${timestamp}_$$";
        export SENTIEON_TMPDIR="$TMPDIR";
        mkdir -p "$TMPDIR";
        if [ ! -d "$TMPDIR" ]; then
            echo "ERROR: Failed to create TMPDIR: $TMPDIR" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            exit 5;
        fi
        echo "TMPDIR created: $TMPDIR" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        ls -ld "$TMPDIR" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        df -h /fsx/scratch >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        export APPTAINER_HOME="$TMPDIR";
        trap 'rm -rf "$TMPDIR" 2>/dev/null || true' EXIT;

        # Find jemalloc in active conda env
        jemalloc_path="";
        for _dir in "$CONDA_PREFIX/lib" "$CONDA_PREFIX/lib64" "$CONDA_PREFIX/lib/x86_64-linux-gnu"; do
            if [[ -d "$_dir" ]]; then
                for _ext in so dylib; do
                    _candidate=$(find "$_dir" -maxdepth 1 -name "libjemalloc*.$_ext*" 2>/dev/null | head -n 1);
                    if [[ -n "$_candidate" && -r "$_candidate" ]]; then
                        jemalloc_path="$_candidate";
                        break 2;
                    fi
                done
            fi
        done

        if [[ -n "$jemalloc_path" ]]; then
            export LD_PRELOAD="$jemalloc_path";
            export MALLOC_CONF=background_thread:true,metadata_thp:auto,dirty_decay_ms:5000,muzzy_decay_ms:5000;
            echo "LD_PRELOAD set to: $LD_PRELOAD" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log;
            echo "MALLOC_CONF set to: $MALLOC_CONF" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log;
        else
            echo "WARNING: libjemalloc not found in CONDA_PREFIX=$CONDA_PREFIX" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log;
        fi

        # --- bin/dayoa_sentieon_cli sentieon-pangenome (accelerated pipeline) ---
        cli_out="$TMPDIR/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug";

        echo "bin/dayoa_sentieon_cli sentieon-pangenome starting (Ultima, CRAM input mode)" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        echo "  input_cram=results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        echo "  model=/fsx/references/runtime_assets/cached_envs/sentieon-genomics-202503.02/bundles/SentieonUltimaPangenomeRealignWGS1.3.bundle" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        echo "  hapl=/fsx/references/genomic_data/organism_references/H_sapiens/panhg38/hprc-v2.0-mc-grch38.hapl" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        echo "  gbz=/fsx/references/genomic_data/organism_references/H_sapiens/panhg38/hprc-v2.0-mc-grch38.gbz" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        set +e;
        bin/dayoa_sentieon_cli sentieon-pangenome             -r /fsx/references/genomic_data/organism_references/H_sapiens/hg38_broad/Homo_sapiens_assembly38.fasta             --hapl "/fsx/references/genomic_data/organism_references/H_sapiens/panhg38/hprc-v2.0-mc-grch38.hapl"             --gbz "/fsx/references/genomic_data/organism_references/H_sapiens/panhg38/hprc-v2.0-mc-grch38.gbz"             -m "/fsx/references/runtime_assets/cached_envs/sentieon-genomics-202503.02/bundles/SentieonUltimaPangenomeRealignWGS1.3.bundle"             --pop_vcf "/fsx/references/genomic_data/organism_references/H_sapiens/panhg38/pop-v20g41-20251216.vcf.gz"             -i results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/ug/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.cram             -b "/fsx/references/genomic_data/organism_references/H_sapiens/panhg38/hg38_canonical.bed"             --dbsnp "/fsx/references/genomic_data/organism_annotations/H_sapiens/hg38/gatk/Homo_sapiens_assembly38.dbsnp138.vcf.gz"             -t 128             "${cli_out}.vcf.gz" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        cli_rc=$?;
        set -e;
        echo "sentieon-cli exit code: $cli_rc" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        if [ $cli_rc -ne 0 ]; then
            echo "ERROR: bin/dayoa_sentieon_cli sentieon-pangenome failed with exit code $cli_rc" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            exit $cli_rc;
        fi

        # --- Reheader VCF: rename sample to cluster_sample ---
        if [ -f "${cli_out}.vcf.gz" ]; then
            oldname=$(bcftools query -l "${cli_out}.vcf.gz" | head -n1);
            echo -e "${oldname}\tR3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA" > "$TMPDIR/rename.txt";
            bcftools reheader -s "$TMPDIR/rename.txt" -o results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz "${cli_out}.vcf.gz" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            bcftools index -f -t --threads 128 -o results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz.tbi results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
        else
            echo "ERROR: VCF not produced by bin/dayoa_sentieon_cli sentieon-pangenome" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;
            exit 20;
        fi

        end_time=$(date +%s);
        elapsed_time=$((($end_time - $start_time) / 60));
        echo "Elapsed-Time-min:\t$itype\t$elapsed_time" >> results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/log/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.log 2>&1;

        

[Tue Jun  9 10:51:32 2026]
localrule produce_pangenome_ug_vcf:
    input: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz.tbi
    output: gatheredall.pangenome_ug
    log: gatheredall.pangenome_ug.log
    jobid: 0
    benchmark: logs/benchmarks/produce_pangenome_ug_vcf.bench.tsv
    reason: Missing output files: logs/benchmarks/produce_pangenome_ug_vcf.bench.tsv, gatheredall.pangenome_ug; Input files updated by another job: results/day/hg38_broad/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA/align/pangenome_ug/spmd/snv/sentpg/R3x-HG003-D0-0-D0-PCR-FREE-UG-ULTIMA.pangenome_ug.spmd.sentpg.snv.sort.vcf.gz.tbi
    priority: 48
    resources: mem_mb=<TBD>, disk_mb=<TBD>, tmpdir=/dev/shm, time=100

( touch gatheredall.pangenome_ug ;

        ls gatheredall.pangenome_ug ) >> gatheredall.pangenome_ug.log 2>&1;
        
Job stats:
job                         count    min threads    max threads
------------------------  -------  -------------  -------------
pre_prep_ultima_cram            1             48             48
prep_results_dirs               1              1              1
produce_pangenome_ug_vcf        1              1              1
sentieon_pangenome_ug           1            128            128
stage_supporting_data           1              1              1
workflow_staging                1              1              1
total                           6              1            128

Reasons:
    (check individual jobs above for details)
    input files updated by another job:
        prep_results_dirs, produce_pangenome_ug_vcf, sentieon_pangenome_ug, workflow_staging
    missing output files:
        pre_prep_ultima_cram, prep_results_dirs, produce_pangenome_ug_vcf, sentieon_pangenome_ug, stage_supporting_data, workflow_staging

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
RETURN CODE: 0
[INFO] Workflow exited with status 0
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

DAY-EC activated.
(DAY-EC) ubuntu@ip-10-0-0-45:/fsx/analysis_results/ubuntu/pg_ultima_ug_dryrun_20260609T104950Z/daylily-omics-analysis$
