DAY-EC activated.
DAY-EC activated.
30 of 36 steps (83%) done
[Sat May 30 19:11:22 2026]
Finished job 20.
31 of 36 steps (86%) done
[Sat May 30 19:11:32 2026]
Finished job 24.
32 of 36 steps (89%) done
[Sat May 30 19:12:13 2026]
Finished job 18.
33 of 36 steps (92%) done
[Sat May 30 19:12:23 2026]
Finished job 22.
34 of 36 steps (94%) done
Select jobs to execute...

[Sat May 30 19:12:23 2026]
rule prep_for_concordance_check:
    input: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.snv.sort.vcf.gz, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.snv.sort.vcf.gz.tbi, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_altair-v1.1/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_altair-v1.1_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_clinvar_genes_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC_x_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_x_clinvar_genes_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38_m_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_m_giabHC_concordance.mqc.tsv
    output: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.done, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.fofn, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.fin.cmds
    log: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/logs/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.concordance.log
    jobid: 7
    benchmark: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/benchmarks/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.concordance.bench.tsv
    reason: Missing output files: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.done; Input files updated by another job: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.snv.sort.vcf.gz.tbi, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38_m_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_m_giabHC_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_altair-v1.1/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_altair-v1.1_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_clinvar_genes_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.snv.sort.vcf.gz, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC_x_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_x_clinvar_genes_concordance.mqc.tsv
    wildcards: sample=TVBCG5X-HG003-5x-1-D0-PF-CG-MGI, alnr=sentcg, ddup=dmd, snv=cgt7p
    priority: 48
    threads: 8
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, threads=16, time=5440, partition=i192mem,i192bigmem, vcpu=16, distribution=block, exclusive=, constraint=, exclude=, include=


            set +euo pipefail;

            mkdir -p $(dirname results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.done);
            mkdir -p $(dirname results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/logs/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.concordance.log);

            utc_ts=$(date -u +%Y-%m-%dT%H:%M:%SZ);

            # Write informational fofn
            {
                echo "# Refactor: per-ROI jobs are scheduled by Snakemake; this file is informational.";
                echo "# generated_at_utc=$utc_ts";
                echo "# truth_dir=/fsx/references/genomic_data/organism_annotations/H_sapiens/hg38/controls/giab/snv/v4.2.1/HG003/";
                echo "# footprints=altair-v1.1,clinvar_genes,giabHC,giabHC_x_clinvar_genes,hg38,hg38_m_giabHC";
                for mqc in results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_altair-v1.1/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_altair-v1.1_concordance.mqc.tsv results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_clinvar_genes_concordance.mqc.tsv results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_concordance.mqc.tsv results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC_x_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_x_clinvar_genes_concordance.mqc.tsv results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_concordance.mqc.tsv results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38_m_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_m_giabHC_concordance.mqc.tsv; do
                    echo "$mqc";
                done;
            } > results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.fofn;

            # Write informational fin.cmds
            {
                echo "# Refactor: see Snakemake DAG for exact commands.";
                echo "# generated_at_utc=$utc_ts";
            } > results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.fin.cmds;

            # SKIPPED sentinel if no mqcs
            if [ "6" -eq 0 ]; then
                echo "No truthset ROI directories found; concordance skipped." > results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.done.SKIPPED;
            fi;

            # Log
            echo "Concordance sentinel complete. footprints=altair-v1.1,clinvar_genes,giabHC,giabHC_x_clinvar_genes,hg38,hg38_m_giabHC mqcs=6" >> results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/logs/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI.sentcg.dmd.cgt7p.concordance.log;

Submitted job 7 with external jobid '4642'.
[Sat May 30 19:12:33 2026]
Finished job 7.
35 of 36 steps (97%) done
Select jobs to execute...

[Sat May 30 19:12:33 2026]
localrule produce_snv_concordances:
    input: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.done, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_altair-v1.1/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_altair-v1.1_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_clinvar_genes_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC_x_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_x_clinvar_genes_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38_m_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_m_giabHC_concordance.mqc.tsv
    output: results/day/hg38_broad/other_reports/giab_concordance_mqc.tsv
    jobid: 6
    reason: Missing output files: results/day/hg38_broad/other_reports/giab_concordance_mqc.tsv; Input files updated by another job: results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38_m_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_m_giabHC_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_hg38/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_hg38_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/concordance.done, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_altair-v1.1/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_altair-v1.1_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_clinvar_genes_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_concordance.mqc.tsv, results/day/hg38_broad/TVBCG5X-HG003-5x-1-D0-PF-CG-MGI/align/sentcg/dmd/snv/cgt7p/concordance/_giabHC_x_clinvar_genes/snv_TVBCG5X-HG003-5x-1-D0-PF-CG-MGI_giabHC_x_clinvar_genes_concordance.mqc.tsv
    priority: 48
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/dev/shm, threads=1, time=5440, partition=i192,i128,i192mem, vcpu=1, distribution=block, exclusive=, constraint=, exclude=, include=

loading global: /fsx/analysis_results/ubuntu/ccv20260530r54_complete_genomics_mgi_snv_concordance/daylily-omics-analysis/config/global_AWSPC.yaml
loading profile rule_config: /fsx/analysis_results/ubuntu/ccv20260530r54_complete_genomics_mgi_snv_concordance/daylily-omics-analysis/config/day_profiles/slurm/rule_config.yaml
INFO::: The genome build hg38_broad is supported.  The genome build prefix is 'chr''.
...WARNING: No aligners set in the config.
...INFO: Auto-detected aligners from env: ['sentcg']
aligners (final): [sentcg]
...INFO: No dedupers set in config. Defaulting to na (no dedup).
...INFO: Auto-detected dedupers. DDUP updated to: ['dmd']
deduper (final): [dmd]
...WARNING: No snv_callers set in the config.
...INFO: Auto-detected SNV callers from env: ['cgt7p']
SNV Callers (final): [cgt7p]
Somatic SNV Callers:[senttn]
... WARNING: No sv_callers set in the config.
SV Callers (final): []
A    N   A   L  Y S I S    SAMPLE TABLE DETECTED ::: /fsx/analysis_results/ubuntu/ccv20260530r54_complete_genomics_mgi_snv_concordance/daylily-omics-analysis/config/samples.tsv
A    N   A   L  Y S I S    UNIT TABLE DETECTED ::: /fsx/analysis_results/ubuntu/ccv20260530r54_complete_genomics_mgi_snv_concordance/daylily-omics-analysis/config/units.tsv
Building DAG of jobs...
All Wildcards:    all snv_CALLERS:  ['cgt7p']  all ALIGNERS:  ['sentcg']  all CRAM_ALIGNERS:  []  all samples:  {'TVBCG5X-HG003-5x-1-D0-PF-CG-MGI': ['TVBCG5X-HG003-5x-1-D0-PF-CG-MGI']}  all concordance samples:  dict_keys(['TVBCG5X-HG003-5x-1-D0-PF-CG-MGI'])
Using shell: /home/ubuntu/miniconda3/envs/DAY-EC/bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, threads=1, time=5440, vcpu=1
Select jobs to execute...
All Wildcards:    all snv_CALLERS:  ['cgt7p']  all ALIGNERS:  ['sentcg']  all CRAM_ALIGNERS:  []  all samples:  {'TVBCG5X-HG003-5x-1-D0-PF-CG-MGI': ['TVBCG5X-HG003-5x-1-D0-PF-CG-MGI']}  all concordance samples:  dict_keys(['TVBCG5X-HG003-5x-1-D0-PF-CG-MGI'])
All Wildcards:    all snv_CALLERS:  ['cgt7p']  all ALIGNERS:  ['sentcg']  all CRAM_ALIGNERS:  []  all samples:  {'TVBCG5X-HG003-5x-1-D0-PF-CG-MGI': ['TVBCG5X-HG003-5x-1-D0-PF-CG-MGI']}  all concordance samples:  dict_keys(['TVBCG5X-HG003-5x-1-D0-PF-CG-MGI'])
[Sat May 30 19:12:36 2026]
Finished job 6.
36 of 36 steps (100%) done
Complete log: .snakemake/log/2026-05-30T183722.883830.snakemake.log


	WARNING!!!! ALL Directories Are Remaining Unlocked. To have directories lock automatically upon successful completion, set --lock-analysis-dirs when calling day-run.



]]]]--------WORKFLOW SUCCESS====================]]]]]--
RETURN CODE: 0
⚠ Active conda environment is 'DAYOA', not DAY-EC. Continuing anyway; the
supported path is `source ./activate`.

── EXPORT ──
  › Preparing direct FSx export DRA
2026-05-30 19:14:00,658 INFO Found credentials from IAM Role: dyec-test-RoleHeadNode-eJMKI1vZVQBE
