DAY-EC activated.
DAY-EC activated.
=== status ===
{
  "command": "bin/day_run produce_sentmm2_align produce_na_dedup_cram produce_sentdpb_snv_vcf produce_alignstats produce_snv_concordances -p -j 2 -k -T 1",
  "completed_at": "2026-05-29T22:40:31Z",
  "exit_code": 1,
  "repo_path": "/fsx/analysis_results/ubuntu/ccv20260529r6_pacbio_snv_alignstats/daylily-omics-analysis",
  "session_name": "ccv20260529r6_pacbio_snv_alignstats",
  "started_at": "2026-05-29T21:55:22Z"
}
=== repo ===
/fsx/analysis_results/ubuntu/ccv20260529r6_pacbio_snv_alignstats/daylily-omics-analysis
=== latest snakemake logs ===
1780094428.0000000000 .snakemake/log/2026-05-29T215545.957359.snakemake.log
1780094433.0000000000 .snakemake/log/2026-05-29T224031.046805.snakemake.log
=== latest log path: .snakemake/log/2026-05-29T224031.046805.snakemake.log ===
=== latest log tail ===
Unlocking working directory.
=== rule logs with errors ===
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:11:[E::hts_open_format] Failed to open file "results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/tp_vcf_gz_stripped.vcf.gz" : No such file or directory
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:12:[init_data] Error: cannot write to "results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/tp_vcf_gz_stripped.vcf.gz": No such file or directory
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:13:tbx_index_build3 failed: results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/tp_vcf_gz_stripped.vcf.gz
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:15:Traceback (most recent call last):
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:28:FileNotFoundError: [Errno 2] No such file or directory: 'results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/tp_vcf_gz_stripped.vcf.gz'
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:30:During handling of the above exception, another exception occurred:
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:32:Traceback (most recent call last):
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:36:FileNotFoundError: [Errno 2] No such file or directory: 'results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/tp_vcf_gz_stripped.vcf.gz.X_tv_snp.vcf'
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:44:[E::hts_open_format] Failed to open file "results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fp_vcf_gz_stripped.vcf.gz" : No such file or directory
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:45:[init_data] Error: cannot write to "results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fp_vcf_gz_stripped.vcf.gz": No such file or directory
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:46:tbx_index_build3 failed: results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fp_vcf_gz_stripped.vcf.gz
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:48:Traceback (most recent call last):
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:61:FileNotFoundError: [Errno 2] No such file or directory: 'results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fp_vcf_gz_stripped.vcf.gz'
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:63:During handling of the above exception, another exception occurred:
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:65:Traceback (most recent call last):
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:69:FileNotFoundError: [Errno 2] No such file or directory: 'results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fp_vcf_gz_stripped.vcf.gz.X_tv_snp.vcf'
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:71:[E::hts_open_format] Failed to open file "results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fn_vcf_gz_stripped.vcf.gz" : No such file or directory
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:72:[init_data] Error: cannot write to "results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fn_vcf_gz_stripped.vcf.gz": No such file or directory
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:73:tbx_index_build3 failed: results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fn_vcf_gz_stripped.vcf.gz
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:75:Traceback (most recent call last):
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:88:FileNotFoundError: [Errno 2] No such file or directory: 'results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fn_vcf_gz_stripped.vcf.gz'
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:90:During handling of the above exception, another exception occurred:
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:92:Traceback (most recent call last):
results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.altair-v1.1.parse_vcfeval_summary.log:96:FileNotFoundError: [Errno 2] No such file or directory: 'results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/concordance/_altair-v1_1/fn_vcf_gz_stripped.vcf.gz.X_tv_snp.vcf'
logs/slurm/sentmm2_align_sort/sentmm2_align_sort.TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.3.err:55:        ulimit -n 65536 || echo "ulimit mod failed";
logs/slurm/sentmm2_align_sort/sentmm2_align_sort.TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.3.err:62:            echo "ERROR: Failed to create TMPDIR: $TMPDIR" >> results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2_sort.log 2>&1;
logs/slurm/sentmm2_align_sort/sentmm2_align_sort.TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.3.err:102:            echo "ERROR: TMPDIR disappeared before pipeline start: $tdir" >> results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/logs/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2_sort.log 2>&1;
logs/slurm/sent_snv_pacbio/sent_snv_pacbio.TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.1.err:62:        ulimit -n 65536 || echo "ulimit mod failed" >> results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/log/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.snv.log 2>&1;
logs/slurm/sent_snv_pacbio/sent_snv_pacbio.TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.1.err:99:            echo "ERROR: sentieon-cli dnascope-longread failed with exit code $cli_rc" >> results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/log/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.snv.log 2>&1;
logs/slurm/sent_snv_pacbio/sent_snv_pacbio.TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.1.err:110:            echo "ERROR: SNV VCF not produced by sentieon-cli" >> results/day/hg38_broad/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO/align/sentmm2/na/snv/sentdpb/log/TVBPB5X-HG003-5x-1-D0-PF-PACBIO-REVIO.sentmm2.na.sentdpb.snv.log 2>&1;
